FINAL EVALUATION REPORT USAID ADVANCING KOSOVO TOGETHER Final Performance Evaluation DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION ii FINAL EVALUATION REPORT USAID Advancing Kosovo Together Final Performance Evaluation USAID/Kosovo AID-167-TO-17-00007; AID-167-I-17-00003 August 9, 2017 COVER PHOTO CREDIT: ADVANCING KOSOVO TOGETHER - LOCAL SOLUTION This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Mateusz Pucilowski, Jean-Camille Kollmorgen, Besnik Cecelia and Dijana Spalevic on behalf of Social Impact, Inc. Contact: Social Impact, Inc. 2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 Kerry Bruce, Executive Vice President kbruce@socialimpact.com Phone: +1 703.465.1884 Fax: +1 703.465.1888 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION iii CONTENTS Tables and Figures ....................................................................................................................... iv Acronyms ....................................................................................................................................... v Executive Summary ..................................................................................................................... vi Project Background ..................................................................................................................................................... vi Evaluation Purpose and Questions .......................................................................................................................... vi Evaluation Methods and Limitations ....................................................................................................................... vii Evaluation Question 1 – To what extent has the project been effective? ................................................... viii Evaluation Question 2 - What are the key lessons learned from the project to date? ............................. xii Recommendations ..................................................................................................................................................... xiii Project Background ...................................................................................................................... 1 Program Background and Description .................................................................................................................... 1 Evaluation Purpose and Evaluation Questions ........................................................................... 4 Evaluation Purpose ....................................................................................................................................................... 4 Evaluation Questions ................................................................................................................................................... 4 Evaluation Methods and Limitations ........................................................................................... 5 Methods .......................................................................................................................................................................... 5 Limitations ...................................................................................................................................................................... 7 Findings and Conclusions.............................................................................................................. 8 Evaluation Question 1 ................................................................................................................................................. 8 Conclusions .................................................................................................................................................................. 30 Evaluation Question 2 ............................................................................................................................................... 33 Recommendations ...................................................................................................................... 38 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION iv TABLES AND FIGURES Table 1: Implementation Components within AKT .................................................................................................. 3 Table 2: AKT-LS Performance Under Objective 1 .................................................................................................... 9 Table 3: AKT-US Performance Under Objective 1 ................................................................................................. 10 Table 4: AKT-US Performance Under Objective 3 ................................................................................................. 18 Table 5: AKT-LS Performance Under Objective 3 .................................................................................................. 19 Table 6: Sex and Ethnic Disaggregation AKT-US..................................................................................................... 21 Table 7: AKT-US Performance under Objective 2 .................................................................................................. 25 Table 8: AKT-LS Performance under Objective 2 ................................................................................................... 26 Table 9: AKT-US Performance under Objective 4 .................................................................................................. 29 Table 10: Respondents Interviewed ................................................................. Error! Bookmark not defined. Figure 1: AKT Results Framework ................................................................................................................................ 2 Figure 2: Data Collection Sites by Municipality ......................................................................................................... 6 Figure 3: Respondent Types ............................................................................................................................................ 6 Figure 4: Repondent Gender by Municipality ................................................Error! Bookmark not defined. Figure 5: Map of Kosovo Municipalities ...........................................................Error! Bookmark not defined. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION v ACRONYMS AKT Advancing Kosovo Together ASM Serb-Majority Municipalities B2B Business 2 Business CDCS Country Development Cooperation Strategy CDF Community Development Fund CF Community Fieldwork COR Contracting Officer Representative CSO Civil Society Organization DRG Democracy, Human Rights and Governance EQUI Evaluation Quality, Use, and Impact ET Evaluation Team EU European Union FGD Focus Group Discussion FTE Full-time Equivalent GI Group Interview IP Implementing Partner IR Intermediate Result JWG Joint Working Group KCC Kosovo Chamber of Commerce KII Key Informant Interview KRD Kosovo Relief Development LEAP Local Environmental Action Plan LOP Life of Project LS Local Solutions LUL Law on the Use of Languages M&E Monitoring and Evaluation NGO Non-Governmental Organization OCA Organizational Capacity Assessment OLC Office of the Language Commissioner PMP Performance Management Plan QA Quality Assurance RAE Roma, Ashkali and Egyptians RFA Request for Application SI Social Impact SIAP Service Improvement Action Plan MSME Minority-Owned Small and Medium Enterprise SOW Scope of Work TPM Team Planning Meeting USAID United States Agency for International Development USG United States Government USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION vi EXECUTIVE SUMMARY USAID/Kosovo contracted Social Impact Inc. to conduct a performance evaluation of the Advancing Kosovo Together (AKT) activity1 on January 17, 2017. This report presents findings, conclusions and recommendations structured around the two primary evaluation questions. PROJECT BACKGROUND Advancing Kosovo Together (AKT) is a USAID-funded activity designed to increase constructive inter￾ethnic cooperation between Kosovo’s majority and minority populations. AKT is implemented through two complementary mechanisms. Chemonics International implemented the $7,587,654 AKT Prime mechanism (also known as AKT-US) between April 2014-April 2017. Community Development Fund (CDF) continues, at the time of writing, to implement the $6,999,706 AKT Local Solution mechanism (also known as AKT-LS). AKT-LS began in October 2014 and is scheduled to conclude in March 2018. The two mechanisms were designed to work together in order to increase constructive inter-ethnic cooperation and interaction by achieving three intermediate Objectives: Objective 1 – Improved autonomous, horizontal communication between (a) Kosovo Albanian and non-majority communities; and (b) the Government of Kosovo and non-majority communities Objective 2 – Improved economic opportunities in target municipalities Objective 3 – Increased efficiency and capacity of target municipal administrations to respond to the needs of all their citizens In addition to these three shared objectives, AKT-US was designed to provide capacity building support to the AKT-LS implementer. Accordingly, AKT-US had an additional Objective 4. AKT implements activities across 16 Kosovo partner municipalities, including six Serb-majority municipalities in the South, six Albanian-majority municipalities, and four municipalities in northern Kosovo. EVALUATION PURPOSE AND QUESTIONS The purpose of the evaluation, as articulated by the USAID/Kosovo Mission was to: 1. Inform the 2018-2023 USAID/Kosovo Country Development Cooperation Strategy (CDCS) and future programing; and 2. Know whether the AKT engagement—focusing on minority community integration—has been working. The primary target audience for the evaluation is USAID/Kosovo and AKT implementing partners (IPs). Other key stakeholders include other US Government (USG) entities and international donors supporting ethnic minority integration in Kosovo; the Government of Kosovo (GoK), particularly local government agencies such as the Ministry of Local Government Administration, Ministry of Communities and Returns, and Ministry of Finance; and municipal government officials. The evaluation had two main evaluation questions, each with a number of sub-questions: 1 When AKT was first designed, it was referred to as a “program” within USAID. The term now commonly used per the ADS to refer to a program such as AKT is “activity”. AKT is referred to as an activity throughout the remainder of the report. Similarly, smaller activities undertaken by AKT are referred to as “interventions” throughout the report. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION vii 1. To what extent has the project been effective? a. Has the program achieved its intended results, especially under Objectives 1 and 3? Why or why not? b. What are AKT achievements in North Kosovo to date? What worked and what did not? c. How has the sustainability of USAID investments been addressed? 2. What are the key lessons learned from the project to date? a. Should there be any programmatic shifts to the AKT program to achieve better results? Are there new opportunities for minority integration that future programming could address? b. Is there a potential for greater AKT program collaboration with other USAID programs and/or other donors? EVALUATION METHODS AND LIMITATIONS Primary data was collected through two rounds of data collection. In both rounds, the ET held in-briefs and out-briefs with USAID. In the first round the ET purposively selected 10 out of the 16 AKT municipalities using three geographic and ethnic clusters. From these municipalities, the team selected individuals and organizations across five priority respondent strata: implementing partners, central government officials, municipal government officials, village/community councils, and local businesses. Sampling frames were provided by IPs in the form of detailed beneficiary lists, mapped against programmatic objectives and/or activities. The final sampling plan included a municipal official from each municipality, and two to five community and/or civil society beneficiaries to address the USAID focus on Objectives 1 and 3. The ET selected business representatives in four municipalities: two in the North, one Serb-majority municipality in the South, and one Albanian-majority municipality. The ET also selected national-level officials from the ministries with selection based on their involvement in or knowledge of AKT. The second round was designed to deepen understanding of community-level implementation, dynamics and outcomes. The ET selected a sub-set of previously sampled municipalities to reflect: varying ethnic compositions, North-South geographic dispersion, and AKT implementation dimensions identified through first round interview notes. Within each municipality, the ET selected villages and attempted to speak with both Serb and Albanian community leaders in separate interviews. As detailed in Annex C, the ET completed the set of interviews in six villages. In addition to this core design element, second round data collection included four case studies in North Kosovo (three school-level quick impact projects and one Service Improvement Action Plan intervention at the municipal level), and interviewed one informal women’s group representative per municipality. Across both rounds, the ET completed 61 interviews with 107 respondents (80 male, 27 female) through key informant interviews (KIIs) and group interviews (GIs). The detailed methodology and data collection tools are available in Annex B – Evaluation Design and Methodology. The research design has four primary limitations. First, at the time of the interview, some respondents had not interacted with AKT for over a year. This elapsed time could lead to recall bias. Second, sampling frames were largely derived from IP beneficiary lists. While the ET used other sources to identify respondents, this primary sampling approach could have introduced some selection bias. Specifically, interview selection could have been positively biased toward individuals with more interaction with the activity and overly represented male perspectives. The ET used a number of strategies to interview female beneficiaries, however but the gendered nature of community leadership and participation in Kosovo skewed participation, and consequently sampling frames, toward males. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION viii Third, the socio-political context of ethnic-relations in Kosovo is highly complex and charged. The team was expecting to encounter response bias, especially with regard to underreporting tensions, and tried a number of strategies to solicit truthful responses. Respondents were provided the option of meeting location, interviews were conducted in respondent mother-tongue whenever possible, and some interviews were conducted solely by local team members. EVALUATION QUESTION 1 – TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE?2 FINDINGS Objective 1 Improving communication between Kosovo Albanians and non-majority communities According to monitoring data, both AKT-US and AKT-LS met or exceed the majority of their performance targets. AKT-LS data indicate 3,360 direct beneficiaries and 99,471 indirect beneficiaries (targets = 2,500 and 90,000, respectively). AKT-US data register 24,276 community members as beneficiaries of small-scale infrastructure project and an additional 4,189 attendees at events designed to strengthening understanding and mitigating conflict between groups (targets = 21,266 and 1,240, respectively). Respondents at the community and municipal levels were very satisfied with the professionalism, dedication, and efficacy of AKT staff. Working through local coordinators, AKT-US organized community meetings to identify and reach consensus on infrastructure needs. The model relied on broad-based community participation to foster inter-ethnic (and as a secondary consideration, cross￾gender) communication and cooperation. The activity was successful in generating demand at the community level, with community forums generally well-attended. However, in many instances the lack of direct financial benefit to participants and the time required to participate in meetings resulted in significant attrition. The ET found one exception to equitable ethnic participation in a Serb-majority community where, according to the Serb community leader, AKT did not generate any Albanian participation. In almost all interviews with community members, respondents described inter-ethnic relations in the community forums as cordial and equitable. The team found evidence of ethnic groups genuinely working together through AKT activities, though there were also a limited number of instances where ethnic tension may have been exacerbated. These included arguments over infrastructure priorities, some perceptions of inequitable benefits from infrastructure improvements, and an instance where an AKT-supported road was named in a way that offended a minority ethnic group. While these cases are worthy of attention, it is important to note that they are outliers in a process that solicited largely equitable participation from community members of different ethnic groups. The activity clearly galvanized short-term communication and cooperation between ethnic groups in structured activities. There was some evidence that community-level impacts materialized through three different causal pathways. First, by working together toward a common goal in a structured environment, interpersonal bonds were created and strengthened between community members. Second, multiple respondents brought up use of common spaces by different ethnic groups, particularly 2 The evaluation question formally prioritized AKT Objectives 1 & 3. We present results from these two objectives in the executive summary. Detailed findings from the other objectives can be found in the body of the report. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION ix as they related to child and youth play. Third, some respondents reported low-level improvements in community relations. These sentiments were, however, outweighed by community responses indicating a lack of belief in lasting, significant impacts on inter-ethnic integration. These responses could be grouped in two categories. First, respondents claimed that inter-ethnic cooperation was positive before the activity came and, as such, was not affected any substantive way by AKT. Second, the intensity and duration of the intervention were not sufficient to affect such high-level and culturally-entrenched outcomes in a sustainable manner. Improving communication between the Government of Kosovo and non-majority communities AKT worked through municipal officials in all aspects of programming. In addition to municipal cost￾sharing, municipal representatives participated in trainings, managed procurement for the infrastructure projects, participated in Joint Working Group (JWG) meetings, and led AKT-supported public budget hearings. These activities were intended to facilitate structured interaction between municipal officials and communities. When asked about community-municipal relations before AKT, the prevailing sentiment across community respondents was that there was little-to-no interaction. The general sentiment of respondents was that municipalities had limited budgets and did not have much to offer to villages. Even in more affluent municipalities, however, respondents reported a generalized lack of engagement from local government. AKT galvanized increased interaction between the two stakeholder groups leading to some increased goodwill on the part of the community. A number of community respondents noted that AKT interventions were the only ones that got municipal officials to visit the communities, and the respondents appreciated the municipal officials’ presence. Other community representatives spoke about personal connections they made with municipal officials and how these connections have assisted them. While there were certainly cases of improved relations, the ET found evidence of some interactions leading to a worsening of perceptions toward local government. Some of this was driven by frustrations in the procurement process, where a number of community representatives reported feeling disempowered. Achievements in North Kosovo Due to concerns about potential corruption, the AKT model was modified for the North Kosovo context. In lieu of small-scale infrastructure investments, the activity completed ‘quick impact’ interventions. The intent was similar, though the prioritization process and procurement thresholds were more constrained. The ET interviewed stakeholders from three supported schools in Zubin Potok/Zubin Potok (two in Serb majority communities and one in an Albanian majority community). In the two Serb community schools supported by AKT-LS, the team found evidence of a robust consensus￾building process between school management, teachers and parents association representatives. In contrast, the school in the Albanian community was more resource-constrained and received a less intensive engagement from AKT. According to a key informant, AKT only worked with one individual as opposed to a consensus-driven process with multiple community constituencies. While these quick impact interventions deliver educational materials that will be used by many children, the fact that all schools in Zubin Potok/Zubin Potok are ethnically homogenous is a serious impediment to achieving the programmatic objective of increasing inter-ethnic cooperation. Not only are all benefits from these investments at the school level used exclusively by a single ethnic group, the fact that Serb schools received more financial support than the single Albanian majority school, the activity runs the risk of perpetuating perceptions of inequality. Sustainability USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION x Benefits stemming from physical investments funded by AKT, particularly small-scale infrastructure interventions, are likely to persist for years. These investments seem to have been well-constructed and well-utilized. Quick impact interventions in North Kosovo will also persist, though asset depreciation should be expected to be greater in school environments. Community mobilization through formalized village-level meetings seem very unlikely to persist beyond the life of the activity. There was no evidence of any spontaneous, structured meetings not affiliated with AKT. Similarly, it is highly unlikely that the gains in community-municipality communication affected by AKT will persist. While it’s possible that some aspects of community outreach will continue, without external financing municipal governments will most likely revert to pre-intervention behavior. Lastly, the prospects for sustaining advances made in the overall programmatic objective of bringing majority and minority ethnic groups together are not promising. It is highly likely that community members will continue using infrastructure financed by AKT and that this common use will facilitate some small-scale interactions (e.g. children playing together). It is also possible that interpersonal relationships formed through participation in the activity will persist. However, because the activity was not successful in substantively affecting community-level changes in ethnic cooperation, despite achieving and sometimes exceeding output targets, it is highly unlikely for these benefits to sustain. Objective 3 Enhance capacity of municipal administrations for strategic planning and to deliver and oversee quality service AKT supported municipal governments through a number of complementary activities. In the first year of the activity, AKT facilitated self-assessments of municipal capacity for each of the 16 targeted municipalities. On the basis of these documents, municipalities drafted Capacity Development Plans (CDPs) to strengthen perceived weaknesses. In conjunction with this process, municipalities developed Service Improvement Action Plans (SIAPs) to identify and co-finance investments that would improve municipal service delivery and received various trainings. These plans included a variety of interventions supporting energy efficiency, public lighting, waste management, primary health care, education (university and preschool), and cultural centers. Performance monitoring data indicate that an estimated 30,267 people are receiving improved service as a result of these interventions, with almost 80 percent of estimated beneficiaries Serb. Additionally, municipal officials interviewed by the ET reported the development of SIAPs as a positive step in improving service delivery and for planning economic development. AKT provided training to municipal workers on a wide range of topics, including: public procurement, strategic planning, human resource management, budgeting, project management, environment protection, public transparency, democratization, gender and human rights. According to the most up￾to-date performance monitoring data the ET could acquire, between the two mechanisms AKT trained a total of 2,311 people with the goal of strengthening local government. This figure represents a 38 percent overachievement relative to the performance target of 1,680. Field interviews with municipal representatives who attended these trainings suggested a significant divergence in satisfaction. Positive feedback included one respondent attributing improved service delivery to AKT capacity building activities, an official reporting that the municipality was operating without a human resources person until one was trained by AKT-US, and multiple respondents requesting additional training. Feedback was not uniformly favorable, however. Criticisms of the AKT capacity building approach centered on four factors. A number of respondents spoke of heavy workloads that inhibited the ability of municipal workers to participate. Secondly, some respondents spoke of a generalized training fatigue among municipal officials. Third, a number of respondents expressed a perception that AKT trainings did not help them perform their jobs better. Finally, although few respondents mentioned translation issues USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xi for different trainings across objectives, the ET registered one complaint about an Albanian AKT trainer being sent to North Kosovo to train municipal officials. Achievements in North Kosovo Northern municipalities face unique political dynamics and staffing shortages and participated less than other municipalities in AKT programming. Respondents from these municipalities identified more pronounced capacity-building needs, including difficulty filling job vacancies, training newly-hired and seasoned staff, learning and adjusting to Kosovo law, and building more effective relations with central institutions. Municipal officials noted that due to staffing gaps, the municipality cannot always participate in trainings that are offered. The ET investigated one SIAP intervention in North Kosovo. The plan resulted in the replacement of a boiler to improve efficiency of heating the municipal building and an adjacent cultural center. While the process of consensus building, procurement preparation and management may have increased competencies of those involved, benefits of the investment seemed to accrue primarily to municipal workers. According to the director of the cultural center, the improved heating has improved the ability of the municipality to deliver cultural programming but only ethnic Serbs use the facility. Sustainability The physical investments made through SIAPs should be expected to persist for years and provide services to municipal residents. The plans themselves were designed to be actionable for four years and the engagement of community forum members to monitor activities could strengthen demand-side pressures. If municipal officials continue to follow through on them, the design feature should promote sustainability of capacitation efforts. However, many municipal respondents expressed skepticism that the momentum will continue absent AKT. Because municipalities are operating with limited funds, their ability to serve their constituencies is inhibited. In this context, capacity building can only do so much toward improving service provision. CONCLUSIONS Objective 1  AKT was largely successful in fostering community participation from different ethnic groups. Representatives of these groups collaborated to identify community needs in a broadly equitable manner, and infrastructure improvements were delivered to the satisfaction of communities.  The transition from decision-making at the community forum level to procurement management at the municipal level was viewed by some community members as opaque, disenfranchising and, in some instances, disrespectful.  There were indications of small-scale improvements in inter-ethnic relations, particularly from strengthened/created interpersonal relationships among participants, common usage of infrastructure funded by AKT (especially spaces for children and youth), and low-level trust￾building through the process of working on structured interventions. However, the intervention was not sufficiently intensive, neither in duration or contact time, to galvanize substantive community-level impacts.  The activity succeeded in fostering communication between community and municipal representatives. Community perceptions of these interactions and their opinion of municipal leadership varied widely. Some respondents spoke of improved relations, some of worsened relations, though most reported not changing their perception. Even in instances where cooperation improved, these changes were relatively small and are unlikely to affect inter-ethnic cooperation at a community-level. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xii  Infrastructure investments will persist for many years after AKT closes out. However, improvements in community communication and collaboration should be expected to atrophy absent any future, structured engagements.  AKT’s activities in North Kosovo will likely improve the quality of education for participating schools. However, due to the fact that all schools in Zubin Potok/Zubin Potok are ethnically homogenous, these investments will not have any substantive impact on inter-ethnic communication or collaboration. Furthermore, because Serb-majority schools received far more aid than the Albanian-majority school, the activity runs the risk of exacerbating inter-ethnic tensions. Objective 3  AKT was largely successful in meeting its performance targets under objective 3. All 16 targeted municipalities received CDPs and successfully drafted SIAPs.  The successful engagement and cost-sharing with the municipalities in excess of $600 thousand should be viewed as a success. While it was beyond the scope of this evaluation to independently verify beneficiary estimates, AKT claims these investments will benefit over 30,000 people.  AKT provided 2,353 people a wide variety of trainings. Some respondents reported deep satisfaction with these sessions, particularly when trainings directly affected job functions (e.g. procurement certification). However, the ET found evidence of trainings being underattended due to heavy workloads, training fatigue, and, in some cases, perceptions that offerings were not useful.  Overall, despite the improvements in municipal capacity, most municipal officials themselves— plus central-level officials, community and business respondents, and other knowledgeable experts interviewed—noted that municipalities will need to improve service delivery to address citizen needs. EVALUATION QUESTION 2 - WHAT ARE THE KEY LESSONS LEARNED FROM THE PROJECT TO DATE? Theory of Change While both mechanisms achieved impressive results from an output perspective, AKT did not seem to affect substantive changes in the programmatic goal of increasing inter-ethnic cooperation. Field data indicate that implementation intensity was insufficient to affect the ambitious programmatic objectives due to an ambitious program design and a relatively short 3-year timeframe. Targeting The selection of ethnically homogenous schools in North Kosovo or villages that are practically ethnically homogenous in South Kosovo is misaligned with the program theory of change. For activities working on improving inter-ethnic cooperation, there should be some benchmark of minimum participation to be considered for support. Community Leaders The efficacy of activities like AKT depends largely on the community leaders. Successful leaders are the product of a number of factors. They must have (1) time to engage with work on behalf of their community, the (2) capacity to represent their community, (3) be proactive in advocating for community interests, and (4) command the respect of their communities. Elite Capture USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xiii Activities relying on disbursement of goods or services through participatory means will always have to cope with stakeholders attempting to divert investments in personally beneficial means. While the evaluation did not find evidence of corruption, the ET documented three cases of potential elite capture. The first example revolved around the selection of schools for receipt of AKT support in Zubin Potok/Zubin Potok, the second example concerned municipal officials receiving the majority of programmatic benefits, and the third was about a mayor potentially subverting a democratic process to select an infrastructure priority. Inter-ethnic Cooperation through Work Interview data across all respondent types evidenced a consistent, shared belief that the best conduit toward true inter-ethnic cooperation was through shared economic interests. Taken in conjunction with the finding that gains in inter-ethnic cooperation are likely to atrophy without structured activities to bring people from different communities together, the notion that a long-term, shared engagement through work seems like an approach that would promote inter-ethnic cooperation and provide spillover benefits with respect to economic growth. RECOMMENDATIONS  The ET recommends significantly simplifying programming, particularly in this complex socio￾political context. AKT implementers were engaged in too many disparate activities over too short a time to focus programming and affect substantive changes.  Inter-ethnic, community-level programming in Kosovo has to be highly contextualized. To the extent possible, USAID should design implementation mechanisms with the flexibility necessary for meaningful adaptive management. This could take the form of a trial-and-error process where different approaches are piloted and, if successful, scaled. For this approach to work, a more robust monitoring and evaluation (M&E) system is necessary.  If USAID is considering a co-implemented activity in the future, the Agency should conduct more thorough planning before procuring either mechanism. Ideally, both mechanisms should be designed in advance of procurement, with clearly delineated scopes (either geographically or sectorally). Additionally, both procurements should make reference to the other mechanism and explain in what ways the two contracts are expected to cooperate. These signals will result in more realistic budgeting during the procurement phase and a more efficient start-up process.  If USAID plans on implementing similar programming in the future, eligibility criteria for communities should be strengthened to increase the likelihood of affecting community-level outcomes and increasing the cost-effectiveness of USG programming. The ET recommends two specific factors to consider in setting community minimum eligibility screening: communities should (1) have a minimum number of ethnic minority residents to participate, (2) register a minimum level of participation from targeted ethnic groups for programming to continue.  Given the broad consensus from respondents at all stakeholder levels, USAID should continue to incorporate job creation activities in this type of programming. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xiv USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 1 PROJECT BACKGROUND PROGRAM BACKGROUND AND DESCRIPTION Attempting to reduce tension in the North, the Government of Kosovo (GoK) and Government of Serbia (GoS) entered into a European Union (EU)–facilitated Brussels Agreement of 2013.3 This agreement outlined key principles toward sustainable peace and integration, including how to address local governance issues such as police, judiciary, and local elections. Though municipalities still encounter challenges in delivering essential services and meeting the GoK’s target commitments related to these local governance issues, implementation of the agreement is in progress. The Northern Serb population is starting to show signs of increased cooperation with other parts of the country and is following Belgrade institutions less strictly. Paradoxically, this relaxation has created new challenges; any move toward political compromise between Kosovo Serbs and Kosovo Albanians is viewed as undesirable among both Albanian and Serb nationalists. Due to past conflicts, relations between the Kosovo-Albanian majority and the Kosovo-Serb minority population remain fragile. Kosovo Serbs and other minorities report lingering difficulties, e.g. in accessing markets and value chains dominated by the majority community, and in receiving public services. Kosovo Serbs form a majority in ten municipalities. Four of these municipalities in the North used to operate under direct control of Belgrade after the war of 1998–99. The GoK has only recently started exercising control over the North as a result of the 2013 Agreement. The other six Serb-majority municipalities in the rest of the country are often regarded as enclaves, although they are considered more structurally integrated into the Kosovo system than the municipalities in the North. Under the Brussels Agreement, Serbia agreed to dismantle its parallel institutions in Kosovo. These parallel institutions still employ thousands of Kosovo Serbs, even if many employees draw a salary without doing actual work. In turn, Kosovo agreed to create an Association/Community of Serb￾Majority Municipalities (ASM). The integration of the parallel institutions is viewed as an opportunity for the development of Kosovo. ASM is seen to represent a central Kosovo Serb entity, which would not only channel the assistance/subsidies of Belgrade but also represent an umbrella organization that would serve as an intermediary between territorially and politically divided Serb municipalities. The health and education sectors are foreseen to be managed by the ASM, and further subsidized by Belgrade. Though the GoK continues to make progress toward political and economic stability, it also faces a number of key challenges regarding the integration of non-majority populations into Kosovo’s institutions and society. Advancing Kosovo Together (AKT) is a USAID-funded activity aiming to increase constructive inter-ethnic cooperation and interaction between Kosovo’s majority and minority populations, which is vital to the security and stability of the region.4 AKT is implemented through two complementary mechanisms—AKT Prime (also known as AKT-US and implemented by Chemonics International) and AKT Local Solution (AKT-LS, implemented by Community Development Fund, or CDF). AKT-US began in April 2014 and concluded in April 2017 (Award Number: AID-167-C-14- 00003, Award Amount: $7,587,654), and AKT-LS began in October 2014 and will conclude in March 2018, following a no-cost extension that was granted at the time of this evaluation (Award Number: AID-167-A-14-00008, Award Amount: $6,999,706). See Annex A – Statement of Work for more 3 The Brussels Agreement was made between the governments of Serbia and Kosovo in 2013, with the goal of normalizing relations between the two countries. The agreement process calls for the parties to sign particular sections of the agreement as those issues are agreed to. 4 https://www.usaid.gov/news-information/fact-sheets/advancing-kosovo-together-akt USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 2 information on the two mechanisms. Additionally, both AKT-US and AKT-LS have three subcontractors each. The subcontractors for AKT-US are: Business Support Center Kosovo, Development Professionals Inc. and the Berman Group. The subcontractors for AKT-LS are: Kosovo Relief Development (KRD), Centre for Peace and Tolerance, and AKTIV NGO. AKT’s theory of change for both mechanisms posits that increased constructive inter-ethnic cooperation and interaction will be achieved as a result of: 1. Improved autonomous, horizontal communication between: a) Kosovo Albanian and non￾majority communities; and b) the GoK and non-majority communities (Objective 1) 2. Improved economic opportunities in target municipalities (Objective 2) 3. Increased efficiency and capacity of target municipal administrations to respond to the needs of all their citizens (Objective 3) Within USAID/Kosovo’s Country Development Cooperation Strategy, AKT is intended to contribute towards Intermediate Result (IR) 1.3, “Improved integration of ethnic minorities into Kosovo society,” which, in turn, supports Development Objective 1, “Improved rule of law that meets citizen needs.” Figure 1: AKT Results Framework In addition to the three shared objectives mentioned above, AKT-US was intended to provide capacity building to the AKT-LS implementer, and thus, includes an additional and separate Objective 4: “Build the capacity of host country organizations by leveraging local, cross-ethnic leadership to implement program objectives 1, 2, and 3.” This implementation structure also supports USAID Forward and efforts to promote sustainability at a local level. AKT implements activities across 16 Kosovo partner municipalities, including six Serb-majority municipalities in the south, six Albanian-majority municipalities, and four municipalities in northern USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 3 Kosovo (See Annex F for list of AKT municipalities). AKT works across three distinct implementation components, namely: 1) municipal development; 2) business development; and 3) citizen engagement.5 Table 1 outlines the major implementation components within AKT. Table 1: Implementation Components within AKT TECHNICAL AREA ACTIVITIES Municipal Development Support GoK in implementation of laws that protect minority rights Improve government communications, responsiveness, and engagement with non￾majority communities Build skills of minorities and women to obtain public-sector positions Provide constructive engagement through non-infrastructure and small-scale infrastructure projects Enhance capacity of municipal administrations for strategic planning and to deliver and oversee quality services Support municipalities to increase own-source revenue for municipal services and capital projects Build capacity of Municipal Offices of Communities and Returns Strengthen municipal capacity to manage government-to-government awards Business Development Assess and facilitate market linkages between majority and non-majority businesses Promote small and medium enterprise (SME) competitiveness and access to finance Support youth employability Support women-owned enterprises Citizen Engagement Strengthen mechanisms to promote transparency and community engagement Build the capacity of host-country organizations by leveraging local, cross-ethnic leadership to implement program objectives Facilitate community-led initiatives among majority and minority communities to address common issues Increase CSO capacity to preserve and advocate for minority rights 5 AKT-LS Year 1 Annual Report USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 4 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE This report is a performance evaluation of the Advancing Kosovo Together (AKT) activity, inclusive of its two mechanisms, AKT-US and AKT-LS. Considerable time elapsed between the drafting of the original Statement of Work (SOW) for the evaluation and the conduct of the evaluation itself. Importantly, AKT was nearing its conclusion when the evaluation team (ET) was mobilized. The purpose of the evaluation, as articulated by the USAID/Kosovo Mission during the in-brief in February 2017,6 was to: 1. Inform the 2018-2023 USAID/Kosovo Country Development Cooperation Strategy (CDCS) and future programing; and 2. Know whether the AKT engagement—focusing on minority community integration—has been working. The above two points help to refine the purpose as outlined in the evaluation SOW:7 1. Evaluate the programmatic progress of the AKT program and its two separate implementing mechanisms—AKT-US Prime and AKT-LS; 2. Identify strengths and weaknesses, and describe the effectiveness of the project; 3. Provide key findings, conclusions, and recommendations for the two AKT activities; and 4. Provide recommendations for future activities. The primary target audience for the evaluation is USAID/Kosovo, AKT IPs, and their partners. Other key stakeholders include other US Government (USG) entities and international donors supporting ethnic minority integration in Kosovo, the GoK, particularly local government agencies such as the Ministry of Local Government Administration, Ministry of Communities and Returns, and Ministry of Finance, and municipal government officials. EVALUATION QUESTIONS The evaluation had two main evaluation questions, each with sub-questions. 1. To what extent has the project been effective? 1.1 Has the program achieved its intended results, especially under Objectives 1 and 3? Why or why not? 1.1a What are AKT achievements in North Kosovo to date? What worked and what did not? 1.2 How has the sustainability of USAID investments been addressed? 2. What are the key lessons learned from the project to date? 2.1 Should there be any programmatic shifts to the AKT program to achieve better results? Are there new opportunities for minority integration that future programming could address? 2.2 Is there a potential for greater AKT program collaboration with other USAID programs and/or other donors? 6 See meeting notes, AKT Evaluation Team In-brief with USAID/Kosovo, February 21, 2017. 7 See Modification 1 to Evaluation SOW. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 5 EVALUATION METHODS AND LIMITATIONS METHODS The evaluation was implemented between January 17 and July 14, 2017, in three phases: PHASE 1: EVALUATION PREPARATION AND DESIGN Upon contract award, the Evaluation Team (ET) conducted a team planning meeting, consultations with USAID/Kosovo and AKT implementing partners, a desk review of activity documentation and secondary data, sample design, instrument design, and field protocol development. The last three items were formalized in an Inception Report submitted to USAID. The evaluation design incorporated a utilization￾focused and mixed-methods approach with consideration given to ethnicity and gender. SI integrated quality assurance (QA) through all phases of design and implementation through its proven EQUI® (Evaluation Quality, Use, and Impact) approach, while integrating gender and social analysis at each stage. PHASE 2: DATA COLLECTION Primary data was collected through two rounds of data collection. In both rounds, the ET held in-briefs and out-briefs with USAID. The first round was based on multi-stage sampling, where the ET purposively selected 10 out of the 16 AKT municipalities using three geographic and ethnic clusters: Cluster I consisted of six Serb-majority municipalities in the South; Cluster II consisted of the six Albanian-majority municipalities; and Cluster III consisted of four Serb-majority municipalities in the North. From these municipalities, the team selected individuals and organizations across five priority respondent strata: IPs, central government officals, municipal government officials, village/community councils, and local businesses. Sampling frames were provided by IPs in the form of detailed beneficiary lists mapped against programmatic objectives and/or activities. In addition to IP-supplied lists, USAID/Kosovo recommended knowledgeable experts to be consulted. The final sampling plan included a municipal official from each municipality, and two to five community and/or civil society beneficiaries to address the USAID focus on Objectives 1 and 3. The ET selected business representatives in four municipalities: two in the North, one Serb-majority municipality in the South, and one Albanian-majority municipality. The ET also selected national-level officials from the ministries with selection based on their involvement in or knowledge of AKT. The second round was designed to deepen understanding of community-level implementation, dynamics and outcomes. The ET selected a sub-set of previously sampled municipalities to reflect (1) varying ethnic compositions, (2) North-South geographic dispersion, and (3) AKT implementation dimensions identified through first round interview notes. Within each municipality, the ET selected villages and attempted to speak with both Serb and Albanian community leaders in separate interviews. As detailed in Annex C, the ET completed the set of interviews in six villages. In addition to this core design element, second round data collection included four case studies in North Kosovo (three school-level quick impact interventions and one Service Improvement Action Plan intervention at the municipal level), and interviewed one informal women’s group representative per municipality. Across both rounds, the ET completed a total of 61 interviews with 107 respondents (80 male, 27 female) through key informant interviews (KIIs) and group interviews (GIs). The breakdown by respondent type is presented in Figure 3 below. The ET was careful to schedule data collection in locations where, and during times when, both women and men could participate. The detailed methodology and data collection tools are available in Annex B – Evaluation Design and Methodology. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 6 Figure 2: Data Collection Sites by Municipality Figure 3: Respondent Groups8 8 The graph shows 65 respondents in total; 4 of which were repeat interviews. Zubin Potoku / Zubin Potok Vushtrri / Vučitrn Obiliq / Obilić Prishtina / Priština Gračanica / Graçanicë Shtërpcë / Štrpce Zveçan / Zvečan Mitrovicë / Mitrovica Leposaviq / Leposavic Kllokot / Klokot Gijlan / Gnjilane USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 7 15 15 10 9 5 4 4 3 Community Municipal Government Civil Society AKT Other Central Government USAID Business Number of Interviewed Conducted PHASE 3: ANALYSIS Upon return from the field, the ET performed open coding on interview notes. Once a clear set of thematic codes emerged, the team closed-coded the 61 interview notes. In addition to overt focus on addressing evaluation questions, key analytical aspects included differential implementation and impacts across key respondent groups, geographic variance in implementation and efficacy, and differences between AKT-US and AKT-LS. This primary data was combined with activity documentation and secondary data to triangulate and deepen evaluation findings. The ET drafted a findings, conclusions and recommendations matrix, and integrated findings from the various data sources on an iterative basis. This matrix became the empirical basis of the evaluation report. LIMITATIONS The research design has four primary limitations. First, at the time of the interview, some respondents had not interacted with AKT for over a year. This elapsed time could lead to recall bias. Second, sampling frames were largely derived from IP beneficiary lists. While the ET used other sources to identify respondents, this primary sampling approach could have introduced some selection bias. Specifically, interview selection could have been positively biased toward individuals with more interaction with the activity, and overly represented male perspectives. The ET used a number of strategies to interview female beneficiaries; however, the gendered nature of community leadership and participation in Kosovo skewed participation, and the resultant sampling frames, toward males. Third, the socio-political context of ethnic relations in Kosovo is highly complex and charged. The team expected to encounter response bias, especially regarding underreporting tensions, and tried a number of strategies to solicit truthful responses. Respondents were provided options for meeting locations, and interviews were conducted in respondent mother-tongue whenever possible, with some interviews conducted solely by local team members. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 8 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 Evaluation Question 1 examines the effectiveness of AKT in relation to stated programmatic objectives formalized in AKT’s theory of change. The evaluation question was, “To what extent has the project been effective?” There were three sub-questions: a. Has the program achieved its intended results, especially under Objectives 1 and 3? Why or why not? b. What are AKT achievements in North Kosovo to date? What worked and what did not? c. How has the sustainability of USAID investments been addressed? As expressed in the evaluation sub-questions, USAID prioritized Objectives 1 and 3 over others, and requested particular focus on programming in North Kosovo, as well as questions of sustainability. This section will present findings for all sub-questions by programmatic objective. FINDINGS RELATED TO OBJECTIVE 1- IMPROVE AUTONOMOUS, HORIZONTAL COMMUNICATION BETWEEN: A) KOSOVO ALBANIAN AND NON-MAJORITY COMMUNITIES; AND B) THE GOK AND NON-MAJORITY COMMUNITIES Has the program achieved its intended results? Why or why not? The primary programmatic activities under Objective 1 in terms of significance—as defined by USAID, the implementers, and as a percentage of the budget—were small-scale infrastructure interventions implemented via community forum and the joint working groups. Respondents at the community and municipal levels were very satisfied with the professionalism, dedication, and efficacy of AKT staff. In most instances, community respondents identified the activity through the AKT field coordinator. The strength of these individuals was a key success factor in the activity’s ability to deliver. Additionally, because respondents tend to co-identify AKT and USAID, this goodwill was extended to the Agency as a whole. It was common for respondents to make note of the strong community-level engagement and contrast the USAID approach against other donors. A community leader noted that “USAID is different from other donors. For other projects, nobody came down to see us. AKT even went down into the woods, stepping into the mud” (Male Community Member, Štrpce/Shtërpcë). Another respondent noted that “I’m from (Josip Broz) Tito’s time. This was the first time there was an initiative that someone asked citizens what they want” (Male Community Member, Vushtrri/Vučitrn). PERFORMANCE MONITORING DATA From an output perspective, AKT was very successful in reaching, and even exceeding, performance targets. According to monitoring data, both AKT-US and AKT-LS met or exceed the majority of their performance targets. AKT-LS data indicate 3,360 direct beneficiaries and 99,471 indirect beneficiaries (targets = 2,500 and 90,000, respectively). AKT-US data register 24,276 community members as beneficiaries of small-scale infrastructure intervention and an additional 4,189 attendees at events designed to strengthen understanding and mitigating conflict between groups (targets = 21,266 and 1,240, respectively). Aside from a handful of indicators where actuals were just shy of targets, each mechanism underperformed on one indicator. AKT-US failed to meet a key satisfaction metric: “Percent of citizens in AKT's partner municipalities satisfied or highly satisfied with the work of local authorities.” AKT-LS registered less than half of the expected minority-oriented issues addressed by local USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 9 governments. In both instances, these outcomes were mediated by municipal performance, something that AKT supported but could not directly affect. Tables 2 and 3 detail Objective 1 performance monitoring data from AKT-LS and AKT-US, respectively. Table 2: AKT-LS Performance Under Objective 1 Objective 1: Improve autonomous, horizontal communication between (a) Kosovo Albanian and non-majority communities, and (b) the GoK and non-majority communities (IR 1.3.2) Performance Indicators Year 1 Year 2 Year 3 (Q1-Q2) Project Total (Y1 – Y3 Q2) Target Actual Target Actual Target Actual Target Actual 1.1 USG-assisted facilitated events geared towards strengthening understanding and mitigating conflict between groups 30 30 50 272 30 284 110 586 1.2a Number of direct beneficiaries from USG Assistance N/A N/A N/A 1841 N/A 1519 2500 3360 1.2b Number of indirect beneficiaries from USG Assistance 20,000 20,000 40,000 42,051 30,000 37,420 90,000 99,471 1.3 Jointly implemented minority - majority activities 30 30 30 110 20 45 80 185 1.4 Number of minority-oriented priority issues addressed by targeted local governments 4 4 4 4 10 0 18 8 1.5 Number of participants in USG￾funded programs supporting participation and inclusion of traditionally marginalized ethnic minority groups 100 254 200 163 100 6 400 423 1.6 Percentage of the minority population with confidence in key Government institutions N/A N/A 27.3% 32.1% 32.3% 0 32.3% 32.1% USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 10 Table 3: AKT-US Performance Under Objective 1 Objective 1: Improve autonomous, horizontal communication between (a) Kosovo Albanian and non-majority communities, and (b) the GoK and non-majority communities (IR 1.3.2) Performance Indicators Project Total (Y1 – Y3 Q2) Target Actual 1.1.1 Number of jointly implemented minority-majority activities 12 34 1.1.2 Number of minority-oriented priority issues addressed by targeted local governments 60 60 1.1.3 Number of community members directly benefitting from small-scale infrastructure and procurement projects 21,266 24,276 1.2.1 Number of advocacy initiatives led by non-governmental constituencies that protect minority rights, including cultural and historic rights of non-majority communities, with AKT assistance 20 255 1.2.2 Number of capacity building activities delivered to civic groups that protect and cultural and historic rights of non-majority communities 3 4 1.3.1 Number of complaints received by municipalities with regards to implementation of the Law on Languages 70 69 1.3.2 Number of complaints addressed by municipalities with regards to implementation of the Law on Languages 50 72 1.4.1 Number of USG facilitated events geared toward strengthening, understanding and mitigating conflict between groups 124 269 1.4.2 Number of people attending USG facilitated events geared toward strengthening, understanding and mitigating conflict between groups 1,240 4,189 1.4.3 Percent of citizens in AKT's partner municipalities satisfied or highly satisfied with the work of local authorities 55.90% 31.19% 1.5.1 Number of minorities, including women, trained in skills for obtaining public sector positions 610 824 1.6.1 Number of constituent groups working in partnership with AKT to deliver AKT activities 110 119 1.7.1 Number of minorities, including women, participating in local government activities with USAID AKT assistance 1,216 1,631 1.8.1 Number of community members involved in local development efforts 277 271 1.9.1 Percentage of community-based small-scale infrastructure and procurement projects with impact evaluated above average or higher by community members. 60.00% 81.40% 1.1.1 Number of jointly implemented minority-majority activities 12 34 1.1.2 Number of minority-oriented priority issues addressed by targeted local governments 60 60 1.1.3 Number of community members directly benefitting from small-scale infrastructure and procurement projects 21,266 24,276 PROGRESS TOWARDS IMPROVING AUTONOMOUS, HORIZONTAL COMMUNICATION BETWEEN ALBANIAN AND NON-MAJORITY COMMUNITIES Community Meetings Working through local coordinators, AKT-US organized community forums for the purpose of identifying community infrastructure needs and reaching consensus on priority interventions. These priorities were operationalized with the help of AKT and municipal representatives into competitive procurements and, ultimately, public works. The model relied on broad-based community participation to foster inter-ethnic (and as a secondary consideration, cross-gender) communication and cooperation. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 11 The activity ensured that households represented multiple ethnicities, and community forum participation required the inclusion of members of different ethnic groups, as well as women. Data from field interviews suggest the activity was successful in generating demand at the community level. Working through community leaders and village-level mobilization, interview data indicate initial community forums were well attended. However, in many instances, the lack of direct financial benefit to participants and the time required to participate in meetings resulted in significant attrition. One community leader noted that, although about fifty community members came to the first meeting, “as soon as they learned that there was no personal benefit, the majority left the meeting and about 10-15 people remained” (Male Community Member, Vushtrri/Vučitrn). In another instance, an Albanian respondent in a Serb-majority community commented that he attended one meeting and would have attended others, but was not able to attend more than one meeting due to work responsibilities. The one exception the ET found in equitable ethnic participation was in a Serb-majority community in which, according to the Serb community leader, AKT did not generate any Albanian participation. This community consists of 20 percent Albanian households, and though “there were numerous attempts to organize those meetings (by AKT) … in writing and in person… meetings were not attended by Albanians” (Male Community Member, Obiliq/Obilić). According to this respondent, no Albanians attended any of the community forum meetings. The ET tried to independently verify this information with Albanian representatives of the community. Contact was attempted through Serb community representatives, municipal representatives, Albanian representatives of surrounding communities, AKT field coordinators, and unannounced field visits. Exhausting all reasonable means, the team was unable to find any representatives willing to speak. The team did, however, ask Albanian representatives of surrounding communities why participation in this village was so low. According to one such informant, the Serb and Albanian communities in the village “get along very well” but the Albanian residents “are hardworking agricultural people. They don’t have time to meet” (Male Civil Society Representatives, Obiliq/Obilić). While it was impossible for the ET to independently verify this information, lack of participation may suggest a mismatch between scheduling of community forum meetings and agricultural obligations (either time of year or time of day). The Serb respondent from this surrounding community mentioned that meetings were generally held “early in the morning during working time.” In almost all interviews with community members, respondents described inter-ethnic relations in the community forums as cordial and equitable. In one case, community leaders representing different ethnic groups displayed strong cross-ethnic collaboration and reciprocity in a joint working group (JWG) meeting. After all village representatives presented their respective priorities, an Albanian leader voted to support another community’s proposal to build a road because “it’s an underdeveloped community” and the road “would connect Serbian and Albanian communities” (Male Community Member, Obiliq/Obilić). In turn, the Serb representative of the community proposing the road reciprocated and voted for the Albanian community’s school intervention. By way of counterpoint, there were two instances in which inter-ethnic tensions may have been partially exacerbated through community-level AKT programming.9 In the first case, “there were arguments between the two ethnicities over [JWG] priorities.” According to a community respondent, “over time and through discussion, a certain number of Albanians supported me and we resolved the issues.” (Male Community Member, Vushtrri/Vučitrn). This individual was, however, on the winning side of the debate. It was not possible for the ET to confirm that true consensus was reached, and that the Albanian leaders do not harbor any ill-will. The second case was reported by a 9 A third instance of exacerbated inter-ethnic tensions is reported in the subsequent section. In this case, the tension emanated from the municipality and how it handled the naming of an infrastructure project. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 12 Serb community member who stated that a sports facility was built “only for the Serb community. It was only made for Serbs.” (Male Community Member, Obiliq/Obilić). According to this individual, members of the Serb community tried unsuccessfully to direct a second intervention in an inequitable manner: “We tried to make the second project for Albanians too. They wanted to make it for Serbs but we insisted to do something for the Albanians as well.” If these statements represent broader community opinion, it is likely that Albanian residents’ opinion of Serb neighbors were adversely affected. Overall, the priority activity selection mechanism was democratic. In communities with non-equal ethnic representation, this process could lead to inequitable gains by different community groups. The ET probed this phenomenon in all interviews, and, with the minor exception noted above, did not detect any such occurrences. The process of holding community forum meetings for the purpose of identifying community infrastructure needs was overall successful. Infrastructure Procurement A number of respondents expressed frustrations with the manner in which community priorities became operationalized into infrastructure procurement. However, because this feedback focuses on the role of the municipality in this process, the findings are presented in the next report section. These dissatisfactions notwithstanding, community respondents were happy with the ultimate AKT infrastructure investments. Respondents were pleased with the technical quality of the infrastructure, and there was broad consensus that these interventions helped the community in broadly equitable terms. Inter-ethnic Cooperation and Interaction The AKT activity goal was to “increase constructive inter-ethnic cooperation and interaction – particularly between Kosovo Albanian majority and Kosovo Serb non-majority population.” 10 AKT clearly galvanized short￾term communication and cooperation between ethnic groups in structured activities. There was some evidence that community-level impacts materialized through three different causal pathways. First, by virtue of working together toward a common goal in a structured environment, interpersonal bonds were created and strengthened between community members. One Albanian representative from a Serb-majority community stated, “there was a positive impact. Some people in our community had no communication. Because of the project they started relationships with Serbs” (Male Community Respondent, Gračanica/Graçanicë). A more expansive example of this phenomenon came from an Albanian community leader from a Serb-majority community, who stated that AKT “create(d) intercommunity relationships. We exchanged contact details. Exchanged phone numbers. We meet from time to time. We had dinner recently. I met new people: some Serb, some Ashkali. It erased barriers of misunderstanding between communities” (Male Community Respondent, Gračanica/Graçanicë). Second, multiple respondents brought up use of common spaces by different ethnic groups, particularly as they related to child and youth play: “The project playground brought the groups closer…Serb/Albanian kids did not hang around together before. Now spontaneously they are able to meet in one place and play joint games… now mothers spontaneously exchange a couple of words while watching their kids play” (Male Community Respondent, Vushtrri/Vučitrn). Third, some respondents reported low-level improvements in community relations. For example, one community leader stated, “AKT relaxed people. Now we’re all more open to hang around and communicate with each other” (Male Community Respondent, Obiliq/Obilić). These sentiments, however, were outweighed by community responses indicating a lack of belief in lasting, significant impacts on inter-ethnic integration. Broadly-speaking, these challenges to the program theory of change resulted from two types of responses. First, respondents claimed that inter-ethnic 10 AKT-US Annual PMP February 2016 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 13 cooperation was positive before the activity came, and, as such, was not affected any substantive way by AKT. For example, with regards to a playground being built, an Albanian respondent from a Serb￾majority community commented that children of different ethnicities played together prior to AKT intervention, and that the playground only “improved the quality of the playing” by providing better play equipment. A respondent from a different community similarly stated, “The project did not impact community relationships in any substantive sense because communication / cooperation existed already and it was a small project” (Male Community Respondent, Vushtrri/Vučitrn), and yet another stated, “I don’t see any changes in interactions between communities, because also before we used to get along with Albanians, but maybe the quality of playing and education of kids is better and that makes parents happier” (Male Community Respondent, Obiliq/Obilić). Second, the intensity and duration of the intervention were not sufficient to affect such high-level and culturally-entrenched outcomes. Running contrary to the activity goal, the ET found two cases where the activity may have led to increased inter-ethnic tension. In one case, a community collaboratively decided to prioritize the construction of a road serving Serb, Albanian, Roma, and Ashkali families. However, once the municipality took over formal planning, the road was given the name “King Aleksander,” a reference to a historical Serb king that could be viewed negatively by Albanians. At the time of fieldwork, a formal complaint about the name had been lodged, and, while the respondent was optimistic about a change, the issue remained unresolved.11 The second instance revolved around a road construction intervention in a Serb-majority community that was cancelled by AKT. At the time of the community forum creation, there was only one Albanian household in the community. In the intervening years, a number of Albanian families moved into the village. While AKT representatives reported informing the Serb community member multiple times about the cancellation, the individual was under the impression that the construction was forthcoming. This respondent reported that Albanian community members were wondering if the cancellation of the intervention was due to their arrival.12 While these cases are worthy of attention, it is important to note that they are outliers in a process that solicited largely equitable participation from community members of different ethnic groups (at least those that stayed after the first meeting). PROGESS TOWARDS IMPROVING AUTONOMOUS, HORIZONTAL COMMUNICATION BETWEEN THE GOK AND NON-MAJORITY COMMUNITIES Community to GoK Interactions AKT worked through municipal officials in all aspects of programming. In addition to municipal cost￾sharing, municipal representatives managed procurement for the infrastructure interventions. Municipal representatives participated in JWGs and trainings, and led the public budget hearings. These activities were intended to facilitate structured interaction between municipal officials and communities. When asked about community-municipal relations before AKT, the prevailing sentiment across community respondents was that there was little or no interaction. A representative statement from a village leader read, “before this project, the municipality never came to this village” (Male Community Respondent, Štrpce/Shtërpcë). The general sentiment of respondents was that municipalities had limited budgets and could not offer much to villages. With nothing to offer the villages, representatives did not 11 Subsequent report sections provide more detail on this story. 12 While it is not possible to independently determine the perceptions of the new Albanian residents, it is possible that this miscommunication could have exacerbated tensions. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 14 make many visits. Even in more affluent municipalities, respondents reported a generalized lack of engagement from local government. Increased interaction between community and municipal representatives in JWG meetings and trainings did foster some goodwill on the part of the community. Some community respondents noted that AKT interventions were the only ones that prompted municipal officials to visit the communities, and the respondents appreciated the municipal officials’ presence. Other community respondents spoke in positive terms about the budget hearings. A number of respondents spoke about personal connections they made with municipal officials, and how these connections have assisted them personally or as representatives of their communities. At one end of this spectrum are instances where village leaders were able to obtain information, as in the case of a leader who said, “I was introduced to other people working for the municipality. I had the opportunity to discuss land titling/formalization and met with director of urbanism, where I was able to find out the future developmental plans for this community” (Male Community Respondent, Vushtrri/Vučitrn). Another poignant example of how AKT-facilitated relationships could plausibly lead to a substantive change came from a Serb-majority village in a Serb-majority municipality. As mentioned in the previous section, this municipality decided to name the AKT-supported road “King Aleksander.” Having taken offense at this action, the Albanian leader wrote and delivered a formal complaint letter to the municipality. In his words, “The AKT training gave me the skills to write this document. When I went to the municipality, the officer that received me was in my AKT training. I cannot be sure but I think he treated me better because of the shared experience. I think he also knew that I knew regulations because of the training and he did not want to cause trouble. He told me that it was a political period (elections) but that he promises he will fix the issue in the future.” When asked whether he would have behaved in this manner absent of the activity, the respondent replied that he would not have. While there were certainly cases of improved relations between communities and municipalities, there were a number of instances where the interaction led to either no change or a worsening of perceptions toward local government. Asked to describe the relationship between the community and the municipality, one community respondent from the minority ethnic groups reported, “The relationship with the community is catastrophic… Municipality only co-funded the road because the road connects the regional road to 10 Albanian majority communities. Through project activities, we actually realized how much they were unsupportive to us.” (Male Community Respondent, Vushtrri/Vučitrn). A similar sentiment came from another village representative: “Cooperation with the municipality was not there. If you compare cooperation before and after the project, it made our goals even harder” (Male Community Respondent, Obiliq/Obilić). Even in instances where community-municipality interaction increased in a substantive manner during the implementation of AKT, the prevailing sentiment among community respondents was that engagement will cease once external funds are exhausted. Some respondents expressed this opinion in terms sympathetic to fiscal challenges experience by the municipality. Others voiced their displeasure with municipal leadership in stronger terms. When asked about the impact of AKT on community￾municipal relations, one respondent stated, “No effect. I don’t trust the municipality, they’re thieves. Besides investment in sewer construction, they haven’t done anything. I was not informed or invited to any public hearings” (Male Community Respondent, Vushtrri/Vučitrn). The ET observed that respondents from more affluent municipalities (e.g. Gračanica/Graçanicë) tended to have fewer issues than resource-constrained municipalities. The ET also observed that respondents’ reports about municipal officials were highly influenced by the personal leadership styles of these officials (especially the mayor) and community leaders. Multiple community respondents made observations USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 15 about how relations with municipal officials improved with personnel changes from the mayor’s office down to administrative levels. The ET also observed that ethno-historic considerations continue to play a role in respondent reports. In geographies where historical ethnic tensions were more pervasive, community-municipal cooperation was constrained (if not antagonistic). Municipal Management of Infrastructure Procurement Multiple respondents expressed dissatisfaction with the manner in which community priorities were operationalized into the procurement process. Complaints were aimed at the perceived lack of transparency in this process. One respondent stated, “I have to say that I wasn’t satisfied. The municipality took over the technical documentation development…. And no information about work progress was shared with the community forum members. This was not alright, as it turned out that we were not treated equally” (Male Community Respondent, Obiliq/Obilić). Community frustrations seem to stem from two sources. First, having been engaged in a highly participatory manner by AKT, community leaders felt slighted by perceived secrecy on the part of the municipality. A community leader from another village reported that a municipal official told him, “who are you to tell us what should be done?,” and “you should just be there to participate” (Male Community Respondent, Obiliq/Obilić). The second reason why this shift in community participation rankled village representatives was their hope that infrastructure interventions might create employment opportunities for local people. The sentiment was summarized by one community leader: “the community was not involved (in the procurement process) … the agreement between us and the project was, we would get involved so that during construction of the road maybe people can get engaged in it, but we were not involved as community forum at all” (Male Community Respondent, Obiliq/Obilić). While it is possible that excluding the community from procurement is beneficial for efficiency, compliance, and conflict of interest purposes, AKT could have done a better job communicating the extent of community participation through the process. ACHIEVEMENTS IN NORTH KOSOVO IN IMPROVING AUTONOMOUS, HORIZONTAL COMMUNICATION The AKT model was modified for the North Kosovo context. In lieu of small-scale infrastructure investments, the activity completed “quick impact” interventions. The intent was similar, though the prioritization process and procurement thresholds were more constrained. According to activity documentation, AKT-LS had initiated 12 and AKT-US had completed 28 quick impact interventions at the time of this evaluation. The ET interviewed four stakeholder or stakeholder groups across three schools (two in Serb-majority communities, and one in an Albanian-majority community) aligned with this programmatic goal in two northern municipalities. The process in schools was for the school director, teachers, and parents to agree on priorities. In the two Serb schools supported by AKT-LS, the team found evidence of a robust consensus-building process. All three stakeholder groups agreed on school needs and worked collaboratively to operationalize them. In both cases, priority lists were translated into competitive procurements, and school officials were waiting for delivery. In contrast, the school in the Albanian community was visually less resourced than the other two, and engagement with AKT was minimal. According to the community-level stakeholder interviewed, AKT only worked with this individual himself (as opposed to a consensus-driven process with multiple community members). AKT declined to provide the individual’s request for school books for the school, and instead procured gym equipment and maps. This respondent stated that he was happy that the activity helped provide resources for the school, but that the AKT activity had no effect on inter-ethnic collaboration (since AKT only worked with one individual), nor did the activity change this individual’s perception of the municipality or its representative who visited the school. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 16 The ET found three other areas of concerns outside of the contrasting experiences between the Serb and Albanian schools. First, when asked about the process by which schools were selected for participation in the activity, a representative of AKT’s local implementer stated that the primary eligibility criterion was need. However, it came to the attention of the ET that this individual had children enrolled in both of the selected schools in that area. This would not ordinarily be cause for concern; however, these schools seemed to be fairly well-resourced. One of them was a short walk from downtown Zubin Potok/Zubin Potok, and was primarily used by residents of the municipality’s largest population center. To further probe the status of these schools relative to other schools in the municipality, the ET identified and made a call to a representative of the municipal Directorate for Education, Culture, Youth and Sport. When asked about which schools were most in need of support, this individual mentioned two primary schools that were not supported by AKT, “it’s known already, primary school in Zupce, and for example a branch unit of ‘Petar Kocic’ primary school in the Bube village is in bad condition.” While the evidence base is not definitive, data suggest that there may have been an element of elite capture in the selection of recipient schools. If the ultimate goal of the activity in Northern Kosovo was to improve perceptions of municipal functioning, this potential misallocation of resources could cause a hardening of attitudes. Second, the ET found evidence of significant procurement delays. One of the schools has been waiting for one and a half years for relatively minor materials. It is the professional judgement of the ET that these delays are largely driven by the large volume of investments. Unlike in the South, where AKT funded relatively large infrastructure investments, these schools in the North were ordering a large number of small items. One of the schools requested 16 different things, many of which required separate procurement actions. These delays are likely to cause some frustration among beneficiary schools, and the administrative cost of processing so many small transactions burdens project staff and reduces overall efficiency. The third are of concern and potentially the most direct impediment to increasing communication and inter-ethnic cooperation is the ethnic homogeneity of all schools in Zubin Potok/Zubin Potok. As a result, all benefits from these investments are captured by a single ethnic group. Furthermore, given the disconnect in the support provided to the Serb-majority schools and the single Albanian majority school, the intervention runs the risk of perpetuating perceptions of inequality. SUSTAINABILITY IN IMPROVING AUTONOMOUS, HORIZONTAL COMMUNICATION Benefits stemming from physical investments funded by AKT, particularly small-scale infrastructure interventions, are likely to persist for years. These investments seem to have been well-constructed and well-utilized. Quick impact interventions in North Kosovo will also persist, though asset depreciation should be expected to be greater in school environments. Community mobilization through formalized village-level meetings seem very unlikely to persist beyond the life of activity. There was no evidence of any spontaneous, structured meetings not affiliated with AKT. In many communities, there have been no meetings held for over a year. A commonly expressed explanation for this phenomenon is the fact that “there has to be a purpose for bringing people formally together. For example, to discuss how to resolve infrastructure issues. We cannot solve problems on our own. We need external financing” (Male Community Respondent, Obiliq/Obilić). Stated in more direct terms, “There will be no gatherings without donor support” (Male Community Respondent, Vushtrri/Vučitrn). Similarly, it is highly unlikely that the gains in community-municipality communication affected by AKT will persist. While it is possible that some aspects of community outreach will continue, without USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 17 external financing, municipal governments will most likely revert to pre-intervention behavior. No lasting changes in municipal operating procedures were observed that were extraneous to AKT organizing. Lastly, the prospects for sustaining advances made in the overall programmatic objective of bringing majority and minority ethnic groups together are not promising. It is highly likely that community members will continue using infrastructure financed by AKT, and that this common use will facilitate some small-scale interactions (e.g. children playing together). It is also possible that interpersonal relationships formed through participation in the activity will persist. However, because the activity was not successful in substantively affecting community-level changes in ethnic cooperation, despite achieving and sometimes exceeding output targets, it is highly unlikely these benefits will be sustained. FINDINGS RELATED TO OBJECTIVE 3- ENHANCE CAPACITY OF MUNICIPAL ADMINISTRATIONS FOR STRATEGIC PLANNING AND TO DELIVER AND OVERSEE QUALITY SERVICE Programmatic activities AKT supported municipal governments through a number of complementary activities. In the first year of the activity, AKT facilitated self-assessments of municipal capacity for each of the 16 targeted municipalities. On the basis of these documents, municipalities drafted Capacity Development Plans (CDPs) to strengthen perceived weaknesses. In conjunction with this process, municipalities developed Service Improvement Action Plans (SIAPs) to identify and co-finance investments that would improve municipal service delivery. According to AKT, municipalities contributed between 20-40% of these SIAP costs. In addition to these core activities, AKT also provided a suite of trainings modeled on CDPs, facilitated public budget hearings, financed development/implementation of e-governance hardware and software (e.g. e-kiosks), and supported implementation of the Law on the Use of Languages (LUL), and Local Environmental Action Plans (LEAPs). The evaluation focused primarily on the training component. Effectiveness of AKT to enhance capacity of municipal administrations For Objective 3, AKT again performed well with respect to outputs. AKT-US (Table 4) reached its targets, and AKT-LS (Table 5) is on track to reach its targets for number of individuals trained and number of SIAPs developed. Both mechanisms also show positive results for indicators that suggest change in higher level outcomes related to an increase in citizen engagement, such as, “Number of citizens participating in municipal budget hearings in target municipalities” (AKT-US achieved its target, whereas AK￾LS is currently 22 individuals away from meeting its target as of Q2), and municipal capacity, “Increase in property tax collection in targeted municipalities” (AKT achieved its target). USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 18 Table 4: AKT-US Performance Under Objective 3 Objective 3: Increase the efficiency and capacity of target municipal administrations to respond to the needs of all their citizens (IR 1.3.1) Project Total (Y1 – Y3 Q2) Performance Indicators Target Actual 3.1.1 Number of individuals who received USG-assisted training, including management and fiscal management skills to strengthen local governments and/or decentralization 1,400 1,863 Male 1363 Female 500 3.1.2 Average percent of citizens in targeted municipalities satisfied or highly satisfied with select services 60% 31.19% 3.1.3 Number of people receiving improved services derived from AKT-supported SIAP projects 38,640 30,267 3.2.1 Number of targeted municipalities that develop Service Improvement Action Plans (SIAPs) with AKT support 16 16 3.2.2 Number of project implemented in SIAPs 9 24 3.3.1 Percent increase in the number of youth and women included in municipal budget hearings in target municipalities 15% 73.89% 3.4.1 Increase in property tax collection in targeted municipalities 25% 41.40% 3.5.1 Number of actions taken as result of meetings between Municipal Offices of Communities and Returns and senior municipal leaders 24 24 3.5.2 Number of AKT-supported initiatives involving MOCRs in local decision￾making on policies and programs targeting local minorities and returnees 87 91 3.6.1 Percentage of Municipal Assembly Acts posted on municipal websites 64% 96.14% 3.7.1 Number of citizens participating in municipal budget hearings in target municipalities 4,355 5,176 Male 4196 Female 980 Albanian 2950 Serbian 2161 Bosnian 19 RAE 43 Turk 3 3.8.1 Number of individuals who are not civil servants and are involved in setting development priorities for municipal service delivery 42 91 Male 68 Female 23 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 19 Objective 3: Increase the efficiency and capacity of target municipal administrations to respond to the needs of all their citizens (IR 1.3.1) Performance Indicators Year 1 Year 2 Year 3 (Q1-Q2) Project Total (Y1 – Y3 Q2) Target Actual Target Actual Target Actual Target Actual 3.1 Number of sub-national government entities receiving USG assistance to improve their performance (cumulative) 12 16 16 16 16 0 16 16 3.2 Percent citizens in targeted municipalities satisfied or highly satisfied with select services N/A N/A 30% 31.20% 35.00% 0 35% 31.20% 3.3 Number of people who receive USG-assisted trainings, including 50 150 80 295 50 45 280 490 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 20 Table 5: AKT-LS Performance Under Objective 3 SIAPs According to AKT documentation, AKT-US supported the development of customized CDPs and SIAPs in all 16 targeted municipalities.13 In keeping with its performance target, AKT-US completed nine SIAP interventions by activity closeout. At the time of writing, AKT-LS had completed three interventions, with another four in process.14 SIAPs were completed in all 16 targeted municipalities, and 9 of the interventions were completed at the time of the evaluation.15 These plans included a variety of interventions supporting energy efficiency, public lighting, waste management, primary health care, education (university and preschool), and cultural centers. SIAP implementation carried a minimum municipal cost-share of 15 percent. According to AKT-LS staff, the aggregate cost-share requirement of USD 600 thousand was exceeded. Performance monitoring data indicate that an estimated 30,267 people are receiving improved service as a result of these interventions. Almost 80 percent of these estimated beneficiaries are Serb. Municipal officials interviewed by the ET deemed the development of SIAPs a positive step in improving service delivery and for planning economic development. For most, it was a first-time exercise that entailed hard work and gave participants confidence to initiate change. A participant in the development of a SIAP in Leposavić/Leposaviq said, “It was [a] nice experience to develop an action plan. I already had some ideas that I hoped to implement with assistance from the community. This experience enhanced my hope. AKT helped develop a plan that can make an impact.” Finally, participation in SIAPs had positive spillovers in municipal abilities to collaborate with other international donors. One municipal respondent stated that the “SIAP helped us be prepared for working with three additional donors because we are prepared to continue our project work” (Male and Female Municipal Government Respondents, Gjilan/Gnjilane). Training AKT provided training to municipal workers on a wide range of topics, including: public procurement, strategic planning, human resource management, budgeting, project management, environment protection, public transparency, democratization, gender, and human rights. 13 AKT-US Year 2 Annual Report, p.47 14 AKT-LS Year 2 Annual Report, pp. 37-38. While the AKT-LS annual reports describe the work with youth, unlike AKT-US, AKT-LS does not have specific indicators for youth. The AKT-LS PIRS (June 2016) indicates that there should be disaggregation by age for some indicators, but the final list of indicators shared with the ET does not contain this disaggregation. 15 AKT-US PMP update. Unclear what AKT-LS has accomplished. management to Strengthen Local Government/or Decentralization 3.4 Number of citizens included in municipal budget hearings in targeted municipalities N/A N/A N/A 578 N/A 0 600 578 3.5 Number of developed SIAPs in target municipalities 2 2 3 2 3 3 8 7 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 21 According to the most up-to-date performance monitoring data the ET could acquire,16 between the two mechanisms, AKT trained a total of 2,353 people with the goal of strengthening local government. This figure represents a 38 percent overachievement relative to the performance target of 1,680. More than three-quarters of this achievement was reported by AKT-LS. Sex and ethnic disaggregations are presented for AKT-US below (the team did not have access to current, disaggregated data for AKT-LS). Table 6: Sex and Ethnic Disaggregation AKT-US Field interviews with municipal representatives who attended these trainings suggested a significant divergence in satisfaction. Positive feedback included one respondent attributing improved service delivery to AKT capacity building activities. This person stated that the “trainings on democratization, project cycle management, procurement, budgeting, etc. increased the quality of services we provide to all citizens.” (Male and Female Municipal Government Respondents, Gjilan/Gnjilane). In Zvečan/Zveçan, officials reported that the municipality had been operating without human resources personnel until one individual was trained by AKT-US. Municipal officials also appreciated informal capacity building of their staff as a byproduct of their cooperation with AKT. For example, in Vushtrri/Vučitrn, a respondent said that AKT taught “out-of-the-box” thinking and transparent and quality procurement procedures, and this person requested continued support. An indicator of satisfaction was the number of requests for additional trainings. One respondent highlighted the “need for further capacity building activities namely in dealing with public funds: e-procurement management, budget planning and technical evaluation of projects” (Male Municipal Government Respondent, Vushtrri/Vučitrn). Feedback was not uniformly favorable, however. Criticisms of the AKT capacity building approach centered on four causal factors. A number of respondents spoke of heavy workloads that inhibited the ability of municipal workers to participate, such as, “many trainings were refused by the municipal officers due to high pressure at work” (Male Municipal Government Respondent, Obiliq/Obilić). Secondly, some respondents spoke of a generalized training fatigue among municipal officials. One individual noted that donors have been providing training to government functionaries for “6-7 years…sometimes people are tired of being trained. One central government representative had 10 simultaneous requests for training by different donors and didn’t know which [would be] most effective” (Male Central Government Respondent, Prishtina/Priština). Third, a number of respondents expressed a perception that AKT trainings did not help them perform their jobs better. One individual stated, “the feedback I received is that trainings were not that useful…. as soon as colleagues see me they make the face ‘Is it another training again?’ I think it’s too much for them” (Male Municipal Government Respondent, Gračanica/Graçanicë). Finally, although few respondents mentioned translation issues for different trainings across objectives, the ET registered one complaint about an Albanian AKT trainer being sent to North Kosovo to train municipal officials. The respondent expressed dissatisfaction with the translation, stating “we needed trainers in the Serbian language to train people” (Female Municipal Government Respondent, Zvečan/Zveçan). 16 AKT-LS Quarterly Report (Year 3, Quarter 2) – April 2017; AKT-US Performance Management Plan – March 6, 2017 Total Percent Sex Male 1302 73% Female 474 27% Ethnicity Albanian 1072 60% Serbian 601 34% Bosnian 45 3% RAE 32 2% Turk 22 1% Other 6 0% USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 22 Interview data suggest that trainings most directly applicable to job functions were reviewed the highest, with multiple respondents praising a procurement training that led directly to certification. Factors associated with reduced satisfaction included lengthy multi-day trainings, particularly if these sessions were not viewed by participants as beneficial. The opportunity cost of leaving work for two to three days was significant, both for the trainee and the municipality. Public Budget Hearings A total of 5,176 citizens participated in public budget hearings. Of these, 57 percent were Albanian, 42 percent were Serb, and the rest were Roma, Ashkali and Egyptians (RAE), Bosnian and Turk. From a gender perspective, 81 percent of participants were male. AKT brought municipal officials into villages and increased participation of community members in public hearings. While some municipal representatives are optimistic that these hearings will continue (Male Municipal Government Respondent, Obiliq/Obilić), many community respondents were skeptical. E-Kiosk E-kiosk and translation software interventions were well received by respondents. Municipal officials and community members appreciated that e-kiosks serve all citizens (and communities) equally, are accessible at any time, and ease the administrative burden on the municipality for issuing public records. There were some issues in launching these services, with one respondent expressing frustration at initial delays. The ET observed two non-functioning e-kiosks in Gračanica/Graçanicë and North Mitrovica/Mitrovica e Veriut. Support for Implementation of the Law on the Use of Language AKT-US identified communication in different languages as a barrier for inter-ethnic cooperation, especially in the public sector. AKT-US cooperated with the Office of the Language Commissioner (OLC) and its municipal offices to provide broad support for LUL implementation. AKT-US also organized public debates in 12 southern municipalities with participation from 337 people from the communities, municipalities, and Civil Society Organizations (CSOs) to learn more about the OLC’s role, mandate, and work. 17 AKT-US worked with the municipalities to streamline the complaint system, and developed and launched an Android and iOS mobile application (“Language Officer”) to improve communication between OLC and citizens. The activity purchased and installed multilingual informative signs and door markings in 12 municipalities, and purchased and installed state-of-the-art translation software in all 16 municipalities. A high-ranking individual in the OLC stated that, after a “difficult 6-month period” in which AKT-US and the OLC disagreed about the need for international experts, support improved and AKT helped with three “successful” interventions. In 2015, AKT led an exercise to define roles and responsibilities. The product of this engagement was a joint handbook for the municipalities on managing language issues in local institutions that broke down responsibilities of different offices into the very specific competencies. For example, the administrative department is responsible for the appearance of buildings, and buildings must make forms and signs available in all languages. The document was operationalized through a series of roundtables and the development of an action plan. The second engagement was a report detailing the experiences of “secret clients.” Representatives of minority groups were asked to approach their municipality and judge how services were provided. The respondent noted that this was “very useful,” stating that the document provided a “realistic picture of ground realities” (Central Government Respondent, Gračanica/Graçanicë). The third engagement was the complaint handling application (“app”). The application allows citizens to send photos of problems to the office. A respondent stated, “we are getting dozen photos every week now. It used to be just 50 per year.” While AKT support was well 17 AKT-US Year 2 Annual Report, p.10 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 23 received, the language problem remains pervasive. In the words of one respondent, “[we] have better cooperation with municipalities now. No municipality doing well on implementing Law on Languages. But improving.” Achievements in North Kosovo Northern municipalities face unique political dynamics and staffing shortages, and participated less than other municipalities in AKT programming. Respondents from these municipalities identified more pronounced capacity building needs, including difficulty filling job vacancies, training newly-hired and seasoned staff, learning and adjusting to Kosovo law, and building more effective relations with central institutions. Municipal officials noted that due to staffing gaps, the municipality could not always participate in trainings that were offered. One respondent describing the interlinked challenges in the North: “Problems have piled up over time. We have problems with the central authorities as well. […] We do not have property documents and we do not know [the] property changes in the previous 15–20 years. As a result, we cannot issue licenses and permits. Furthermore, use of different license plates complicates transportation. Public transport is not functional. Salaries are still not distributed from Kosovo budget.” (Male Municipal Government Respondent, North Mitrovica/Mitrovica e Veriut) Another informant described the need for enhanced budget management skills in the North: “Kosovo local government is not working [at] its full capacity. Local government has been established, but there [are] still mixed competencies with the parallel system. The municipality did not have the capacity to spend all funding allocated for municipal development. As result, 40 percent of funds were returned to Prishtina/Priština.” (Male Civil Society Respondent, Leposavić/Leposaviq) While the evaluation could not definitively validate this information, the ET identified one media report that indicated that just one of four municipalities in the North (Zubin Potok/Zubin Potok) realized all its capital investment projects planned for 2015. The report identifies delays in signing the memoranda of understanding between the Kosovo ministries and the mayors of four Northern municipalities as a primary reason for the planned interventions not being completed.18 From an output perspective, AKT was successful in implementing in the North. While only 15 percent of AKT-US training recipients were from the four Northern municipalities, all administrative units passed SIAPs, responsible for 69 percent of total AKT-US SIAP beneficiaries. See Tables 4-5 above. The ET investigated one SIAP intervention in North Kosovo. The municipal assembly member and SIAP coordinator was not clear about basic process facts. For example, the individual could not recall the names, positions, or affiliations of civil sector SIAP members. Additionally, the respondent’s recollection of a data collection effort intended to inform plan development was unclear and, at times, contradictory. Plan development was conducted in conjunction with another municipality, both focusing on the area of energy efficiency. The SIAP proposed the replacement of a boiler to improve efficiency of heating the municipal building and an adjacent cultural center. The work was completed to the satisfaction of 18 Sanja Sovrlic, January 7, 2016, Kosovo’s three northern municipalities implement only half of planned projects, accessed from http://prishtinainsight.com/three-Northern-municipalities-implement-half-planned-municipal-projects/, Accessed on February 21, 2017 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 24 representatives of both structures. While the process of consensus building, procurement preparation, and management may have increased competencies of those involved, the interview with the coordinator in this one municipality called into question the organization of this process. It could plausibly be claimed that more comfortable municipal workers are better able to serve their constituents, although the primary benefits of this activity are likely to be increased energy efficiency of the municipal building and improved comfort of municipal staff. According to a cultural center staff member, the improved heating has increased the ability of staff to use the facility: “before we could not use and work in some premises and now we can…. For example, one room in multimedia programs was not active and now it is active.” Additionally, this individual reported a decrease in the use of electric heaters. While this investment has improved the ability of the municipality to deliver cultural programming, it is important to note that only ethnic Serbs use the facility. In the words of the respondent, “[Albanians)] did not use, nor do I believe, they use today our services” (Municipal Government Representative, Zubin Potok/Zubin Potok). There was some evidence of increased linkages between Northern municipalities and the national government. In the words of one municipal representative, “I think that project assisted a lot with connecting us with Prishtina/Priština and now we have access to different services in Prishtina/Priština, for Municipality this was the most important. I think that systemic assistance was the most beneficial.” Sustainability The physical investments made through SIAPs should be expected to persist for years and provide services to municipal residents. The plans themselves were designed to be actionable for four years, and the engagement of community forum members to monitor activities could strengthen demand-side pressures. If municipal officials continue to follow through on them, the design feature should promote sustainability of capacitation efforts. However, many municipal respondents expressed skepticism that the momentum will continue absent AKT. One informant working with implementation of the LUL stated that while progress was made, we “absolutely cannot continue without external support. [We’re] facing enormous issues. Only 6 people in the office with 150k EUR annual budget. Need expertise, need support. [We] don’t have the financial resources to implement what is needed.” A community respondent from a minority group in North Kosovo stated that “without the USAID or some foreign actor such project will be difficult to continue, because local NGOs and public institutions neglect our village.” Because municipalities are operating with limited funds, their ability to serve their constituencies is inhibited. In this context, capacity building can only do so much toward improving service provision. Lastly, and specific to the North Kosovo context, the sustainability of gains in Northern municipalities is further complicated by unresolved political disputes between Prishtina/Priština and Belgrade. This part of the country still receives substantial support from Belgrade. In the words of one informant from the region, “sustainability depends on political stability.” FINDINGS RELATED TO OBJECTIVE 2 – IMPROVED ECONOMIC OPPORTUNITIES IN TARGET MUNICIPALITIES Both AKT-US and AKT-LS facilitated trade fairs and Business to Business (B2B) meetings aimed at facilitating market linkages between majority and non-majority business, and conducted trainings. AKT- USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 25 US’s focus also included providing training and other technical assistance to youth- and women-owned businesses. AKT-LS issued business grants to boost private sector performance.19 Performance monitoring data Monitoring data from the Performance Management Plans (PMPs) show that AKT-US met or exceeded its life of project (LOP) targets for 9 out of 11 indicators for Objective 2, 20 implying achievement of the majority of AKT-US’s expected outcomes related to increasing market linkages and employability for non-majority groups. AKT-LS similarly shows a positive trend in project achievement. For Objective 2, AKT-LS met the targets for three out of three indicators in Year 1, two out of three indicators for Year 2, and has already met or exceeded its LOP targets for two out of its four indicators. 21 The indicators with LOP targets not yet met include “Indicator 2.2: The value of sales of minority-owned small and medium enterprises (MSMEs) that are assisted with USG funds,” and “Indicator 2.3: Full-time and full-time equivalent (FTE) jobs created as a result of USAID assistance.” The results for the latter indicator (50 percent achievement thus far; Target: 220; Actual to date: 102) are not surprising given that targets for AKT-LS were roughly four times that of AKT-US. According to AKT-US, the focus was more on providing in￾depth training to MSMEs than on creating jobs; though AKT-US did achieve its target for FTE jobs created, similar to AKT-LS, it did not achieve its target for value of sales. Should this also hold true for AKT-LS, it would be expected that job growth would be limited for AKT-LS. Overall, the limited achievement with regards to both indicators suggests that although AKT succeeded in providing support to MSMEs, the activity did not grow these MSMEs in a substantive way, allowing them to hire significantly more numbers of staff, or otherwise increase their sales. See Tables 7-8. Table 7: AKT-US Performance under Objective 2 Objective 2: Improve economic opportunities in target municipalities (IR 1.3.3) Project Total (Y1 – Y3 Q2) Performance Indicators Target Actual 2.1.1 Value of sales of minority-owned SMEs that are assisted with USG funds 1,873,932 1,725,095 2.2.1 Full time equivalent (FTE) jobs created as a result of USAID assistance 60 74 2.3.1 Full time equivalent (FTE) jobs created and filled by youth as a result of USAID assistance 10 29 2.4.1 Number of new and existing youth owned businesses assisted with AKT support 30 38 Male 24 Female 14 2.4.2 Number of new and existing women owned businesses assisted with AKT support 77 84 2.5.1 Number of minority owned SMEs linked to majority markets and larger scale 12 26 19 Chemonics, “Advancing Kosovo Together Annual Report, Year 1.” 2015; Chemonics, “Advancing Kosovo Together Annual Report, Year 2.” 2016; CDF, “Advancing Kosovo Together-Local Solutions Annual Report, Year 1.” 2015; CDF, “Advancing Kosovo Together-Local Solutions Annual Report, Year 2.” 2016; 20 One indicator, “Number of new minority member firms in private business associations as a result of AKT training,” had a target of 40 firms and reached 39 firms. Given the small difference between the two figures, the evaluation team is considering the target as “met” for the purposes of this evaluation. 21 AKT-LS officially has four indicators for Objective 4. However, “Indicator: The value of sales of minority-owned MSMEs that are assisted with USG funds” did not have a target for Years 1 and 2, which is why the evaluation is only counting a total of three indicators for Years 1 and 2. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 26 firms through AKT activities 2.6.1 New and existing non-majority-owned businesses assisted with AKT support 183 222 Male 93 Female 129 2.7.1 Number of SMEs receiving business development services as a result of USG assistance. (KRA 2 Outcome 6) 155 226 Male 131 Female 95 2.7.2 Number of MSMEs receiving loans as a result of USG assistance 1 0 2.8.1 Number of new non-majority owned member firms in private business associations as a result of AKT training 40 44 2.9.1 Individual returnees and displaced persons receiving AKT support to income generating activities 3 18 Male 15 Female 3 Table 8: AKT-LS Performance under Objective 2 Objective 2: Improve economic opportunities in target municipalities (IR 1.3.3) Performance Indicators Year 1 Year 2 Year 3 (Q1-Q2) Project Total (Y1 – Y3 Q2) Target Actual Target Actual Target Actual Target Actual 2.1 Number of MSMEs supported by USG enterprise assistance 70 80 80 119 70 88 220 287 2.2 The value of sales of minority￾owned MSMEs that are assisted with USG funds N/A N/A N/A $5,490, 389 N/A 0 $7,400,0 00 $5,490, 389 2.3 Full-time and full-time equivalent (FTE) jobs created as a result of USAID assistance 70 80 80 22 70 0 220 102 2.4 Number of minority MSMEs linked to majority markets and 10 10 10 15 5 0 25 25 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 27 Support to Business Grantees Business grantee respondents stated that participation in AKT interventions allowed them to develop essential business skills (e.g. marketing skills, management skills, how to create a business development plan, and how to write project proposals). They stated that they did not possess these skills before. Grantees also affirmed that they used AKT grants to purchase equipment (e.g. greenhouses, freezers, cutting machines for furniture production) to expand or improve their business operations and production quality. All business grantee respondents expressed satisfaction with the equipment received. According to one grantee in Gračanica/Graçanicë, “buyers see us as more professional” due to the AKT supported procurement of higher quality equipment, and the equipment also helps businesses expand production. This grantee also stated that the professionalism of the business had an inter-ethnic dimension, in that “[Buyers] pay attention to quality more than origin of the product.” Factors for why AKT interventions under Objective 2 appear successful include the quality of AKT support, perceived fairness of AKT, a good process for selecting grantees, the cost-share requirement for the grants, and, for AKT-US, the lack of job creation pressures compared to other interventions. Business grantees remarked on the good quality of the training received, and appreciated the follow-up from AKT during grant implementation. A business grantee from Gračanica/Graçanicë stated, “[AKT] visited us several occasions. There were there to help us.” Both IP and business grantees remarked that AKT’s process for advertising grants was fair in its advertisement and selection of grantees, with AKT-LS organizing field visits and information sessions about how the application process worked. According to one grantee, “CDF was very fair. We didn’t have to have any connections with anyone to get the grant.” Multiple grantees believed that AKT did a good job in selecting grantees, using mechanisms like requiring grantees to interview and “prove their professionalism,” as stated by one grantee, and then having grantees cost-share the investment, which participants felt effectively weeded out potential grantees who might misuse the funds. Lastly, AKT-US had a lower target for job creation than both AKT-LS and the USAID Empower activity. AKT-US staff felt that this lower “pressure” on job creation enabled them to focus more on inclusion, relationship building, and providing in-depth support. Respondents mentioned four factors for limiting activity effectiveness under Objective 2. First, multiple grantees remarked that the amount of the grants was helpful, but too small to make any serious investments that would substantially grow businesses to create new jobs and employ people from non￾majority ethnic groups. Second, a business grantee in Leposavić/Leposaviq stated that AKT staff did not give enough advanced noticed before making site visits (only one day in advance), although adequate notice was given for any meetings in Prishtina/Priština. This concerned the grantee because business demands and poor network coverage often inhibited the grantee from responding to AKT right away, and the grantee was concerned about giving a negative impression to AKT if there was not adequate time to prepare. From the IP side, both AKT-US and AKT-LS staff noted that the different requirements per municipality (i.e. different cost-share requirements and grantee selection criteria) sometimes created confusion among municipalities. Lastly, AKT-LS reported having initial challenges complying with environmental review checklist requirements related to business grants. Connections between businesses from majority and non-majority ethnicities From the IP perspective, one of the most successful outcomes under Objective 2 was the creation of inter-ethnic linkages, both formal and informal. Formally, AKT-US established approximately 24 signed contracts between Albanian-majority buyers and Serb-non-majority sellers. larger-scale firms as a result of USG assistance to the value chain USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 28 Some, but not all, business respondents had participated in the AKT supported B2B meetings, where they created informal connections through interactions with other business owners. Serb grantees who had participated confirmed that through participation in AKT supported events they were able to increase their visibility and make contacts with Albanian buyers for the first time. This occurred for Serb business participants from both Gračanica/Graçanicë and the northern municipality of Leposavić/Leposaviq. Participants varied in the market types and locations in which they were able to participate, and whether or not contacts actually led to cooperation. For example, a Serb grantee from Gračanica/Graçanicë was able to sell to an Albanian business in Gjilan/Gnjilane, but not able to penetrate the market in Prishtina/Priština. A grantee in the North reported making linkages with businesses in the southern municipalities as well as Prishtina/Priština, whereas another Serb grantee from a different northern municipality did not cooperate with any Albanians. Another northern grantee cooperated with Albanians, but only in the North. One grantee who could not penetrate Albanian markets explained that although he exchanged contacts with Albanian companies through AKT-organized B2B meetings, the Albanian managers hung up on the grantee when he called. This example suggests that informal contacts may not be enough to establish substantive linkages. North Kosovo AKT focused on connecting Serb businesses from North Kosovo to buyers in southern municipalities, and from an output perspective, it did well to create formal linkages that did not previously exist. In North Mitrovica/Mitrovica e Veriut municipality, AKT created a network of 25 women-owned businesses comprised of Albanian, Serb, and Turk participants, and facilitated the network registration with the Ministry of Public Administration. According to an IP respondent, AKT was the first activity to help northern businesses become official members of the Kosovo Chamber of Commerce. Sustainability According to IP respondents, the business contracts and the business memberships with the Kosovo Chamber of Commerce established through AKT are still in effect, and these relationships are important to being invited to future business meetings and for other donor opportunities. However, due to time restraints and evaluation focus on Objectives 1 and 3, the ET did not collect evidence from direct beneficiaries related to how a contract or membership with the Chamber is affecting their business. In terms of informal linkages and growth, although business grantee respondents noted that AKT events allowed them to make contacts with people of other ethnicities and from other municipalities, few grantees gave concrete examples of continuing to take advantage of those contacts. One Serb grantee from Zubin Potok/Zubin Potok said that he continued to sell to an Albanian company, but that he established that linkage before AKT. On a positive note, the grantee respondents all continued to operate their businesses. One grantee from Zubin Potok/Zubin Potok stated he was expanding his business using his own funds. IP respondents acknowledged during interviews that despite their achievements in putting on events and connecting businesses, business relationships often remain tenuous and personality-based. AKT also established linkages between municipalities and businesses, and these linkages could be threatened—or new linkages will need to be made, at least—if municipal officers turnover after the next election cycle. In summary, there is some evidence of businesses continuing to grow and take advantage of business connections after receiving AKT support, but relationships established remain vulnerable. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 29 FINDINGS RELATED TO OBJECTIVE 4 – BUILD THE CAPACITY OF HOST COUNTRY ORGANIZATIONS BY LEVERAGING LOCAL, CROSS-ETHNIC LEADERSHIP TO IMPLEMENT PROGRAM OBJECTIVES 1, 2, AND 3 The original activity design planned that AKT-US would provide technical capacity building support to AKT-LS through formal training and on-the-job support, providing templates for meeting USAID’s management and reporting requirements, creating joint implementation plans, and supporting ongoing coordination. AKT-US implemented an Organizational Capacity Assessment (OCA) process for AKT-LS in the second and third activity years.22 Both IPs reported that AKT-LS had more capacity than originally anticipated during activity design, which resulted in the two mechanisms adopting an effective partnership approach, rather than AKT-US assuming a capacity development role. OCA assessments and interviews with USAID, AKT-US, and AKT-LS staff unanimously reveal that CDF had high organizational capacity from the outset. USAID administered the first OCA with CDF in 2015, at which time CDF scored 3.72 out of 4, designating it from the beginning as an organization of moderate-strong capacity. 23 AKT-LS underwent a second OCA in Activity Year 2, this time administered by AKT-US using a modified version of the USAID administered OCA tool. CDF scored 3.47 (-7 percentage points) during this second round. Nonetheless, the second round score still put CDF within the moderate-high capacity range, and activity documentation is careful to mention that several factors could have influenced the decreased score, including “different facilitators; an improved understanding and experience with the tool; and timing, as CDF was in start-up for AKT LS… in February 2015 and they may not have been able to dedicate time to critically self-assess nor had the opportunity during AKT LS implementation to focus on improving performance across 47 organizational development areas.” 24 Table 9: AKT-US Performance under Objective 4 Objective 4 Project Total (Y1 – Y3 Q2) Performance Indicators Target Actual 4.1.1 Number of tools, trainings, and advisory services provided to CDF and sub￾awardees to improve their technical capacity 75 100 4.2.1 Average OCA score of CDF in human resource management, financial management, organizational management, and program management 3.77 3.73 4.2.2 Number of tools, trainings, and advisory services provided to CDF and sub￾awardees to improve delivery of grants 21 23 4.2.3 Number of tools, trainings, and advisory services provided to CDF and sub￾awardees to improve other operational areas 11 29 4.3.1 Number of jointly implemented AKT activities 53 57 4.4.1 Number of tools, trainings, and advisory services provided to CDF and sub￾awardees to improve reporting capacity 17 30 22 Chemonics, “Advancing Kosovo Together Annual Report, Year 1.” 2015; Chemonics, “Advancing Kosovo Together Annual Report, Year 2.” 2016 23 The USAID OCA Tool assesses seven areas of organizational capacity, including: Governance and legal structure; Financial management and internal control systems; Administration and procurement systems; Human resources systems; Program management; Project performance management; Organizational management and sustainability. 24 AKT-US Annual Report Year 2, p.71 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 30 4.5.1. Number of tools, trainings, and advisory services provided to CDF and sub￾awardees to advance their financial capacities 9 19 4.6.1 Number of linkages facilitated between AKT Local Solutions consortium and stakeholders 56 52 AKT-LS and AKT-US staff agree that the assumption that AKT-US would need to provide capacity development support to CDF proved false, and that as a result, much less needed to be done under Objective 4. Support was given in response to AKT-LS voicing specific requests (e.g. providing consultants for CDF). IP staff also commented that the change in the nature of the AKT-US to AKT-LS relationship freed AKT-US to re-allocate money from Objective 4 towards more critical activities under the other three objectives, which may explain why there was overachievement for some indicators in other Objectives. USAID staff confirmed the effectiveness of the partnership arrangement, and noted that USAID could see the improvement in the two IPs’ joint development of work plans after Year 1. Indeed, this stakeholder described the fluid coordination between AKT-US and AKT-LS as a chief outcome of the activity. The effective coordination was further recognized by a municipal official from the North during an interview. IP respondents also noted that USAID’s initial management of the two separate mechanisms challenged the ability of both IPs to effectively work together. AKT-US began six months prior to AKT-LS, and USAID did not adequately inform AKT-LS of AKT-US’s scope. This caused AKT-LS to propose duplicate activities that later required revision. This was eventually resolved. Even with its high OCA scores and good coordination with AKT-US, one municipal official from the South and two community members spoke about ways in which AKT-LS and its consortium could improve. The municipal official believed that AKT-US had more technical acumen than AKT-LS and was better at helping beneficiaries develop “out of the box solutions.” Community members said they preferred working with AKT-US because they were better at communication than AKT-LS (e.g. informing CF members if an intervention was approved). Community members also criticized local partner AKTIV as being “less professional” and not providing as much information to stakeholder as the staff from AKT-US. CONCLUSIONS OBJECTIVE 1  AKT was largely successful in fostering community participation from different ethnic groups. Representatives of these groups collaborated to identify community needs in a broadly equitable manner, and infrastructure improvements were delivered to the satisfaction of communities.  The transition from decision-making at the community forum level to procurement management at the municipal level was viewed by some community members as opaque, disenfranchising, and, in some instances, disrespectful.  There were indications of small-scale improvements in inter-ethnic relations, particularly with regard to strengthening/creating interpersonal relationships among participants, common usage of infrastructure funded by AKT (especially spaces for children and youth), and low-level trust￾building through the process of working on structured interventions. However, the intervention was not sufficiently intensive, neither in duration or contact time, to galvanize substantive community-level impacts. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 31  The activity succeeded in fostering communication between community and municipal representatives. Community perceptions of these interactions and their opinion of municipal leadership varied widely. Some respondents spoke of improved relations, some of worsened relations, and some of AKT not affecting any significant changes. Even in instances where cooperation improved, however, these changes were relatively small and are unlikely to affect community-level impacts on increased inter-ethnic cooperation.  Infrastructure investments will persist for many years after AKT closes out. However, improvements in community communication and collaboration should be expected to atrophy absent any future, structured engagements.  AKT’s interventions in North Kosovo were well-managed at the school level and will provide materials that improve the quality of education for participating schools. However, due to the fact that all schools in Zubin Potok/Zubin Potok are ethnically homogenous, these investments will not have any substantive impact on inter-ethnic communication or collaboration. Furthermore, because of some concerns around school selection and Serb-majority schools receiving far more aid than the Albanian-majority school, the activity runs the risk of exacerbating inter-ethnic and intra-ethnic tensions. OBJECTIVE 2  AKT’s approach at combining business grants with training and in-depth technical assistance was effective at building the competitiveness of non-majority businesses. Yet, the small sum of the grants limited the ability of some businesses to make substantive investments that may have led to further job creation or business expansion.  AKT was effective at establishing some formal, contractual relationships primarily between Serb (non-majority) sellers and Albanian (majority) buyers, and created opportunities (e.g. through B2B meetings) for these groups to informally make contacts among themselves. However, these relationships are still nascent, and sustainability is threatened without further third-party facilitation or formalization of the relationship. OBJECTIVE 3  AKT was largely successful in meeting its performance targets under Objective 3. All 16 targeted municipalities successfully drafted SIAPs, and at the time of writing all but four or these plans were implemented. The remaining four were in process.  The successful engagement and cost-sharing with the municipalities in excess of USD 600 thousand should be viewed as a success. While it was beyond the scope of this evaluation to independently verify beneficiary estimates, AKT claims these investments will benefit over 30,000 people.  AKT provided 2,311 people a wide variety of trainings. Some respondents reported deep satisfaction with these sessions, particularly when trainings directly affected job functions (e.g. procurement certification). However, the ET found evidence of trainings being underattended due to heavy workloads, training fatigue, and, in some cases, perceptions that offerings were not useful.  Overall, despite the improvements in municipal capacity cited above, most municipal officials themselves—plus central-level officials, community and business respondents, and other knowledgeable experts interviewed—noted that municipalities will need to improve service delivery to address citizen needs. OBJECTIVE 4 USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 32  AKT-US did not provide as much capacity development support to AKT-LS as originally envisioned during project design due to the latter’s already advanced organizational capacity. Rather, the more equal, joint partnership between the two mechanisms resulted in effective coordination and implementation.  Through its high OCA scores and ability to cooperate with international implementing partners and program USAID funds, AKT-LS shows signs of organizational sustainability after AKT-LS closes. Even so, the organization and its consortium partners still possess organizational gaps that could be strengthened when implementing future activities. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 33 EVALUATION QUESTION 2 What are the key lessons learned from the project to date? 2.1 Should there be any programmatic shifts to the AKT program to achieve better results? Are there new opportunities for minority integration that future programming could address? 2.2 Is there a potential for greater AKT program collaboration with other USAID programs and/or other donors? THEORY OF CHANGE The project theory of change posits that in order to achieve the programmatic objective of “Increased constructive inter-ethnic cooperation and interaction,” three IRs must first be achieved: (1) improved communication between ethnic communities, as well as between minority communities and the Government of Kosovo; (2) improved economic opportunities for non-majority communities, and (3) increased efficiency and capacity of municipal administrations to respond to citizen needs. This model makes intuitive sense from a logical, causal perspective. However, while AKT achieved progress against all three IRs, these gains were not sufficient to affect substantive and sustained change in community￾level outcomes according to qualitative data. This shortcoming is not a deficiency of either AKT mechanism. In fact, both activities achieved an impressive number of outputs, and exceeded many of their initial activity targets over their short periods of performance. The primary challenge in meeting activity objectives relates to project design. It is the professional opinion of the ET that the AKT mechanisms were spread too thin, both geographically, as well as with regard to the number of programmatic activities. Implementing a multimillion-dollar activity within three years is difficult, even without the coordination issues experienced in the first six months of AKT-LS implementation. As a result of this friction between resources and scope, implementation intensity was insufficient to affect the ambitious programmatic objectives. That being said, the activity was, in many ways, a success. Goodwill toward USAID and the American people has been strengthened among beneficiaries. High￾quality infrastructure has been delivered in the South, and high-quality school supplies will be delivered in the North. Many people’s lives will have been improved, some substantively, as a result of this intervention. Furthermore, while the activity did not achieve large-scale changes in inter-ethnic collaboration, many communities were positively affected. There was some evidence of small improvements in community relations, interpersonal relationships between ethnic groups, as well as between communities and municipal officials, and in people of different ethnic groups who are more likely to come into contact through the use of common infrastructure (particularly children and youth). CO-IMPLEMENTATION OF COMPLEMENTARY MECHANISMS Implementation of AKT through two mechanisms was a novel undertaking that had many positive aspects. The process, however, could have been strengthened by beginning with a more purposeful procurement. In an ideal scenario, both mechanisms would be designed in advance of procurement, with scopes clearly delineated (either geographically or sectorally), and unequivocal signals about cooperation sent during the Request for Application (RFA) period. SITE SELECTION With a stated programmatic objective of promoting inter-ethnic communication and collaboration, site selection for activity interventions is critical. Recognizing that the activity had a different approach in North Kosovo, the selection of ethnically homogenous schools for infrastructure/materials support is misaligned with the program theory of change. Similarly, the ET conducted fieldwork in one village that USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 34 was ethnically homogenous except for one minority household. While investments can improve inter￾ethnic relations in this context, the cost-efficiency of this programming is questionable in comparison to the many villages that were not served by the activity. Aside from the ethnic composition of potential programmatic sites, active participation from targeted ethnic groups should be a selection criterion for programmatic support. The village where no Albanians participated in AKT interventions should not have been allowed to participate in the activity. Given the democratic nature of priority selection, the most likely results of programming in these instances are that the participating group will benefit more from investments than the non-represented group. This, in turn, could plausibly exacerbate ethnic tensions. Lastly, the efficacy of community-level interventions to affect substantive and sustained outcomes depends largely on the community leaders. From field interviews, the perceived effectiveness of AKT interventions was strongly correlated with the quality of village leadership. Successful leaders are the product of a number of factors. First, they must have time to engage with work on behalf of their community. They must have an understanding that this time will not likely be compensated, and is done on a volunteer basis. Next, leaders must have the capacity to represent their community. This is partly a function of education, but also depends on interpersonal and consensus-building skills. Given the interface with municipal officials, these individuals must also have the ability to effectively communicate with government functionaries. Leaders should also be proactive in advocating for community interests. Given municipal financial constraints, the ability to lobby and represent the community independently is important for effective leadership independent of donor funding. Lastly, successful leaders should command the respect of their communities. It is difficult to screen people for these character traits. However, given the centrality of these individuals to the proper functioning of these types of activities, it may benefit USAID to consider individual-level capacity assessments and capacitation plans. PROCUREMENT As detailed in Evaluation Question 1, the ET found two issues related to procurement. Schools in North Kosovo were experiencing lengthy delays in receipt of requested materials. This stemmed primarily from the large volumes of relatively low-cost procurement actions. There is a natural limit on how quickly procurement can move, given USG regulations, the necessity of quality reviews to prevent malfeasance, and the inherent lag times for competitive bidding and bid review. Additionally, while large￾scale procurements take more time than small ones, there is a fixed labor cost for moving any action. When presented with multiple small procurements, the process can be slowed down very quickly. Absent hiring more personnel to manage procurement, the easiest solution is to either try to bundle goods, or to increase the average cost per procurement action. The transition between community need prioritization and infrastructure procurement created community-municipality tensions in a number of cases. These problems stemmed from beliefs on the part of village leaders that they were unfairly excluded from participation in procurement actions. Specifically, they bemoaned what they viewed as an opaque and disenfranchising process. It is beyond the scope of this evaluation to pass judgment on the appropriateness of municipal-led procurement, and the ET can see benefits to either approach. However, if the intent is for municipal officials to manage procurement actions in the future, the process and rationale should be well communicated to community members and leaders well in advance. Furthermore, given the sensitive transitional point, it would be beneficial to have active AKT participation and/or facilitation of this process. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 35 ELITE CAPTURE Activites relying on disbursement of goods or services through participatory means will always have to cope with stakeholders attempting to divert investments in personally beneficial means. On the extreme end of this spectrum is malfeasant, corrupt behavior that seeks to personally enrich individuals at the expense of others. There are, however, a range of behaviors that stop short of malfeasance that are, nonetheless, antagonistic to the aims of an activity, or that could create negative, unintended consequences. The evaluation did not find evidence of any overt corruption. The team did, however, document three potential cases of elite capture that are of note, if not for objective wrongdoing, then for possible perceptions of misappropriation or capture of USG funds. All these cases are documented under Evaluation Question 1, but are summarized here for the readers’ convenience. The first example revolves around the selection of schools for receipt of AKT support in Zubin Potok/Zubin Potok. While need was stated as the primary eligibility criteria for support by implementers, the two schools visited by the ET were both relatively well-endowed (e.g. one was in the municipal capital). Furthermore – and this may be an improbable coincidence – one of the implementing staff had both of their children enrolled in these two schools. When the ET contacted a municipal education official about which primary schools were in most need of support, the two answers did not map against AKT sites. It is not possible for the ET to definitively prove that selection of recipient schools was mediated by personal gain, but it is very possible that this perception could be held by residents of Zubin Potok/Zubin Potok. The second example was also from Zubin Potok/Zubin Potok, where municipal officials working on a SIAP were tasked with identifying priorities for AKT investment. While participants contacted public sector enterprises and schools, the ultimate decision was made to replace a boiler in the main municipal building. The auspices under which the activity was approved were that an adjacent community center would also benefit from the investment. However, the primary beneficiaries of this investment were municipal officials themselves. It is not possible for the ET to pass judgement on whether this investment was the best decision, but it is plausible to assume that community members knowledgeable about the process (e.g. employees of contacted public enterprises and schools) would conclude that municipal representatives on the SIAP directed funds in a way that was personally advantageous. This could, in turn, breed negative opinions of the municipal government. The third example was from an infrastructure prioritization session organized between members of the municipal government and representatives of community, youth, and womens’ groups. According to a respondent who participated in the session, one of the four attendees was the mayor, and two other attendees had received their jobs directly from the mayor. When it came time to vote on priorities, this individual was overruled by a unanimous vote. It was this respondent’s opinion that the other members were voting for the mayor’s proposal due to personal, financial ties. If this is true, in addition to subverting the democratic intent of the prioritization exercise, this process can adversely affect perceptions of the municipal government, USAID, and a generalized sense of rule of law. There is no simple solution to this phenomenon. To the extent possible, however, if a process is designed to be democratic and participatory, it should be incumbent on USAID and its IPs to assess the political economy of participants. If the most likely outcome is that the selection process would be subverted toward special interests, the membership structure and/or the governance structures should be revisited. In instances where it is not possible to sufficiently prevent this behavior, it may be best to withhold programming so as not to exacerbate inequities and fuel community suspicions of power brokers. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 36 INTER-ETHNIC COOPERATION THROUGH WORK Interview data across all respondent types evidenced a consistent, shared belief that the best conduit toward true inter-ethnic cooperation was through shared economic interests. Most directly quoted respondents spoke of the benefits of ongoing economic engagements: “when you’re employed and work together every day you’re giving [people] the ability to speak on a daily basis” (Male Community Respondent, Štrpce/Shtërpcë). Respondents spoke of difficult economic prospects and how employment opportunities would “bring people together… since people are doing anything, they would then work together and be happy” (Male Community Respondent, Vushtrri/Vučitrn). Taken in conjunction with the finding that gains in inter-ethnic cooperation are likely to atrophy without structured activities to bring people from different communities together, the notion that a long-term, shared engagement through work seems like an approach that would promote inter-ethnic cooperation and provide spillover benefits with respect to economic growth. In fact, when AKT promoted such engagements through Objective 2, business events and cooperations between different ethnic groups evidenced not just stimulation of economic opportunity, but also increased inter-ethnic exposure and positive collaboration. A business grantee and AKT staff member noted that activities under Objective 2—B2B meetings, seller-buy contracts—did cause members of different ethnic groups to collaborate or work together to receive mutual economic gains. MARKET LINKAGES Formal market linkages have the potential for more constructive engagement and staying power than informal connections. Business grantees reported that although B2B meetings and trade fairs exposed them to potential business partners of another ethnicity, establishing formal business relations was still difficult. They noted that third partner facilitation of business contracts, either through USAID or municipalities, could help cement the market linkages in a more meaningful way. INCLUSION OF MORE MARGINALIZED ETHNIC MINIROTIES IN FUTURE PROGRAMMING Several respondents identified the need to more purposefully include RAE, Turk, and other citizens in future programming. While various AKT activities included and benefited non-Serb ethnic minorities, the focus on Albanian/Serb cooperation drove programmatic decisions, such as working in mixed-ethnic villages. According to government representatives at the central and municipal levels, integration of the RAE community is challenging in these types of interventions because people often live separately on the outskirts of villages rather than in mixed-ethnic villages. AKT PROGRAM COLLABORATION WITH OTHER DEVELOPMENT PROGRAMMING According to interviews with other USAID activities, AKT cultivated a positive cooperation with USAID’s EMPOWER Private Sector activity that aims to increase job creation in Kosovo. Under AKT’s Objective 2, AKT and EMPOWER directed different interventions towards each other that either activity alone was not able or mandated to implement. This mutually beneficial relationship was facilitated by the activities’ use of coordination meetings, an online SharePoint portal, and other mechanisms for exchanging information on beneficiaries so as to avoid double-funding the same beneficiaries or communities. This effective collaboration did not, however, extend to EU programming in northern municipalities. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 37 According to a donor respondent, at the same time as AKT’s kick-off, the EU established a grant scheme focusing on infrastructure interventions and SMEs. This included a special fund for four northern municipalities (Zvečan/Zveçan, Leposavić/ Leposaviq, Zubin Potok/Zubin Potok, North Mitrovica/Mitrovica e Veriut). Although the EU was familiar with AKT’s activities, the two parties held a coordination meeting at the beginning of AKT. While USAID and EU attended donor coordination meetings together, the two entities did not work together strategically. The EU and AKT also differed in their approaches, in that improving inter-ethnic relations was not an integral part of the EU grant scheme, nor did the EU require that business from the North be registered in Kosovo institutions. Although AKT did not actively collaborate with other donors, opportunities exist for increased strategic cooperation and knowledge sharing between USAID activity and the EU. The EU worked in the same northern municipalities as AKT, and could potentially provide important insights to USAID (and vice versa) about working in these challenging environments, as well as sharing lessons learned from different implementation approaches. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 38 RECOMMENDATIONS  The ET recommends significantly simplifying programming, particularly in this complex socio￾political context. While the AKT theory of change was logically and causally sound, implementers were engaged in too many disparate activities to focus programming and effect substantive changes.  Related to the point above, USAID should consider allocating more time to this type of programming. Both AKT grantees accomplished a lot over their 3-year periods of performance. However, given the amount of time that it takes to build relationships at the different programmatic levels, a lengthier period of performance could have increased the ability of implementers to deliver.  Inter-ethnic, community-level programming in Kosovo must be highly contextualized. Each municipality and community is substantively different. To the extent possible, USAID should design implementation mechanisms with the flexibility necessary for meaningful adaptive management. This could take the form of a trial-and-error process, wherein different approaches are piloted, and, if successful, scaled. For this approach to work, however, lengthier programmatic timeframes are required, as is a more robust monitoring and evaluation (M&E) system. There are a number of methodological approaches appropriate for this type of work (e.g. developmental evaluation). The M&E function could be incorporated into the implementation mechanism, as a stand-alone contract, or within a larger Mission or regional￾level M&E platform mechanism.  Co-implementation of AKT through two mechanisms resulted in successes and challenges. If a similar structure is planned in the future, the ET recommends that USAID conduct more thorough planning before procuring either mechanism. Ideally, both mechanisms should be designed in advance of procurement, with clearly delineated scopes (either geographically or sectorally). Additionally, both procurements should make reference to the other mechanism, and explain the ways in which the two contracts are expected to cooperate. These signals will result in more realistic budgeting during the procurement phase, because bidders have a better sense of what is expected of them, and a more efficient start-up process.  If USAID plans on implementing similar programming in the future, eligibility criteria for communities should be strengthened to increase the likelihood of affecting community-level outcomes and increasing the cost-effectiveness of USG programming. The ET recommends two specific factors to consider in setting community minimum eligibility screening: communities should (1) have a minimum number of ethnic minority residents to participate, (2) register a minimum level of participation from targeted ethnic groups for programming to continue.  If USAID intends to implement activities with small-scale procurement actions, as with the quick impact interventions in North Kosovo, guidance should be issued to ensure that a proliferation of small-scale procurement actions does not cause delays or drains on contracting personnel.  Given the broad consensus from respondents at all stakeholder levels, USAID should continue to incorporate job creation activities in this type of programming.  To promote sustainability of market linkages between majority and non-majority communities, USAID should design future programs with a larger emphasis on creating formal contracts between buyers and sellers.  Given the successful approach of coupling business grants with training and technical assistance, USAID should ask IPs to replicate this approach in similar activities. When providing technical assistance, IPs should make sure to give grantees adequate notice before conducting a site visit or announcing a meeting, in order to make sure that grantees are available and their business or personal responsibilities are not disrupted. USAID should also continue the cost-sharing requirement. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 39  To strengthen CDF and its consortium further, CDF should follow up on any management gaps identified in the Year 3 OCA, as well as the communication gaps noted by community respondents, and ensure that they are addressed by the end of implementation in 2018. CDF should also perform OCAs on their consortium partners to increase the quality and uniformity of the consortium’s work. USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 40 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523