VUSHTRRI MUNICIPAL BUILDING (PHOTO BY KRENAR LOSHI), DEMOCRACY PLUS REPORT ON THE PROCUREMENT REVIEW BODY, AND THE PUBLIC PROCUREMENT REGULATORY COMMISSION WEBSITE KOSOVO TRANSPARENT, EFFECTIVE, AND ACCOUNTABLE MUNICIPALITIES MIDTERM EVALUATION REPORT JUNE 2020 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), A Tetra Tech Company. KOSOVO TRANSPARENT, EFFECTIVE, AND ACCOUNTABLE MUNICIPALITIES Midterm Performance Evaluation Report June 4, 2020 IDIQ No. AID-169-1-17-00002 Order No: 72016720F00001 Prepared for United States Agency for International Development/Kosovo Rr. 4 Korriku, nr. 25, Arberia, Prishtine, Kosovo, 10000 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | i CONTENTS ACRONYMS............................................................................................................ II EXECUTIVE SUMMARY ........................................................................................ 1 ANALYSIS ............................................................................................................................... 1 TEAM PROGRAMMING........................................................................................................ 3 TEAM AND FUTURE PROGRAMMING ................................................................................. 4 FUTURE PROGRAMMING .................................................................................................... 4 INTRODUCTION.................................................................................................... 5 EVALUATION PURPOSE AND QUESTIONS................................................... 6 EVALUATION DESIGN AND LIMITATIONS .................................................... 7 DATA COLLECTION............................................................................................................... 7 ANALYSIS PLAN...................................................................................................................... 8 EVALUATION LIMITATIONS ................................................................................................... 8 FINDINGS AND CONCLUSIONS ........................................................................ 9 EVALUATION QUESTION 1 .................................................................................................... 9 EVALUATION QUESTION 2 ...................................................................................................15 EVALUATION QUESTION 3 ...................................................................................................18 EVALUATION QUESTION 4 ...................................................................................................20 EVALUATION QUESTION 5 ...................................................................................................22 TEAM PROGRAMMING.......................................................................................................23 TEAM AND FUTURE PROGRAMMING ................................................................................24 FUTURE PROGRAMMING ...................................................................................................24 ANNEX A: LIST OF REVIEWED DOCUMENTS .............................................26 ANNEX B: DATA TABLES..................................................................................28 USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | ii ACRONYMS AKM Association of Kosovo Municipalities API Application Programming Interface CHU Central Harmonization Unit CSO Civil Society Organization CVA Corruption Vulnerability Assessment DPlus Democracy Plus EQ Evaluation Question ET Evaluation Team EU European Union GLPS Group for Legal and Political Studies GOK Government of Kosovo IAU Internal Audit Unit IT Information Technology KDI Kosovo Democratic Institute KI Key Informant KII Key Informant Interview KIPA Kosovo Institute for Public Administration M&E Monitoring and Evaluation MLGA Ministry for Local Government Administration MPECV Municipal Procurement Effectiveness and Corruption Vulnerability MSI Management Systems International NAO National Audit Office PPRC Public Procurement Regulatory Commission PRB Procurement Review Board T1 Tier 1 T2 Tier 2 TEAM Transparent, Effective, and Accountable Municipalities USAID United States Agency for International Development USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 1 EXECUTIVE SUMMARY To help address Kosovo’s corruption challenges, United States Agency for International Development (USAID)/Kosovo’s Transparent, Effective, and Accountable Municipalities (TEAM) project (2017–2022) aims to improve public procurement fairness, transparency, and oversight by working with municipalities, central bodies, and civil society organizations (CSOs). TEAM has conducted a wide range of analyses and assessments to identify risks, gaps, and needs in procurement processes in municipalities and central bodies, as well as CSO capacity to monitor procurement. USAID/Kosovo requires an independent midterm performance evaluation of TEAM to: ● Assess the extent to which TEAM is on track to achieve the intended activity outputs and outcomes. ● Assess the value added by a new model of technical support. ● Identify gaps in activity design and/or implementation. ● Provide recommendations to USAID and DAI on areas where revisions or additional programmatic support might be warranted. The evaluation team (ET) implemented a mixed-methods approach to address 13 evaluation questions (EQs) and subquestions. The ET scheduled a robust set of visits to municipalities, in addition to Prishtina, due to the geographic spread of TEAM activities: all four of the Tier 1 municipalities outside Prishtina (Gjakove, Gjilan, Peje, and Vushtrri), three Tier 2 municipalities (Lipjan, Prizren, and Suhareke), and one nonparticipating municipality (Rahovec). The ET learned at the Mission inbrief that TEAM had been given a cost expansion to provide the more intensive Tier 1 intervention to all of the Tier 2 municipalities, in addition to four previously nonparticipating municipalities (including Rahovec). The ET interviewed 83 key informants in total (21 female, 62 male). Evaluation limitations included possible response bias and the inability to conclusively isolate TEAM’s contribution to measurable outcomes from that of other external factors. ANALYSIS EQ 1: To what extent did TEAM help five focus municipalities (Tier 1) and six add-on municipalities (Tier 2) address the issues with internal procurement and audit mechanisms identified by the Corruption Vulnerability Assessment (CVA)? How sustainable are the results achieved to date? ● Did the training + intensive coaching model of assistance used in Tier 1 municipalities result in better/worse program outcomes than the training + limited coaching model used in Tier 2 municipalities? ● To what extent can the difference in the outcomes be explained by the TEAM intervention vs. external factors? The ET concluded overall that TEAM strengthened T1 and T2 capacities for procurement and auditing aspects identified in the CVA, with more visible and significant results in T1 municipalities and a larger impact on public procurement and transparency than on internal audit. Municipalities have significant weaknesses in contract management—a broad and complex topic though not a TEAM programming focus. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 2 The results achieved by TEAM are sustainable because key elements of TEAM’s intervention are embedded in municipal and central institutional practices and because of a competitive dynamic among municipalities that is also stimulated by other actors. Financial sustainability remains a challenge, however. TEAM’s intensive coaching model used for T1 municipalities had better outcomes than the less intensive engagement model used for T2 municipalities. Quantitative data from TEAM and external sources demonstrate notable differences in improvement. The ET cannot state with quantitative certainty how much more T1 municipalities gained than T2 municipalities due solely to TEAM’s interventions, however. Comparison of responses from relevant key informants provided multifaceted qualitative evidence that public procurement improved more in T1 municipalities than in T2 municipalities as a result of the support method and intensity. Audit practices improved in both T1 and T2 municipalities, but the extent of TEAM’s contribution is obscured for multiple reasons. External factors boosted or depressed the effects of TEAM’s interventions. A competitive dynamic among municipalities and other initiatives aimed at improving municipal governance helped boost cooperation with and the effects of TEAM, while national budget delays, Procurement Review Board (PRB) complaint processing issues, staffing shortages, and other legislative and political issues acted against TEAM’s effects. EQ 2: To what extent have the central government counterparts of TEAM implemented or committed to the implementation of anticorruption reforms, in particular in the area of municipal public procurement, following TEAM recommendations based on the Corruption Vulnerability Assessment? ● Do the central government counterparts involved in TEAM have the mandate, the capacity, and the political will to implement the reforms recommended by TEAM? ● Did TEAM select the central government partners that are capable of effecting the change in the area of municipal public procurement? Overall, the ET concluded that TEAM contributed strongly to the implementation of anticorruption reforms in procurement through active collaboration with select central institutions. These institutions, except for possibly the PRB, which suffers from a perception of dysfunction and politicization, were generally committed to reforms. TEAM’s central government partners have the legal mandate to implement anticorruption reform in public procurement at the municipal and central levels. All central government partners, however, have technical capacity limitations that threaten the sustainability or at least continued progress of reforms. The PRB and the Central Harmonization Unit (CHU) may also suffer from issues of political autonomy. EQ 3: How did TEAM contribute to increasing the intensity and the quality of corruption tracking and exposure by civil society in Kosovo? Are there examples of the increased tracking and exposure by civil society leading to better procurement practices? The ET concluded overall that TEAM helped increase the intensity and quality of corruption tracking and exposure by civil society. Local procurement monitoring is sustainable for the time being, but organizations may not prioritize this type of work without continuous donor support. Activities by CSOs focused on a broader level had notable positive successes. Participating organizations valued the experience of joint or consortium projects. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 3 Civil society efforts promoted better procurement practices at both broader and and local levels; in some cases, there was a spillover effect from local to broader level, and in the case of national CSOs and their transparency initiatives, the impact is quite visible. EQ 4: Are there internal or external impediments that significantly compromise TEAM’s ability to achieve the intended results following the original schedule? ● To what extent does TEAM implementation adhere to the key crosscutting principles? ● To what extent does TEAM implementation ensure gender equality? The ET concludes overall that several internal or external impediments could affect TEAM’s ability to achieve intended results, but the most serious are the delay in the 2020 national budget (external), which will place tremendous pressure on municipal procurement, and the complexity of improving the conduct of contract management tasks (internal), which range from the pre-tendering to the post-tendering phase and can vary depending on the type of procurement. TEAM made extensive use of well-regarded Kosovo CSOs. Training was conducted in close coordination with Kosovo law and mandated government partners in a sustainable and reproducible manner. TEAM operated with central government counterparts within appropriate anticorruption frameworks and strategies and coordinated with other donors to avoid duplication. Selection of municipalities deliberately and explicitly included consideration of indictments of mayor or senior staff. The ET concluded that TEAM ensured gender-proportionate participation in training and other similar activities; however, it did not further focus on gender equality in other activities that would incorporate gender equality into public procurement processes, such as monitoring procurement projects of more interest to men than women. EQ 5: What are the windows of opportunity for further improvement of the public procurement sector that can be pursued either within the scope of TEAM or by future activities of USAID/Kosovo? The ET has the following recommendations for adjusting TEAM programming or future USAID/Kosovo programming. TEAM PROGRAMMING ● Include as a graduation criterion that the municipality have the minimum legally required number of internal auditors. ● Expand the practice of municipal interdepartmental working groups to actively monitor remediation of National Audit Office (NAO) recommendations and extend this practice to include Internal Audit Unit recommendations. ● Resolve hosting and sustainability issues for key procurement systems and platforms. ● Sponsor applied research into two key uncertainties about PRB-municipal interaction. ● Maintain funding for applied research on municipal procurement by national-level CSOs. ● Expand social audit practice to include gender analysis. ● Strengthen methodological coherence and fidelity of TEAM’s data analytics. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 4 TEAM AND FUTURE PROGRAMMING ● Address weaknesses and challenges in municipal contract management for common or high-risk procurements. ● Continue supporting the NAO’s antifraud activities. ● Restrategize engagement with local CSOs for procurement tracking. FUTURE PROGRAMMING ● Coach municipalities to develop more sophisticated and inclusive public outreach methods to collect information on community needs. ● Continue to engage and strengthen engagement with the Association of Kosovo Municipalities collegia to develop sustainable professional training and encourage content development for distance learning. ● Connect municipalities and public utilities to coordinate project planning. ● Intensify engagement in different ways to improve PRB performance. ● Support the CHU’s efforts to incentivize municipalities to be more responsive to internal audit recommendations. ● Support consideration of gender issues in technical designs where appropriate. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 5 INTRODUCTION Corruption and low capacity of integrity institutions present the biggest challenges to democratic development in Kosovo. Inadequate rule of law through accountability institutions and low central and municipal government capacity produce poor governance outcomes that fail to meet citizen needs. At the same time, civil society organizations (CSOs) lack the technical and financial capacity to perform key watchdog activities. To help address these development challenges, United States Agency for International Development (USAID)/Kosovo’s Transparent, Effective, and Accountable Municipalities (TEAM) project (2017–2022) aims to improve public procurement fairness, transparency, and oversight by working with municipalities, central bodies, and CSOs. TEAM has conducted a wide range of analyses and assessments to identify risks, gaps, and needs in procurement processes in municipalities and central bodies, as well as CSO capacity to monitor procurement. Activities implemented through TEAM contribute to USAID/Kosovo’s Country Development Cooperation Strategy’s Development Objective 1: Improved Rule of Law and Governance that Meet Citizens’ Needs, specifically: ● IR1.2: Strengthened Effectiveness and Accountability of Assemblies, Administrations, and Election Processes. USAID TEAM is improving the accountability of municipal administrations and assemblies through greater transparency and effectiveness. ● IR1.4: Civil Society Strengthened to Increasingly Engage Constructively with Government. USAID TEAM is working with civil society to monitor municipal procurement. TEAM has three components: 1. Develop, Refine, and Roll Out Models for Transparent and Accountable Municipalities. Following a selection process based on performance metrics and other indications of reform commitment, TEAM identified five Tier 1 (T1) or focus municipalities: Gjakove, Gjilan, Peje, Prishtina, and Vushtrri. It also identified six Tier 2 (T2) municipalities: Ferizaj, Istog, Lipjan, Podujeve, Prizren (added later), and Suhareke. Through coaching and mentoring and the introduction of new systems and enforcement mechanisms, TEAM worked on increasing the capacities of municipal procurement managers, contract managers, internal auditors, and other public officials involved in the procurement cycle. The intervention for T1 municipalities included training and frequent, intensive coaching by a dedicated municipal advisor and subject matter experts, and the intervention for T2 featured training and infrequent, limited coaching by subject matter experts. All 38 of Kosovo’s municipalities could participate in training provided through appropriate central bodies. TEAM collected and analyzed large quantities of data about corruption vulnerability areas, organizational capacity, and other procurement-related performance metrics. At the time of the inbrief for this evaluation, the evaluation team (ET) learned that TEAM had been given a cost expansion to move all six of the T2 municipalities and four non-tier municipalities up to the T1 (or “focus”) intervention level. There will be no T2 municipalities going forward. 2. Engage the Central Level in Addressing Municipal Corruption within the Procurement Process. TEAM worked closely with a small set of central institutions to increase their capacity for oversight of municipal procurement. With its main central partner, the Public Procurement Regulatory Commission (PPRC), TEAM revised and implemented an e-procurement system and other guidance materials. TEAM also worked with the Ministry of Finance’s Central Harmonization Unit (CHU) and USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 6 the autonomous National Audit Office (NAO) to strengthen coordination between those bodies and the municipal internal audit units (IAUs) on procurement issues. Finally, TEAM worked with the Procurement Review Board (PRB) to promote transparency and more consistent decisions on procurement complaints filed by bidders. 3. Enable Civil Society to Track and Expose Corruption. TEAM supported national and local CSOs to monitor procurement and corruption and report findings to the public. Through its partner CiviKos, TEAM coached and mentored local CSOs to monitor municipal procurements to increase citizen engagement. TEAM also used a Support Fund for initiatives identified by T1 municipalities, the development of a contractor past performance record system, and CSO engagement in procurement watchdog activities. TEAM has four relatively novel aspects. First, TEAM applied a flexible but deliberate adaptive management approach to implementation. Second, embedding a CSO component inside another technical project allowed for more focus on local CSOs than is usual with technical projects. Third, TEAM applied two different intervention modalities, enabling comparison of modality effectiveness. Finally, TEAM collected and analyzed a large amount of data on municipality and central institution procurement performance. EVALUATION PURPOSE AND QUESTIONS USAID/Kosovo requires an independent midterm performance evaluation of the TEAM activity to: ● Assess the extent to which TEAM is on track to achieve the intended activity outputs and outcomes. ● Assess the value added by a new model of technical support. ● Identify gaps in activity design and/or implementation. ● Provide recommendations to USAID and DAI on areas where revisions or additional programmatic support might be warranted. This evaluation, led by the evaluation team (ET) of Dr. Andrew Green, Krenar Loshi, and Edis Agani for Management Systems International (MSI), addressed the following 13 evaluation questions (EQs) and subquestions: EQ 1: To what extent did TEAM help five focus municipalities (Tier 1) and six add-on municipalities (Tier 2) address the issues with internal procurement and audit mechanisms identified by the Corruption Vulnerability Assessment? How sustainable are the results achieved to date? ● Did the training + intensive coaching model of assistance used in Tier 1 municipalities result in better/worse program outcomes than the training + limited coaching model used in Tier 2 municipalities? ● To what extent can the difference in the outcomes be explained by the TEAM intervention vs. external factors? EQ 2: To what extent have the central government counterparts of TEAM implemented or committed to the implementation of anticorruption reforms, in particular in the area of municipal public procurement, following TEAM recommendations based on the Corruption Vulnerability Assessment? USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 7 ● Do the central government counterparts involved in TEAM have the mandate, the capacity, and the political will to implement the reforms recommended by TEAM? ● Did TEAM select the central government partners that are capable of effecting change in the area of municipal public procurement? EQ 3: How did TEAM contribute to increasing the intensity and the quality of corruption tracking and exposure by civil society in Kosovo? Are there examples of the increased tracking and exposure by civil society leading to better procurement practices? EQ 4: Are there internal or external impediments that significantly compromise TEAM’s ability to achieve the intended results following the original schedule? ● To what extent does TEAM implementation adhere to the key crosscutting principles? ● To what extent does TEAM implementation ensure gender equality? EQ 5: What are the windows of opportunity for further improvement of the public procurement sector that can be pursued either within the scope of TEAM or by future activities of USAID/Kosovo? EVALUATION DESIGN AND LIMITATIONS The ET implemented a mixed-methods approach—consisting of document review and key informant interviews (KIIs)—to determine the extent to which TEAM has built organizational capacity to improve transparent, effective, and accountable procurement. DATA COLLECTION The ET gathered data from multiple sources. The document and materials review included implementer periodic reports, work plans, performance and assessment data, training materials, guidebooks, and other documents such as assessment reports and sustainability plans. In addition, the ET used documents and analyses of municipal performance and CSO partner work plans and research products. The ET also conducted KIIs with representatives of participating municipalities and TEAM’s central and CSO partners. The ET tailored each data collection instrument to generate data on the EQs and finalized the semi￾structured interview protocols after discussions with Mission staff. Each semi-structured KII differed slightly depending on the key informant’s (KI’s) role, the extent of the KI’s involvement in TEAM activities, and the KI’s available time to be interviewed. The ET’s questions addressed knowledge and general perceptions but targeted specific examples of attitudes and behavioral changes. Given the relatively small set of potential KIs with knowledge of procurement activities, the ET selected KIs purposively instead of randomly. The ET scheduled a robust set of visits to municipalities, in addition to Prishtina, due to the geographic spread of TEAM activities: all four of the T1 municipalities outside Prishtina (Gjakove, Gjilan, Peje, Vushtrri); three T2 municipalities (Lipjan, Prizren, Suhareke); and one nonparticipating municipality (Rahovec). The ET learned at the Mission inbrief that TEAM had been given a cost expansion to provide the more intensive T1 intervention to the T2 municipalities, in addition to four previously nonparticipating municipalities (including Rahovec). The ET interviewed 83 KIs in total (21 female, 62 male). USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 8 ANALYSIS PLAN The ET used primarily parallel analysis to examine the evidence from the document review and KIIs. For each question, the ET first used content analysis to examine relevant implementer materials and secondary sources to develop preliminary findings. Second, the ET reviewed data from each type of KI to draw preliminary findings for each through content analysis before synthesizing across all types of KIs. For each of the first two steps, the findings were mapped to the timeline and highlights to develop preliminary findings about USAID-supported good governance activities. Finally, the ET drew conclusions about intervention activities for each EQ. EVALUATION LIMITATIONS The ET attempted to mitigate KI biases through a semi-structured interview protocol that included questions about how new skills and knowledge were used, an opening statement about ET independence and confidentiality, and inclusion of KIs beyond those identified by TEAM. In this way, the ET helped KIs focus on the specific training topics, for example, rather than their overall experience with trainings. Key biases that the ET watched for included the following: Recall bias: Multiple actors in Kosovo have worked on issues related to good governance, so KIs may not have accurately remembered details of particular intervention activities. For example, training participants may have responded to ET questions with answers related to previous USAID-funded activities or those funded by another donor. Veteran KIs in municipalities or central bodies may have blended their experiences over a long period into a composite memory or response. This did not appear to be a problem with KIs interviewed by the ET, because donor interventions have been quite distinct. Response bias: Informants may have contributed positive remarks about the activity because they would like to stay involved with it and because they thought negative remarks could cut off activity-related opportunities. This was clearly the case in many of the interviews with direct beneficiaries—and is always the case with implementers—but the ET’s use of questions about specific behaviors or actions mitigated this bias. Selection bias: Including only KIs identified by the implementers could have led the ET to hear only from people who had had positive experiences. This is often a problem for activities in which the main contacts have a deep relationship with the implementer. Therefore, the ET interviewed KIs at organizations that were indirect beneficiaries to get a different perspective and expanded the set of KIs to include people not listed by the implementer. Contribution: Multiple donor projects have aimed to strengthen municipal governance and/or central government capacity in Kosovo. There are three ways this could have affected evaluation results. First, TEAM’s training may have simply reinforced training provided by an earlier project. It was difficult for the ET to collect information specific enough to quantify this risk, so TEAM’s contribution to improved governance may be misstated. Second, other donor projects may have worked (or be working) with a mix of T1 and T2 municipalities, potentially obscuring any differential effects of the two models of assistance. Past work by DEMOS on public financial management, auditing, and procurement monitoring or EU support for generally strengthening internal control are examples of this contribution identification problem. Third, TEAM’s selection of municipalities was not random; rather, the selection and categorization of T1 and T2 municipal partners were based largely on performance and evidence of reform engagement USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 9 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 EQ 1: To what extent did TEAM help five Tier 1 municipalities and six Tier 2 municipalities address the issues with internal procurement and audit mechanisms identified by the Corruption Vulnerability Assessment (CVA)? The ET concluded overall that TEAM strengthened T1 and T2 capacities for procurement and auditing aspects identified in the CVA, with more visible results in T1 municipalities and a larger impact on public procurement than on internal audit. How sustainable are results achieved to date? The ET concluded overall that the results achieved by TEAM are sustainable because key elements of TEAM’s intervention are embedded in municipal and central institutional practices and because of a competitive dynamic among municipalities that is also stimulated by other actors. Financial sustainability of training activities remains a challenge, however. The ET concluded that certificate training in public procurement is sustainable legally and technically but faces financial and logistical challenges. The ET found that TEAM supported two rounds of public procurement certification training for all procurement officials in collaboration with the Kosovo Institute for Public Administration (KIPA) and the PPRC. The first round conferred a basic certificate that was legally required for all procurement officers; it was urgent to recertify all procurement officers by January 2018 for them to be able to fulfil their duties. The second round conferred an advanced certificate that was legally required of procurement officials within three years of the basic certificate. In both rounds, TEAM, the PPRC, KIPA, and external consultants updated an existing training program to reflect the e-procurement system and other legislative changes. In addition, TEAM, KIPA and PPRC have introduced distance learning by posting several videos/tutorials on using the e-procurement platform. This distance learning was intended for a larger audience, including the economic operators. While KIPA and PPRC funded the second round of certificate training in their budgets, TEAM supported a Training of Trainers effort to expand capacity to conduct future training; moreover, any training not identified in KIPA’s annual planning document would require most if not all of the funding from donors. The ET found that in such massive training endeavours, operational costs other than those for trainers and the training location (coffee, lunch, water, etc.) can be covered only by donor funding, which was KIPA’s and PPRC usual practice. The distance learning tools will remain more within the PPRC capacity and, like the content of any specialized training, would have to be developed by PPRC or another specialized institution outside of KIPA. The ET concluded that the systems, procedures, manuals, templates, and forms developed with TEAM assistance are firmly embedded in the daily routine of municipality procurement processes. System sustainability is not certain, however, due to hosting and personnel issues. The ET found that procurement officials, internal auditors, and finance directors knew and used the procedures manual—in fact, many had the materials at arm’s reach. Similarly, every municipal procurement official mentioned other templates and forms as valuable tools developed by TEAM. For example, procurement-related templates developed for T1 municipalities were shared with all other municipalities through the Collegium for Public Procurement under the Association of Kosovo Municipalties (AKM). USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 10 The ET found that the public sector platforms and systems launched with TEAM support are used by all stakeholders. All municipal procurement officials reported using the PPRC’s e-procurement system as a routine part of the procurement process, as mandated by law. Similarly, the PRB database of complaints and decisions, a key part of the PRB website, and live-streamed Board sessions are used regularly by national-level CSOs and media. Though the e-procurement system should be hosted at the Agency for Information Society to guarantee the availability of technical expertise, PPRC KIs stated that those discussions have not been fruitful. The ET found that the sustainability of platforms outside government, such as the Levizja FOL open procurement platform, are vulnerable to funding gaps; fortunately, in the case of Levizja FOL, a media partnership developed after the ET’s fieldwork may resolve the issue for the Open Procurement Transparency Portal. The ET concluded that due to TEAM’s efforts, municipal transparency has become anchored in an interactive dynamic of legislation, mayors, CSOs, and donors. The ET found that municipal officials valued TEAM’s efforts to develop content for their revamped websites, including links to procurement solicitations, awarded contracts, and municipal budget information The ET found that the TEAM graduation plan and assistance helped boost transparency at the municipalities and helped trigger competition between TEAM focus municipalities at least, and later within T2 municipalities as well. The most visible transparency-increasing initiatives were open contracts and publication of all procurement activities and financial documents, such as time series. All T1 mayors or deputy mayors reported that a transparency competition of sorts has developed among municipalities, spurred in part by mayoral initiatives, including the above-mentioned initiatives, CSO applied research (e.g., Kosovo Democratic Institute’s [KDI’s] Transparency Index supported by TEAM since 2017), DEMOS performance grants, and Ministry for Local Government Administration (MLGA) rankings. One example of expanding transparency is Gjakova’s intent to publish construction permits, which would allow citizens to spot illegal constructions and permit deviations. EQ 1–1: Did the training + intensive coaching model of assistance used in Tier 1 municipalities result in better/worse program outcomes than the training + limited coaching model used in Tier 2 municipalities? The ET concluded overall that TEAM’s intensive coaching model used for T1 municipalities had better outcomes than the less intensive engagement model used for T2 municipalities. Quantitative data from TEAM and external sources demonstrate notable differences in improvement. The ET cannot state with quantitative certainty how much more T1 municipalities gained than T2 municipalities due solely to TEAM’s interventions, however. The ET concluded that public procurement improved more in T1 municipalities than in T2 municipalities as a result of the support method and intensity. The ET found that all T1 municipal procurement officials, internal auditors, finance directors, and public communications officers could give specific examples of how their work had improved thanks to coaching from municipal advisors and experts. Intensive coaching improved procurement procedures and practices in various ways, including template production, introduction of procurement procedures the municipal staff was not aware of or did not understand, and guidelines for dividing tenders in lots and similar actions that would produce more efficient contracting. The ET found that in T1 municipalities, especially in Pristina, the TEAM municipal advisor supported the procurement department by creating notification letters for different phases of tender, standard text for tender dossiers, and guides for various contract types and procurement strategies. The differential effects of engagement intensity were evident in KIIs, as T1 municipalities clearly benefited more than T2 municipalities; in the latter, procurement officials could cite the certification training and procedures manual, but no additional USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 11 specific examples of improvement as their T1 counterparts could. In addition, KIs at all T1 municipalities mentioned TEAM assistance on Integrity Plans and the ethics code. The ET found that the annual procurement planning process has improved through TEAM support. T1 municipalities demonstrated improved intramunicipal cooperation—through the support of the TEAM municipal advisor and experts, the procurement departments are now communicating more regularly and substantively with other departments in developing the procurement plan. Procurement managers reported less difficulty implementing their procurement plans. The ET concluded that the data collected by TEAM indicate significantly higher rates of improvement for T1 municipalities over T2 municipalities. TEAM collected outcome data and aggregated data in multiple ways to track progress as part of its thorough application of adaptive management; it is exceedingly rare for evaluation teams to have such abundant data to provide meaningful insights on project activities. The ET found that T1 municipalities as a set improved at significantly higher rates than T2 municipalities as a set despite starting at or near the same position. TEAM collected performance metrics and generated data through annual rating exercises. The resulting data were used to create a multidimensional Municipal Procurement Effectiveness and Corruption Vulnerability (MPECV) Index; a multidimensional organizational capacity score calculated from a subset of MPECV indicators; and performance metrics, such as percentage of unplanned procurements, percentage of cancelled procurement, and so on. The ratings and performance data were then used to create a scoring and ranking system to identify T1 municipalities for graduation from TEAM support. Data from TEAM and external sources demonstrate that the T1 municipalities improved over time (see Tables E1–E3 in Annex E). For the set of five municipalities, there have been fewer cancelled procurements, more implementation of planned procurements, and fewer unplanned procurements. As can be seen in Table E1, the T1 municipalities as a group had higher average performance than T2 municipalities on these selected metrics, although some individual T2 municipalities scored better than some T1 municipalities for individual metrics depending on the year. Of particular note are the consistent performances across T1 municipalities for implementation of planned procurements, publication of procurement activities on the municipal website, and KDI’s Transparency Index. For the set of five municipalities, organizational capacity has also demonstrably improved over time in planning and budgeting, as well as transparency: as can be seen in Table E2, despite starting at roughly the same overall score, the T1 municipalities improved over two years by 52 percent compared with the T2 municipalities with 20 percent; when adjusted for score inflation (see next paragraph), the comparison is still 45 percent to 15 percent. Moreover, each of the six dimensions making up the organizational capacity score were higher for T1 municipalities in 2019, which was not the case in 2017 or 2018. T2 municipalities also saw improvements but not to the level of T1s and with much more variation within the set of T2s. The ET found that TEAM did not maintain scoring methodology fidelity, however, which inflated scores. The scoring for sub-indicator items under MPECV are ordinal variables with integer scoring between 0 and 4 based on extensive definitions that are mutually exclusive between scores, but TEAM assigned fractional scores for a significant number of sub-indicators.1 Table E3 shows the effect of corrected sub￾indicator scores (rounding down to the nearest integer to reflect the fully achieved scoring level) for 1 Almost 25 percent of MPECV indicator scores for 2018 (96/396) included fractional scores (e.g., 1.5 or 2.5 or 2.7 or 2.8) when no such scoring is defined or justified by TEAM’s own scoring guidance. The use of fractional scores occurred almost evenly across T1 and T2 municipalities. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 12 organizational capacity, which reduces the amount of overall improvement and narrows the T1-T2 gap for some dimensions. This is not just an arcane methodological matter as the sub-indicator scores are used to calculate progress ratings for graduation, so inflated scores at the lowest level would mathematically inflate scores at the highest level (Tables E3 and E4).2 The ET concluded that audit practice improved in both T1 and T2 municipalities, but inadequate auditor staffing in some municipalities and the nonmandatory nature of internal auditing recommendations obscured TEAM’s contribution. The ET found from all its interviews with T1 auditors that TEAM helped increase internal auditors’ efficiency—mostly through clarification of roles and responsibilities but also through internal audit training; the ethics code; risk assessment materials; templates; standard operating procedures; and good practices, such as an active working group for responding to NAO recommendations. TEAM did not provide intensive coaching on auditing as it did for procurement; rather, the audit advisor would visit the T1 and T2 municipalities one or two times per month and all nonparticipating municipalities had access to audit experts. However, T1 and T2 municipalities did not necessarily receive the same amount of expertise and attention: the audit advisor was pulled into T1-specific activities by municipal advisors and TEAM leadership for such issues as addressing recurring NAO audit findings; increasing interaction among municipal audit committees, mayors, and IAUs (which was later done with T2 municipalities as well); technical assistance to the municipal policy and finance committees on implementing NAO and IAU recommendations; and more. There was a synergy among the deeper engagement of municipal advisors, the TEAM leadership’s greater attention to the T1 municipalities, and the audit advisor’s expertise that did not exist for T2 municipalities. T1 auditors could point to specific tools, practices, and knowledge that they were using. TEAM’s contributions to improving internal auditing were obscured (or more latent) because not all T1 and T2 municipalities have as many auditors on staff as required by law: approximately 35 percent of municipalities have only one auditor. Vushtrri and Peje each had two auditors, but both should have had three or four; three of the new T1 municipalities (Prizren, Rahovec, and Suhareke) also had only one auditor. This lack of staffing directly affects the frequency and quality of internal audits, as well as the absorption of support in the form of training or coaching. The staffing deficit is in part a result of budget and legal restrictions on staffing levels but also municipal prioritization for hiring or shifting staff and competition with the private sector for such expertise. “TEAM gave us a voice.” – Tier 1 auditor The ET found that internal auditors at T2 municipalities likewise had the internal audit training and access to guidance on risk assessment. In addition, some of the T2 municipalities had a fully staffed IAU that could have a more active annual work plan. The ET also found that the cooperation between internal auditor and procurement, finance, and other departments differs from municipality to municipality, not from TEAM tier to tier: in some municipalities, such as T2 Lipjan, the internal auditor cooperates very closely 2 Another methodological issue is in the construction of the graduation rating, which uses a mixture of performance metrics and three of the six dimensions of organizational capacity: nowhere is it explained why three dimensions are left out, and the excluded dimensions include at least one of the lowest scores for T1 municipalities in 2018. In addition, the graduation rating scheme essentially double-counts transparency activities and leaves overall ratings vulnerable to volatility due to exogenous factors (e.g,, the delayed 2020 national budget). USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 13 with the procurement department and contract managers, but in T1 Vushtrri this cooperation was not seen. The ET found that the majority of mayors or deputy mayors were aware of the need to respond to NAO recommendations to maintain MLGA ranking. T1 municipalities, however, had been advised to establish an active working group chaired by the finance director and with an action plan and regular reporting; prior practice for municipalities had been to distribute NAO recommendations to the relevant departments and assume the problems were being addressed. No auditor reported that internal audit recommendations were given priority by municipal executives. The ET concluded that contract management presents a significant weakness in municipal procurement, but its temporal and technical complexity cannot be addressed by any single donor project in a short time. The ET found that the most common unmet need or problem at the municipal level was contract management, as reported by nearly all KIs in both T1 and T2 municipalities. The ET also found, however, that respondents varied in the contract management problems they had in mind. Contract managers pointed to the Contract Management Manual and the strategy of searching the e-procurement system for similar examples of procurements being designed but were much more concerned about overwork from too many contracts to manage and issues with quality assurance of implemented work. Some procurement officials were concerned with the quality of technical design, usually for public works, IT, or other equipment. Some auditors were concerned about proper monitoring of contract compliance—that is, deadlines and deliverables. Some procurement officials and auditors were concerned with project implementation—that is, supervising work in progress and practicing appropriate quality control, usually for public works. Still other procurement officials and finance directors pointed to specific issues such as the cost and delays of property expropriation or swap for public works or the lack of coordination with water or electric utility companies. All, however, pointed to the same problem of poor outcomes from awarded contracts and how those outcomes affected citizens. The ET concluded that T1 municipalities were more transparent in their procurement than T2 municipalities, although an emerging issue beyond TEAM’s intervention is that municipal outreach practice is limited and similar in each tier. The ET found that T1 municipalities had more content available on their websites than T2 municipalities, including municipal budgets, annual plans, links to procurement solicitations, awarded contracts, and more. This is due in part to the existence of publicly available information through the e￾procurement system but also to TEAM’s coaching of public communication officers in municipalities. One issue the ET found beyond TEAM’s intervention that emerged during interview discussions about public engagement with mayors, deputy mayors, procurement officials, and public communication officers was that no municipality was deliberately and strategically developing citizen outreach efforts to better ascertain community procurement needs. Procurement plans generally derive from the approved budgets, which often contain underbudgeted activity wish lists, which in turn affects the procurement planning process. developed by municipal leadership based on identification of needs from many sources. Although public hearings were not part of TEAM’s intervention and the ET’s interview protocol did not address them, KIs raised the issue during discussions of transparency and public engagement. Municipalities are legally required to hold two public hearings during the budget planning processes, but all informants claimed that no one or almost no one ever came to the hearings, which were usually scheduled for weekday evening hours at a public building. Those attending were usually CSO representatives or certain political party activists. Another practice that has become more popular among municipalities is consultation with village councils to gather community needs from their perspective. The village councils are elected by the community, but there is USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 14 no evidence of village council deliberative processes. Nevertheless, no municipality seeks out deliberately gender-specific needs for public works or disabled persons’ needs for services. EQ 1-2: To what extent can the difference in the outcomes be explained by TEAM intervention vs. external factors? The ET concluded that external factors acted to boost or depress the effects of TEAM’s demonstrably positive intervention. Among them are initiatives from cooperating donors and the Government of Kosovo (GOK), a competitive dynamic among municipalities, budget delays, PRB complaint processing, and staffing shortages. The ET concluded that MLGA and DEMOS initiatives boosted the already positive effects of TEAM interventions. The ET found that both DEMOS and MLGA have financial incentive schemes that include performance and compliance indicators closely linked to areas of support provided by TEAM. As an iterative feeback loop, TEAM support to municipalities in public procurement and internal audit affects overall improvement of municipal performance and, more specifically, individual performance indicators used in mentioned grant schemes, and municipalities have extra incentive to cooperate with TEAM. “If you drop even just a little bit in the ranking, it is costly.” – Mayor The ET concluded that municipal reputation and competition are a new reality in Kosovo municipalities. Mayors and deputy mayors told the ET that TEAM and the MLGA and DEMOS grant schemes have stimulated a competitive culture that is visible in almost all municipalities and among almost all mayors, improving municipal performance. An example of such reputational motivation is the NAO report and its recommendations, as well as the extent to which these recommendations have been addressed within the financial year. A culture of accountability has started. The ET concluded that budget delays (such as in 2020) negatively impact procurement plan implementation and budget delivery. Delays in approval of the national budget put tremendous implementation pressure on procurement, not simply because procurements must be carried out in a shorter timeframe but also because longer or more complex contracts may not be completed by the end of the year. This would lead to a higher percentage of the budget not being procured as planned, in addition to budget items being moved to the following year. Delays in PRB responses further strain municipalities’ ability to implement the procurement plans in a timely manner because reevaluation of proposals adds to the work stream, delays contract award, and ultimately delays contract implementation. The ET concluded that the lack of internal auditors negatively affects municipal performance despite TEAM interventions. The ET found that not all T1 and T2 municipalities had the legally mandated number of auditors on staff. This is in part an external factor, both because staffing levels are limited by budget and law and because the private sector is more attractive for auditors.3 It is also in part a problem that could be addressed through TEAM intervention, as the shortage is also a question of mayoral priorities: mayors can find available positions for engineers or IT specialists but not for auditors. Without an adequate 3 The positions of municipal auditors that have been suspended pending court processes cannot be filled by replacements, either (e.g., in Peja and Gjilan). This does not, however, prevent municipalities from creating new positions that are filled. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 15 number of auditors, the risk of corruption being discovered is reduced and the inability to monitor remedial actions in accordance with NAO recommendations is reduced. The ET also found two potential external factors not mentioned at the municipal level. One is that central institutions have focused on promulgating new laws on public procurement and internal control, which should resolve other technical or bureaucratic issues experienced by municipalities. In addition, political volatility at the national level leads to planning uncertainty because of the high turnover of government body leadership under a new government. “Can’t fix procurement problems in the municipalities without fixing the PRB.” – CSO key informant EVALUATION QUESTION 2 To what extent have the central government counterparts of TEAM implemented or committed to the implementation of anticorruption reforms, in particular in the area of municipal public procurement, following TEAM recommendations based on the Corruption Vulnerability Assessment? Overall, the ET concluded that TEAM contributed strongly to the implementation of anticorruption reforms in procurement through active collaboration with select central institutions. These institutions—except possibly for the PRB, which suffers from a perception of dysfunction and politicization—were generally committed to reforms. The ET concluded that the PPRC has implemented key procurement reforms and is internally committed to further reforms. The ET found that the PPRC has made a significant shift in organizational culture—once just a regulatory body, it is now a regulator that provides services to a wide variety of stakeholders. The PPRC has worked closely with TEAM, for example, to revise and launch the e-procurement system, adapt additional modules for use by other procurement process stakeholders, and develop human and material help resources. The ET found that the PPRC is also deeply committed to providing and updating manuals, handbooks, and templates for more consistent procurement practice and to providing training for their municipal counterparts as needed. The PPRC and KIPA, with TEAM support, have conducted basic and advanced public procurement trainings for more than 500 procurement officers at both the central and municipal government levels; TEAM further supported both institutions with a Training of Trainers effort and has worked with the PPRC to have online videos for help with specific issues. The ET found that all tenders above EUR 1000 are online, published through the e-procurement platform in accordance with PPRC’-issued notification of 2018 and 2019. This was mainly enabled through TEAM support to the overall functioning of the e-procurement platform managed by the PPRC. The PPRC is committed to launching the contract management and contractor past performance modules with TEAM assistance. The ET concluded that the PRB is internally committed to cooperating with donors on IT-related improvements. Application of decision-making authority is neither consistent nor predictable, however, and the resulting uncertainty contributes to dysfunctional behavior of municipalities and economic operators. External political commitment to a functioning PRB remains in doubt. The ET found that the PRB actively cooperated with TEAM to implement IT upgrades to increase efficiency and public transparency; engagement with the PRB was an activity added through learning and adaptive management. In addition, the UK/PAI project (2018–March 2020) has USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 16 assisted the PRB in developing templates to help the PRB board link new decisions to previous ones from the database of decisions (developed by TEAM), as well as in developing templates for PRB board justification when decisions differ from those of experts. UK is planning a new three-year project that includes digitalization of PRB decision-making processes. TEAM worked closely with both the PPRC and the PRB to launch the e-procurement system module, which allowed for online submission of procurement process complaints by economic operators—a significant improvement over the former paper-based system. TEAM and the PRB also redesigned the website and added content, including, as of September 2019, a growing and searchable database of complaints and PRB decisions. The PRB also began live-streaming Board sessions for the public. The ET found that PRB decisions were inconsistent across similar cases. This universal criticism voiced by the mayors and municipal procurement officials interviewed by the ET was confirmed by DPlus statistics showing that more than 30 percent of the PRB decisions are the opposite of what was recommended by the PRB’s own external technical experts. The new database of decisions and the UK-funded projects could become resources for setting precedents, however. Another universal criticism voiced by relevant municipal-level KIs was that PRB decisions almost always took the full amount of time no matter how simple or complex the underlying complaint. PRB KIs acknowledged the timing problem, pointing to the lack of staff capacity to handle the volume of complaints and lack of Board time to review complaints and issue decisions. The ET found that each side blames the other for shirking their responsibilities in the complaint process—municipal-level KIs claim that the PRB sends tenders back to them for reevaluation because it does not want to make a decision, but PRB KIs claim that municipal procurement officials and selection committees do not fulfill their responsibilities in anticipation that the PRB will do it for them. An MLGA KI noted that ministries and other central bodies do not have this problem with complaints to the PRB. It was not in the ET’s scope to collect enough evidence to draw conclusions about this dynamic, however. A related problem is that the PRB returns complaint submission fees to economic operators, so that the effective cost of complaining is zero and therefore more and more complaints are being submitted; the PRB KIs’ response was that it was not fair to economic operators to make them pay for responsibility shirking by municipalities. The ET found no evidence of political commitment to continue reform efforts beyond the longstanding fact that two of the five Board member positions have been vacant for almost three years due to inaction by the Assembly. The ET concluded that the CHU is committed to internal auditing reforms at the municipal level but stymied at the ministerial level from implementing sanctions on municipalities for poor performance. The ET found that the CHU valued TEAM support for actions that helped strengthen internal auditing at the municipal level, including developing the Manual on Procurement Procedures, providing training on procurement auditing, coaching IAUs on best practices and risk assessment tools, and using municipal working groups to systematically implement and monitor actions in response to NAO recommendations. The ET found that CHU is strongly interested in performance audits as a standard tool for IAUs but needs TEAM technical support for training and piloting. KIs from the CHU expressed their strong interest in developing the practice of performance audits, and the ET learned after fieldwork concluded that TEAM will support two performance audits with significant CHU participation. The CHU has also recommended that the Ministry of Finance sanction municipalities with poor audit response performance through unilateral budget cuts, but this recommendation has not been taken up by the ministry. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 17 The ET concluded that the NAO is committed to identifying and investigating procurement fraud and increasing cooperation with the CHU, IAUs, and prosecutors. The ET found that the NAO greatly valued TEAM support to set up its Antifraud Unit and strengthen the relationship between the NAO and prosecutors. In the last two years, the NAO has sent 87 cases for prosecution and produced an antifraud guide for auditors. The ET also found that the NAO valued TEAM coaching to T1 and T2 municipalities, in addition to short￾term audit advisors that were available to all municipalities, to form working groups to respond to annual NAO recommendations, including an action plan and regular internal reporting. The NAO has been moving into performance audits, but this activity would be covered under a memorandum of understanding with the Swedish NAO. The ET concluded that KIPA is committed as a partner to reform efforts by counterpart central institutions. The ET found that KIPA, the legally mandated training organization for civil servants, benefited substantively from engagement with TEAM. TEAM worked with KIPA to revise a procurement training curriculum and develop and update modules and materials for the basic and advanced public procurement training. In addition, TEAM, in collaboration with KIPA and PPRC, developed video trainings for e-procurement. KIPA has a legal commitment to training but relies on the PPRC and CHU for a substantial portion of their training roster (especially for more specialized topics) and on donor projects for funding of significant training efforts. EQ 2-1: Do the central government counterparts involved in TEAM have the mandate, the capacity, and the political will to implement the reforms recommended by TEAM? EQ 2-2: Did TEAM select the central government partners that are capable of effecting change in the area of municipal public procurement? The ET concludes overall that TEAM’s central government partners have the legal mandate to implement anticorruption reform in public procurement at the municipal and central levels. All central government partners, however, have technical capacity limitations that threaten the sustainability or at least continued progress of reforms. Two partners, the PRB and the CHU, may also suffer from issues of political autonomy. The ET concluded that the PPRC has sufficient capacity and political autonomy to implement its mandated regulatory role, but its complementary service provider role is threatened by human capacity limitations. The PPRC is the legally mandated governmental body for regulating public procurement. The ET found that three elements of the PPRC’s service provider role have significant capacity limitations. First, the IT department, as with all IT departments throughout the GOK, faces difficulties with staff turnover due to salary disparities between the public and private sectors. This is particularly acute for the PPRC, the ET found, because the IT head is hired as a repeat short-term consultant, a human resource strategy that is not viable in even the medium term. Second, the PPRC can hire external IT experts for system updates or new module integration, but that requires significant budgetary prioritization and planning and is difficult to get approved due to the government’s freeze on new employment since 2018. Finally, the training unit has only two people responsible for curriculum revisions and coordination of trainings with KIPA and other stakeholders. The ET concluded that the PRB has the legal mandate to implement reforms, but it suffers from inadequate capacity and perceptions of politicization. The ET found that the PRB has limitations on its capacity to push forward with reforms. In particular, its current staffing level and Board vacancies cannot handle the rising volume of procurement complaints from economic operators. The complaints management department USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 18 is understaffed—it has only four internal review experts and sometimes has difficulty contracting external technical experts in a timely manner. The law requires three of the five Board members to render decisions, but there have been only three Board members for nearly three years, so every decision is made by the same three people. The two vacant Board seats have been the subject of political maneuvering in the Assembly, which lends credence to the universal criticism that the PRB is politically influenced. An additional capacity issue is the lack of IT expertise; the PRB has unsuccessfully asked the PPRC for minor programming changes to its modules. The ET concluded that the CHU has a mandate that would support reforms, but it may not have the technical capacity and political autonomy to act as needed. The ET found that the CHU is responsible for auditor training and certification, but it must supplement its training with external trainers and donor funding. The CHU reports directly to the Finance Minister and therefore is influenced by politics and appointee changes. Despite its statutory authority over municipal-level internal audit units, the CHU cannot force municipalities to hire the number of auditors required by law, nor can it sanction municipalities that do not act upon IAU recommendations. The ET concluded that the NAO has a clear legal mandate, strong autonomy, and sufficient capacity to partner on continued reform implementation. The ET found that the NAO’s mandate and autonomy are anchored in law, and it has sufficient technical capacity. The NAO, however, does need TEAM’s assistance to develop learning materials on fraud detection and cooperate with other donor projects to engage prosecutors on a deeper and more systematic level, including the development of dossier templates and regular engagement at leadership levels. As noted above, TEAM has actively worked with the NAO to stand up the Antifraud Unit, and NAO KIs reported it will need continued technical assistance to strengthen and expand the Unit. The ET concluded that although KIPA is legally mandated to provide training to civil servants, it relies largely on other GOK counterparts and foreign donors for trainers and funding. The ET found that KIPA had been a partner for TEAM, the PPRC, and the CHU for training of procurement officers and internal auditors, but the curriculum content and trainers were supplied by TEAM, the PPRC, and CHU; even the external trainers contracted through a KIPA list were paid for by the others. KIPA has moved into a newer, larger training facility, but it does not yet have distance learning capacity to roll out new certifications or other content training. EVALUATION QUESTION 3 EQ 3-1: How did TEAM contribute to increasing the intensity and the quality of corruption tracking and exposure by civil society in Kosovo? Overall, the ET concluded that TEAM helped increase the intensity and quality of corruption tracking and exposure by civil society. Local procurement monitoring is sustainable for the time being, but organizations may not prioritize this type of work without continuous donor support. Activities by CSOs focused on a broader level had notable positive successes. Participating organizations valued the experiences of joint or consortium projects. The ET concluded that TEAM comprehensively tackled weaknesses in civil society’s ability to track and expose corruption involving public procurement at both the national and local levels through a variety of methodological approaches and activities. There are positive results among both national and local CSOs . At the local level, the USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 19 ET found that all CSO KIs valued the TEAM and CiviKos strategy for workshops, especially the hands-on approach of using actual procurements for learning rather than lectures. The ET also found that very few CSOs at the national or local levels had any experience with procurement monitoring, according to implementer and CSO KIs, but even the experienced local CSOs stated that they had developed a more systematic understanding of procurement processes. TEAM partner CiviKos conducted a needs assessment of CSOs and then organized trainings through an applied learning workshop methodology in seven regions with around 20 participants each; the workshop consisted of two days on procurement processes and monitoring and one day on media engagement. Following the workshops, seven consortia of CSOs were selected for mentoring during direct monitoring of a procurement selected by the CSO in their municipality. The findings were presented in a series of reports that were shared with municipalities and published. At the central level, the ET found that TEAM support increased the transparency of procurement activity. Levizja FOL established an open procurement platform to enable citizens, media, and interested parties to access information on awarded contracts by public institutions. To date, FOL has extracted more than 48,000 scanned contracts from PPRC’s e-procurement system, which were then processed by FOL to add key variables in searchable formats; at this time, only approximately 15 percent of the scanned contracts have been posted on the platform with full data, due to the tremendous labor needed to manually process the scanned documents in the absence of a functioning programming interface. The ET found from central partner KIs that the lack of a programming interface is due to the lack of a law on digital signatures—all contracts must be physically signed and thus uploaded to the e-procurement system as a scanned document that is not machine readable.4 The ET found that TEAM-supported applied research produced valuable public results on procurement issues. Democracy Plus (DPlus) monitored TEAM’s T1 municipalities for procurements of various sizes that are typically riskier (e.g., construction and IT). Problems that DPlus identified, such as application of the incorrect VAT rate on milk for Prishtina schools or the substitution of used computers for new ones in Vushtrri, did lead to action by the municipalities against the economic operators. DPlus also stated that they are continuing procurement monitoring in other municipalities with funding from the Kosovo Civil Society Foundation and the EU. DPlus is also collaborating with FOL to develop a “red flag” system to identify potentially risky procurements for monitoring by CSOs. DPlus is the only CSO monitoring PRB activities. Its research results indicate that PRB decisions are inconsistent, that decisions are delayed, and that complaint fees are almost always returned to economic operators. In addition, there is little to no reliance on past decisions as precedent. The PRB, with TEAM assistance, began uploading complaints and decision documents to a searchable database on its website. The Group for Legal and Political Studies (GLPS) implemented social audits, a methodology for analyzing effects on different populations. The ET found that results from the social audits were shared with municipal leadership and are directly connected to changes benefiting citizens: the need for GPS systems in ambulances to locate residences quickly, the lack of dental checks for children in schools or other centers, and more. The ET found that GLPS valued this new methodology as well as the opportunity to work more directly with municipalities, and it is developing a social audit toolkit for use by local CSOs. 4 The ET learned from the Mission after fieldwork that the PPRC will implement new modules that allow for the draft contract to be machine readable by an API, although the final contract will still be scanned and not machine readable. It is possible that the draft contract could be different from the final contract, and only visual comparison would be able to identify differences. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 20 GLPS also reported that the kind of collaborative approach encouraged by TEAM led to strong relationships with municipal leadership that gave their research results more credibility. EQ 3-2: Are there examples of the increased tracking and exposure by civil society leading to better procurement practices? The ET concluded that civil society efforts promoted better procurement practices at both broader and and local levels; in some cases there was a spillover effect from local to broader levels, and in the case of national CSOs and their transparency initiatives the impact is quite visible. At the local level, CSOs participating in applied learning workshops drafted and published seven monitoring reports (one per region). These reports were shared with mayors, and some were picked up by media, adding pressure to municipal authorities (e.g., Prizren, Mitrovica South). In three cases, the CSO investigations led to corruption investigations by relevant authorities. However, sometimes local CSOs are reluctant to openly criticize their municipalities, fearing loss of access to municipal grants and activities linked to their core mandate (e.g., youth, women). Some CSO KIs reported that USAID project backing was important for gaining cooperation from municipal officials and even that the indirect path through TEAM to mayors was needed on occasion to resolve challenges in access to information. Other than media attention, no CSO KI reported an effect of their monitoring efforts on the selected procurement, although one procurement manager provided an example of a recommendation from a TEAM-supported CSO procurement tracking that resulted in lower costs for school hygiene supplies. The effects of CSO procurement monitoring activities at the local level were reported by national-level CSOs such as DPlus and GLPS. The ET found that municipalities are interested in improving their ranking on transparency and public participation scores—for example, KDI’s Transparency Index, supported by TEAM—and in identifying innovative ways to increase public participation, especially among women, youth, and communities. At the central level, DPlus and GLPS reported positive actions taken by municipalities in response to their findings—for example, the milk VAT rate in Prishtina or dental checks at Vushtrri schools. Both the PPRC and the PRB pointed to the value of a “red flag” system in the FOL open procurement database because this system would increase scrutiny of municipal procurement by national-level CSOs. FOL reported that they have seen more use of the database by media. EVALUATION QUESTION 4 EQ 4: Are there internal or external impediments that significantly compromise TEAM’s ability to achieve the intended results following the original schedule? USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 21 The ET concludes that several potential internal or external impediments could affect TEAM’s ability to achieve intended results (as discussed above under Evaluation Question 1), but the most serious are the delay in the 2020 national budget (external), which will place tremendous pressure on municipal procurement, and the complexity of improving the conduct of contract management tasks (internal), which range from the pre-tendering to the post-tendering phase and can vary depending on the type of procurement. The ET found that TEAM’s adaptive management allowed it to identify potential programming challenges and develop strategies to overcome them. TEAM’s cost expansion to provide T1-level engagement with an additional 10 municipalities will also be handled by expanding staff. The TEAM Chief of Party and Component 1 lead have been with the project for multiple years, so the staff expansion will present more of a coordination problem than a management problem. The ET found that all mayors and procurement officials at municipalities were concerned with improving technical design (pre-tendering) and implementation oversight (post-tendering). Procurements range from office supplies to construction of new schools, so it would be difficult to develop technical guidance beyond templates and standard specifications that may or may not closely match a specific project. The ET found that municipal procurement office KIs were most concerned about construction projects such as roads, buildings, or other public facilities. The ET also found, however, that all mayors, finance department heads, and municipal procurement officials were particularly worried about the effect of the long-delayed 2020 national budget on their ability to procure and implement well. The delay raised the probability that procurement plans would not be completed, that construction projects would not be finished before cold weather stopped work, that the 2021 procurement plan would need to include funding for unfinished 2020 projects, and that resulting poor procurement performance would affect them negatively in ratings used by DEMOS and MLGA. The ET found that the central government partner KIs were concerned primarily with long-needed revisions to such key laws as the Law on Public Procurement and the Law on Internal Control. The fall of the previous government in 2019 and subsequent delay in formation of a new government after the elections stalled the development and consideration of new legislation. EQ 4-1: To what extent does TEAM implementation adhere to the key crosscutting principles? The ET concluded overall that TEAM made extensive use of well-regarded Kosovo CSOs, including think tanks and network organizations. Training was conducted in close coordination with Kosovo law and mandated government partners in a sustainable and reproducible manner. TEAM operated with central government counterparts within appropriate anticorruption frameworks and strategies and coordinated with other donors to avoid duplication. Selection of municipalities deliberately and explicitly included consideration of indictments of mayor or senior staff. The ET concluded that TEAM made extensive use of Kosovar organizations through subcontracts and grants under contract. The ET found that much of TEAM’s effort beyond municipal advisors and technical experts was made by Kosovar organizations such as CiviKos, DPlus, Riinvest, and others. TEAM used CiviKos in particular as the most active part of the civil society component. Other organizations provided applied research or consulting expertise through contracts or grants. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 22 The ET concluded that TEAM supported training sustainably through appropriate partners in the technical sense, but TEAM financial support was still needed by KIPA. The ET found that TEAM collaborated closely with KIPA, the PPRC, the CHU, and AKM to develop and deliver training through KIPA as required by law for civil servant certification. At the same time, the training materials are held outside TEAM and a Training of Trainers effort has taken place. The ET concluded that TEAM coordinated its activities with the Anticorruption Strategy, the National Strategy for Public Procurement Reform, and the activities of other donors. The ET found that TEAM and its experts were fully aware of the two strategies, ensured that the corruption vulnerability assessment addressed key elements of the strategies, and developed project work plans in line with the strategies. The development of Integrity Plans for T1 municipalities was done in collaboration with MLGA, the Ministry of Public Administration, and the Anticorruption Agency. TEAM coordinated often with DEMOS, the most active municipal governance project outside USAID’s portfolio, as well as with MLGA to avoid duplicating or undermining other initiatives. TEAM’s use of adaptive management allowed it to identify and respond to changing priorities and context, for example public information strategies, municipal website content, procurement officer basic certification, engagement with the PRB, and municipality-driven activities beyond intensive coaching and training The ET concluded that TEAM avoided selecting participating municipalities with a mayor or senior staff under indictment. The ET found that this specific variable was included in TEAM’s process of selecting participating municipalities. EQ 4-2: To what extent does TEAM implementation ensure gender equality? The ET concluded that other than ensuring gender-proportionate participation in training and other similar activities, there was no evidence of addressing gender equality throughout TEAM activity implementation. The ET found that during various activities such as training, workshops, and panel discussions, the participation of counterparts was recorded and the data were disaggregated by gender. The number of women participating in the activities, including the Women Week 2019, was commendable. Moreover, the number of women participants in the activities organized with CSOs was satisfactory. However, this was a result of TEAM activities’ efficient implementation rather than targeted, gender-focused activity planning. The ET found from CSO KIs that despite extensive data gathering, analysis, monitoring, and reporting for procurement tracking activities, none had a specific focus on gender equality. The monitored contracts and produced reports (as part of TEAM’s program) did not include any gender-based analysis. Finally, the ET found that the wide range of data collected by TEAM for the CVA and beyond, as well as ranking/graduation schemes for municipalities, did not include gender-related data. EVALUATION QUESTION 5 EQ 5: What are the windows of opportunity for further improvement of the public procurement sector that can be pursued either within the scope of TEAM or by future activities of USAID/Kosovo? USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 23 The ET has the following recommendations, grouped by topic or target, for adjusting TEAM programming or future USAID/Kosovo programming: TEAM PROGRAMMING ● Include as a graduation criterion that the municipality have the minimum legally required number of internal auditors. A key challenge to improving municipal procurement accountability is that many municipalities lack sufficient internal auditing capacity. The Law on Budget limits personnel levels for municipalities, but mayors somehow can find positions for other priorities—for example, Peje used positions that became available through attrition to develop a cadre of nine engineers to handle project technical design. The lack of internal auditing capacity reflects municipal leadership’s priorities, and no municipality should graduate without adequate auditing practices. ● Expand the practice of interdepartmental working groups to actively monitor remediation of NAO recommendations and extend the practice to include IAU recommendations. The working group model promoted by TEAM was more successful at addressing NAO recommendations than was previous practice, but TEAM should keep working with central and municipal partners to ensure that it becomes the standard practice for all municipalities. In addition, this working group should include recommendations from internal auditors as part of its monitoring work. ● Resolve hosting and sustainability issues for key procurement systems and platforms. The e-procurement system should be hosted and maintained at the Agency for Information Society, not the PPRC, which has difficulty funding IT experts. Sustainability is also an issue for the Open Procurement Transparency Portal developed by Levizja FOL, but that may have been resolved through a media partnership developed after the ET’s fieldwork ended. ● Maintain funding for applied research on municipal procurement by national-level CSOs. The applied research results from DPlus, KDI, Levizja FOL, GLPS, and so forth have had identifiable impact on municipal actions and serve a broad constituency. Further iterations of their work would feed into the municipal competition dynamic and have more effect over time. ● Sponsor applied research into two key uncertainties about PRB-municipal interaction. First, approximately 30 percent of PRB decisions run counter to the opinions of their own experts, which is highly problematic for a regulatory body that should be nonpartisan. However, there may be procurement problems other than the kind of technical issues requiring expert input. Second, the PRB and municipalities accuse each other of shirking their responsibilities for upholding procurement standards and processes. An examination and analysis of reevaluation decisions could identify common mistakes or omissions by either side. ● Expand social audit practice to include gender analysis. Social audits are a key tool for CSOs and municipalities to identify critical aspects of how planned projects affect target communities, and a greater focus on various population groups in communities (women, disabled, youth, elderly) would improve project outcomes. ● Strengthen methodological coherence and fidelity for data analytics. Revise all indicator (or raw) scores downward to remove fractional scores that have no definition or justification in scoring guidance. Review and revise the graduation criteria to avoid double-counting effects or exogenously generated volatility. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 24 TEAM AND FUTURE PROGRAMMING ● Address weaknesses and challenges in contract management for common or high-risk procurements. Contract management is a broad and complex enough issue that it could be the focus of a stand￾alone donor project. A project such as TEAM or another USAID project could program against only part of the problem. The ET recommends that activities target specific, prominent procurements such as roads, sidewalks, or lighting for intensive coaching through the entire lifecycle of the procurement—that is, from technical design through approval—using a working group approach instead of the more customary assembly line or stovepipe approach. ● Continue supporting the NAO’s antifraud activities with closer cooperation with the U.S. OPDAT program to strengthen judges’ and prosecutors’ capacities to detect fraud in public procurement and further expand training within the KJA. ● Restrategize engagement with local CSOs for procurement tracking. The current approach for engaging local CSOs has not been as fruitful as hoped for engaging with or influencing selected local procurements; a different approach built on stakeholder incentives could be more effective. For example, a public infrastructure project such as a village road could leverage a local CSO’s monitoring capacity with the village’s desire to have a high-quality, completed road and provide another channel to prompt municipal cooperation. FUTURE PROGRAMMING ● Coach municipalities to develop more sophisticated and inclusive public outreach methods to collect information on community needs. The current practice is to assume that department heads know what is needed in communities or, at best, that village council heads have that knowledge. There are known gender differences for procurement, best expressed by the saying, “men want roads, women want services.” This is simplistic, of course, but gender does matter when prioritizing or designing public works—men may be more concerned with parking space on new roads, whereas women may prioritize better lighting that increases personal safety. Municipalities also need to reach out more deliberately to the disabled and the elderly to ascertain their priorities. ● Continue to engage and strengthen further engagement with the AKM collegia to develop sustainable professional training and encourage content development for distance learning. More specifically, continue the usage of procurement collegium within the AKM as a forum for all procurement officers in municipalities. AKM has dedicated training space and extensive direct connections to municipalities, and better coordination between KIPA and AKM could help develop a culture of continuing professional education. Now that the basic and advanced procurement training curricula exist, for example, there is a need for more specialized learning—for example, security system fundamentals or risk points in roadway construction. Focused modules could be developed for distance learning, which not only extends the reach of such training but also resolves many of the financial and logistical issues faced by KIPA and AKM. ● Connect municipalities and public utilities to coordinate project planning. Water and electric public utilities currently do not coordinate with municipalities as public works or other infrastructure projects are planned. This is particularly important in the feasibility study phase of capital USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 25 investment projects. As a result, expensive new village roads are torn up for water pipes, or electricity distribution towers cut through new public parks. ● Intensify engagement in different ways to improve PRB performance. First, pressure from the international community is needed to fill vacant Board positions or replace the current Board entirely; this is necessary for PRB to function as it should by law and alter its politicized reputation. Second, as UK support to PRB ended in March 2020 and the problem persists, TEAM should cooperate with the PRB until the new UK project kicks in to reduce the number of reevaluation or partial decisions, which have detrimental effects at the municipal level (see also the following recommendation). Third, TEAM or another donor project should work with the PRB to routinize the use of precedent in decisions, which goes beyond mere accessibility of past decisions in an IT system. This would make PRB decisions more predictable and thus influence both municipality and economic operator behavior and provide written justifications for noncompliance with expert opinions. ● Support the CHU’s efforts to incentivize municipalities to be more responsive to internal audit recommendations. Municipalities do not prioritize internal audit recommendations, and the CHU’s authority to force more attention is extremely limited. A municipal-oriented donor project such as TEAM could informally increase incentives for municipal responsiveness through agenda setting, training, or reputational rating activities. ● Support consideration of gender issues in technical designs where appropriate. TEAM should collaborate with the AKM and gender-focused CSOs to develop a manual for identifying and addressing potential gender issues in technical designs. USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 26 ANNEX A: LIST OF REVIEWED DOCUMENTS USAID Transparent, Effective, and Accountable Municipalities, “Needs Gap Analysis, Revised,” April 2017 USAID Transparent, Effective, and Accountable Municipalities, “Year 1 Work Plan” (redacted), June 2017 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Procurement Corruption Vulnerability Assessment,” PowerPoint presentation, August 2017 USAID Transparent, Effective, and Accountable Municipalities, “Quarterly Performance Report” (redacted), 2017 quarters 1, 2, and 3 USAID Transparent, Effective, and Accountable Municipalities, “Analysis of Current Capacity of NAO’s Local Government Division and Performance Audit Department,” October 2017 USAID Transparent, Effective, and Accountable Municipalities, “Baseline Assessment: Civil Society Organizations (CSOs),” December 2017 USAID Transparent, Effective, and Accountable Municipalities, “Activity Management Plan,” December 2017 USAID Transparent, Effective, and Accountable Municipalities, “Year 2 Work Plan” (redacted), January 2018 USAID Transparent, Effective, and Accountable Municipalities, “Assessment of the Procurement Review Body,” January 2018 USAID Transparent, Effective, and Accountable Municipalities, “Annual Performance Report, January 2017-January 2018” (redacted), February 2018 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Capacity Assessment Report,” February 2018 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Selection SPC,” Excel spreadsheet, February 2018 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Graduation Plan,” April 2018 USAID Transparent, Effective, and Accountable Municipalities, “Activity Management Report, January 2017-January 2018,” June 2018 USAID Transparent, Effective, and Accountable Municipalities, “Activity Management Report, January 2018-January 2019,” March 2019 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Capacity Assessment Report,” March 2019 USAID Transparent, Effective, and Accountable Municipalities, “Revised Municipal Graduation Plan,” April 2019 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Comparison Data,” Excel spreadsheet, April 2019 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Graduation Report,” May 2019 USAID Transparent, Effective, and Accountable Municipalities, “Citizens’ Perception Survey on Corruption and Transparency in Public Procurement 2019,” May 2019 USAID Transparent, Effective, and Accountable Municipalities, “Public Contract Management Manual,” June 2019 USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 27 USAID Transparent, Effective, and Accountable Municipalities, “Indicator Data Graduation Semi-Annual,” Excel spreadsheet, September 2019 USAID Transparent, Effective, and Accountable Municipalities, “Revised Activity Management Plan,” January 2020 USAID Transparent, Effective, and Accountable Municipalities, “Activity Management Report, January 2019-January 2020,” February 2020 USAID Transparent, Effective, and Accountable Municipalities, “Annual Report,” February 2020 USAID Transparent, Effective, and Accountable Municipalities, “Municipal Selection MLGA Report,” Excel spreadsheet, February 2020 USAID Transparent, Effective, and Accountable Municipalities, “MPECV Scoring 2019,” Excel spreadsheet, March 2020 Ministry of Local Government Administration, “Municipal Performance Report 2018,” 2019 Kosovo Democratic Institute, “Transparency Index in Public Procurement 2018,” May 2019 Kosovo Democratic Institute, “The Process of Developing Municipal Procurement Planning in Kosovo 2019,” October 2019 Kosovo Democratic Institute, “The 2019 Municipal Budget Development Process in Kosovo,” November 2019 Kosovo Democratic Institute, “2019 Municipal Transparency and Accountability Index,” March 2020 DemocracyPlus, “PRB: The Public Procurement Knot,” June 2017 DemocracyPlus, “Benchmarking Public Procurement,” May 2018 DemocracyPlus, “Towards Municipalities with Open, Accountable and Efficience Public Procurement,” November 2018 DemocracyPlus, “Benchmarking Public Procurement,” February 2019 DemocracyPlus, “The Role of Procurement Review Body in Public Procurement Inefficiency,” June 2019 DemocracyPlus, “Level of Publication of Public Procurement Contracts,” 2020 DemocracyPlus, “The (Many) Problems of Public Procurement,” 2020 Monitoring Group of Non-Governmental Organizations from Peje Region, “Asphalting of Local Roads in Village Vitomerice,” May 2019 Monitoring Group of Non-Governmental Organizations from Prizren Region, “Pool of Irregularities,” May 2019 Monitoring Group of Non-Governmental Organizations from Ferizaj Region, “The Construction of the Sports Hall for Physical Education in Gacke Village Second Phase,” June 2019 Monitoring Group of Non-Governmental Organizations from Gjilani Region, “Construction of the Square Over the River Mirusha-South,” June 2019 Monitoring Group of Non-Governmental Organizations from Mitrovica South Region, “Industrial Recycling Centre for Presorted or Mixed Waste,” June 2019 Monitoring Group of Non-Governmental Organizations from Prishtina Region, “Roundabout in Lakrishte, Roundabout in Arberi,” June 2019 Platforma CiviKos, “Analyses Report on Procurement Activities for the Second Phase of the Construction of Sports Hall and Fire Station in North Mitrovica,” July 2019 USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 28 ANNEX B: DATA TABLES TABLE E1. COMPARISON OF SELECT PERFORMANCE METRICS FOR T1 AND T2 MUNICIPALITIES % implementation of the procurement plan % of publication of procurement activities on mun. web sites % of NAO recommendatio ns implemented for prior report Degree of transparency (measured by KDI) 2017 2018 2019 2018 2019 2018 2019 2018 2019 Prishtinë 34.3% 50.4% 76.9% 99.7% 100% 26.9% 24.0% 100% 86.5% Pejë 91.4% 76.8% 82.7% 74.1% 100% 47.4% 57.9% 75.8 % 81.3% Gjakovë 59.8% 83.9% 87.0% 74.1% 93.0% 31.8% 15.8% 73.0 % 81.3% Gjilan 36.7% 52.6% 83.5% 91.7% 99.2% 33.3% 33.3% 72.3 % 79.2% Vushtrri 84.1% 89.5% 86.5% 97.3% 98.8% 35.0% 47.6% 93.6 % 81.3% T1 Average 61.3% 70.6% 83.3% 87.4% 98.2% 34.9% 35.7% 82.9 % 81.9% Ferizaj n/a 46.2% 63.0% 17.4% 39.0% 17.2% 23.5% 84.7 % 77.1% Istog n/a 76.7% 85.2% 13.1% 35.3% 25.0% 31.3% 64.4 % 68.8% Lipjan n/a 73.9% 76.0% 100.0 % 50.8% 37.5% 46.7% 92.4 % 80.2% Podujevë n/a 46.7% 59.6% 9.2% 47.2% 17.6% 31.8% 59.7 % 69.8% Prizren n/a 40.6% 56.9% 37.3% 57.5% 0.0% 25.0% 61.5 % 80.2% Suharek ë n/a 74.2% 88.5% 1.0% 90.0% 9.1% 42.9% 55.5 % 81.3% T2 Average 59.7% 73.0% 29.7% 53.3% 17.7% 33.5% 69.7 % 76.2% TABLE E2. COMPARISON OF ORGANIZATIONAL CAPACITY SCORES, 2017–2019 ORIGINAL 2017 2018 2019 Change 2017- 19 T1 T2 T1 T2 T1 T2 T1 T2 *Planning and budgeting 2.00 2.03 2.40 2.13 2.83 2.42 +42% +19% *Staff capacity 2.00 1.30 2.40 1.50 2.67 1.65 +34% +27% Contract administration 1.30 1.90 1.70 1.75 2.10 1.84 +62% -3% Audit 1.53 1.63 2.14 2.00 2.47 2.04 +61% +25% Appeals and review 2.00 1.73 2.30 1.80 2.53 1.97 +27% +14% USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 29 ORIGINAL 2017 2018 2019 Change 2017- 19 T1 T2 T1 T2 T1 T2 T1 T2 *Transparency, ethics, anticorruption, anti￾fraud 1.40 1.30 2.60 1.76 2.92 1.91 +109 % +47% Overall 10.2 3 9.89 13.54 10.94 15.5 2 11.8 3 +52% +20% * Variable used for the Municipal Graduation Plan TABLE E3. COMPARISON OF REVISED ORGANIZATIONAL CAPACITY SCORES, 2017-2019 REVISED 2017 2018 2019 Change T1 T2 T1 T2 T1 T2 T1 T2 *Planning and budgeting 2.00 1.93 2.27 1.93 2.67 2.06 +34% +7% *Staff capacity 2.00 1.20 2.00 1.00 2.00 1.33 0% +11% Contract administration 1.30 1.90 1.45 1.50 2.00 1.58 +54% -17% Audit 1.20 1.20 1.70 2.00 2.10 2.00 +75% +67% Appeals and review 2.00 1.73 2.20 1.80 2.53 1.94 +27% +12% *Transparency, ethics, anticorruption, anti￾fraud 1.28 1.20 2.60 1.56 2.88 1.60 +125 % +33% Overall 9.78 9.17 12.2 2 9.79 14.1 8 10.52 +45% +15% Score Deflation% - 4.4% - 7.3% - 9.7% -10.5% - 8.6% -11.1% * Variable used for the Municipal Graduation Plan TABLE E4. COMPARISON OF ORIGINAL AND REVISED 2019 GRADUATION SCORES ORIGINAL 2019 Peja Vushtr ri Gjakova Gjilan Prishtina Procurement (max. 73) 60.00 58.00 47.40 53.00 38.10 *MPECV (max. 27) 25.47 25.47 25.47 25.47 24.27 Total (max. 100) 85.47 83.47 72.87 78.47 62.37 REVISED 2019 Peja Vushtr ri Gjakova Gjilan Prishtina Procurement (max. 73) 60.00 58.00 47.40 53.00 38.10 *MPECV (max. 27) 22.98 22.98 22.98 22.98 21.18 Total (max. 100) 82.98 80.98 70.38 75.98 59.28 USAID.GOV KOSOVO TEAM MIDTERM EVALUATION – FINAL REPORT | 30 Total Score Deflation% -2.9% -3.0% -3.4% -3.2% -5.0% *MPECV Score Deflation% -9.8% -9.8% -9.8% -9.8% -12.7% * The Municipal Graduation Plan only incorporated three organizational capacity variables