This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of IMPAQ International, LLC and do not necessarily reflect the views of USAID or the United States Government. MID-TERM PERFORMANCE EVALUATION OF THE USAID’S GOVERNMENT ACCOUNTABILITY INITIATIVE (GAI) ACTIVITY IN SERBIA November 2020 GAI-SUPPORTED SERVICE CENTER IN THE CITY OF SOMBOR | CREDIT: CITY OF SOMBOR i | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV CONTENTS TABLE OF EXHIBITS I ACRONYMS AND ABBREVIATIONS II EXECUTIVE SUMMARY III 1. INTRODUCTION 1 1.1 Evaluation Purpose 1 1.2 Evaluation Methodology 1 2. FINDINGS 2 2.1 Evaluation Question 1: GAI Accomplishments to Date 2 2.1.1 To strengthen the capacities and connections of the key stakeholders to increase government accountability at national and local levels 3 2.1.2 IR1. Increased public participation in local government decision-making and oversight 5 2.1.3 IR2: Increased follow-up to oversight institutions’ recommendations 6 2.1.4 IR3: Increased adjudication of corruption cases 8 2.2 Evaluation Question 2: Linking the Activity’s Objectives and Results 9 2.2.1 Improving Sector Policies 9 2.2.2 Effects on Participating Institutions 12 2.2.3 Management Practices 13 2.3 Evaluation Question 3: Scope of Work Expansion 14 2.3.1 Assessment of the Current SOW 14 2.3.2 Additional Areas of Intervention that Could Fit Under the SOW 18 2.3.3 Flexibility to Changing Needs and Priorities 21 2.4 Evaluation Question 4: Future Programming 22 2.4.1 Interventions with Greatest Traction 22 2.4.2 Unintended Results 24 2.4.3 Best Practices 25 2.4.4 Lessons Learned 27 2.5 Evaluation Question 5: Gender Equality 28 3. CONCLUSIONS AND RECOMMENDATIONS 29 3.1 Conclusions 29 3.2 Recommendations 31 ANNEX A. EVALUATION STATEMENT OF WORK 33 ANNEX B. ACTIVITY BACKGROUND 36 ANNEX C. ANALYSIS OF KEY PERFORMANCE INDICATORS 40 TABLE OF EXHIBITS Exhibit 1: The WJP Open Government Index for Serbia, 2015-2020 ..................................................................4 Exhibit 2: LTI for GAI-Supported Local Self-Governments.....................................................................................4 Exhibit 3: Perceptions of GAI’s Gender Integration (n=72)..................................................................................29 USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | ii ACRONYMS AND ABBREVIATIONS ACA Anti-Corruption Agency AP Action Plan AVP Automated Case Management System for Courts COP Chief of Party CPI Commissioner for Information of Public Importance and Personal Data Protection CRDA Community Revitalization through Democratic Action CSO Civil Society Organization DAC Development Assistance Committee DO Development Objective DPO Data Protection Officer EC European Commission EQ Evaluation Question ERCC Electronic Register of Corruption Cases EU European Union FGD Focus Group Discussion GAI Government Accountability Initiative GRECO Council of Europe’s Group of States against Corruption IOI Independent Oversight Institution IR Intermediate Result KII Key Informant Interview LAP Local Anti-Corruption Plan LSG Local Self-Government LTI Local Transparency Index MELP Monitoring, Evaluation, and Learning Plan NA National Assembly NACS National Anti-Corruption Strategy OECD Organisation for Economic Co-operation and Development PII Personally Identifiable Information PPO Public Prosecutors’ Offices SAI State Audit Institution SAPO Software Application for the Prosecutor’s Office SOW Scope of Work TOC Theory of Change USAID United States Agency for International Development WJP OGI World Justice Project Open Government Index iii | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID) in Serbia engaged IMPAQ International, LLC. (IMPAQ) to conduct an independent, mid-term evaluation of the USAID-funded Government Accountability Initiative (GAI). This $8.5 million, four-year activity started in February 2018 and will extend through February 2022. The evaluation team independently assessed the effectiveness of the activities implemented to date, determined whether the Activity has achieved planned results, identified gaps in performance against targets, and provided recommendations on how to close these gaps in the Activity’s remaining period of performance. This evaluation informs USAID and GAI implementing partners of the success of the Activity, captures useful lessons learned, and provides recommendations to enable GAI to make course corrections and strengthen programming to achieve a greater and more sustained impact. METHODOLOGY The evaluation team conducted primary and secondary research, collecting qualitative, and quantitative data, to address the evaluation and assessment questions in the Scope of Work (SOW), between September 23 and October 13, 2020. The team’s primary research consisted of 44 Key Informant Interviews (KIIs) and four Focus Group Discussions (FGDs), reaching a total of 66 persons. It also included an online survey, which received 72 responses. The team answered the following five key evaluation questions (EQs): 1. To what extent is the program on track to accomplish the inputs, outputs, and likely outcomes expected under all three intermediate results of the activity?1 2. What links can be determined between progress towards the Activity’s objectives and the Activity’s results? 3. What other areas of interventions could fit under the SOW beyond what is already planned? 4. What are the interventions with greatest traction, unintended outcomes, and/or lessons learned from USAID assistance that the Mission can use in making decisions for future programming? 5. Did the Activity affect gender equality within the area(s) of focus? FINDINGS AND CONCLUSIONS EQ1: To what extent is the program on track to accomplish the inputs, outputs, and likely outcomes expected under all three intermediate results of the activity? The evaluation team found that GAI is on track to successfully deliver the expected inputs and outputs, meeting the established targets and contributing to organizational development and capacity building of the institutions that are critical to increasing government accountability. Still, to date, GAI’s efforts have not led to any measurable improvements in government accountability, the desired likely outcomes, especially at the national level. ● IR1: Increase public participation in local government decision-making and oversight. Authorities from participating municipalities recognized that support from GAI has been effective in enhancing public participation, especially concerning local budget planning.2 Namely, GAI’s efforts to develop capacities and create a culture of openness, engagement, and public information sharing contributed to an enabling environment for public participation in GAI municipalities. Within this 1 The evaluation changed the original wording of EQ1 included in its statement of work and the work plan to reflect that GAI is structured according to three IRs to achieve an overall objective. 2 KII_08, KII_14 USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | iv framework, GAI also has supported the preparation, adoption, and implementation of local anti￾corruption plans (LAPs). Citizen participation in reporting on high-level corruption cases has increased: in 2019 courts in Serbia received 152 new cases based on the law on whistle-blower protection (compared to 122 in 2018). Out of the total caseload of 220 cases, 160 cases were finalized (compared to124 in 2018). National partners recognized that GAI’s support for the development of capacities and mechanisms to protect whistleblowers at the local level contributed to the courts in Serbia receiving more cases based on the law on whistle-blower protection. 3 The full extent to which GAI’s support is contributing to increased participation, however, is yet to be seen. ● IR2: Increase follow-up to oversight institutions’ recommendations. The evaluation finds that IR2, by prioritizing only follow-up to independent oversight institution (IOI) recommendations, did not include important dimensions of IOI organizational development and capacity strengthening. Analyzed from this broader perspective, GAI is on track to successfully enhance IOIs, most notably, the State Audit Institution (SAI) and the Commissioner for Information of Public Importance and Personal Data Protection (CPI). With assistance from GAI, SAI enhanced its performance audit capacities and sound risk assessment system to select public institutions (including LSGs) for audit, among other services. SAI’s activity report (including performance audit reports) was discussed in the relevant parliamentary committee and also in the plenary session (in June 2019). 4 Consequently, the National Assembly (NA) adopted a conclusion recommending the government to act within its powers to ensure the implementation of SAI recommendations.5 Also, GAI facilitates interaction between SAI and LSGs, ensuring appropriate post-audit assistance and implementation of recommendations on the most challenging topics (for example, public procurement processes and implementation of program budgeting). The Anti-Corruption Agency (ACA) recorded some improvements regarding the follow up on its recommendations: ACA submitted in 2019 a total of 76 requests related to a failure to promptly submit asset declarations for misdemeanor proceedings (compared to 45 in 2018). Based on ACA’s work, misdemeanor courts issued 65 convictions in 2019 (compared to 49 in 2018).6 In this context, GAI developed the prioritization criteria for selecting declarations for annual review and proposed the methodology to review assets and income declaration statements, thus, prioritizing the public officials that submitted assets declarations.7 However, challenges were identified at the ACA management level that impeded activities related to assets and income declaration, due to “inadequate software.” 8 In addition, GAI supports ACA to prepare and adopt its mid-term strategic plan and its 2020 operational plan. 3 Progress on implementation of the Action Plan for Chapter 23-The Ministry of Justice of the Republic of Serbia 4 http://www.parlament.gov.rs/aktivnosti/kalendar.998.html 5 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf, page 118 6 The ACA filed 25 criminal charges in 2019 (2018: 16) due to reasonable suspicion that a public official had not reported 27 assets or had given false information about assets intending to conceal the facts. Based on the agency’s previous work, 19 final judgments were issued. 7 The Law on Prevention of Conflict of Interest in Discharge of Public Office proscribes an obligation to public officials to report their assets to the Republic Board for resolving the conflict of interest. Under "public officials", the Law includes all persons elected, nominated or appointed to organs of the Republic of Serbia, autonomous provinces, municipalities, and towns, as well as to organs of public companies founded by the Republic, province, municipality or town. 8 KII_02 v | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV Concerning the implementation of CPI’s recommendations, improvements have not been recorded. 9 Most public authorities do not comply with the obligation and recommendations to provide data to CPI regarding citizens’ requests for information. 10 Nevertheless, GAI has been effective in supporting CPI to increase awareness and deliver educational programs on personal data protection for local administrations and entities directly or indirectly funded by their public budgets. ● IR3: Increase adjudication of corruption cases. Although corruption remains an issue of concern, the implementation of the Law on the Organization and Competences of State Organs in the Fight Against Organized Crime, Terrorism and Corruption has decreased the number of finalized high-level corruption cases (compared with the previous years). 11 In this context, the key informants recognized that "GAI played a critical role in implementing the new Law from its adoption."12 According to the national partners, "GAI provided capacity development support to specialized anti￾corruption court units (established under this law), including tools for judges, public prosecutors, and other legal specialists to implement the legal framework."13 Part of these efforts included GAI’s assistance for the design of an Electronic Register of Corruption Cases (ERCC) to enable Serbia’s enforcement and judicial institutions to collect and analyze corruption related data, thus creating basis for assessing the effectiveness of anti-corruption measures and policies. GAI sensitized stakeholders and contributed to the creation of an enabling environment for the adjudication of corruption cases, GAI also implemented the Anti-Corruption Legal Clinic at the Faculty of Law in Belgrade, providing students with “a condensed set of knowledge on the most important anti-corruption topics.”14 EQ2: What links can be determined between progress toward the activity’s objectives and the activity’s results? Due in part to GAI’s efficient implementation, adaptive management approach, and meaningful engagement with partners, preliminary results (largely at the policy and institutional levels) indicate that GAI is steadily progressing towards its objectives. ● Improving Sector Policies. The policy cycle model shows that GAI’s implementation has yielded positive achievements to date. For example, GAI brought to the policy agenda the need to further strengthen, expand, and institutionalize the protection mechanism for whistleblowers. GAI also produced tested models that could be used to prepare specific sectoral policies, including support to LSGs to implement the new Law on Public Procurement. Also, the Post-Mortem Analysis of Corruption Cases could serve as a policy tool to improve specialized anti-corruption court units and Public Prosecutors Offices (PPOs). Furthermore, GAI has effectively designed tools and instruments to facilitate policy implementation, such as the Performance Audit Policies and Procedures Manual, 9 EU Progress Report for the Republic of Serbia, https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 10 The CPI recognized in 2019 that a total of 4,321 (in 2018: 3,444) of citizens’ complaints – most of them concerning administrative silence – were well-founded and the authorities should have provided the information requested by citizens according to the law on access to information of public importance. In 1,786 or 41 percent (2018: 1,889 or 55 percent) of these well-founded cases, the authorities reported that they acted the CPI's request to provide the information to the complainants. Administrative enforcement of the decisions taken by the CPI has yet to be ensured. Ref to EU Progress Report, 2020 and the CPI Progress Reports 11 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf, page 25 12 KII_06 13 The reference has been made to the Practicum for Investigation and Adjudication of Corruption Cases and the accompanying training program. 14 KII_11 USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | vi the Plea Agreement Guidelines, and the Practicum for Investigation and Adjudication of Corruption Offences.15,16 ● Institutional Level Impact. Thus far, GAI has successfully applied tools and methods to enhance accountability and transparency of partner-institutions at national and local levels, ensuring a high degree of ownership and participation of partners. ● Management Practices. GAI has efficiently implemented activities and demonstrated flexibility when encountering operational challenges to date. GAI’s achievement of planned results can, in part, be attributed to its management mechanisms, including implementation modalities, organizational structure, and the role of each partner under GAI’s components. GAI has established a clear work plan and a regular flow of reporting, although with limited results-oriented practice. However, the Monitoring, Evaluation, and Learning Plan (MELP) was not sufficient to effectively monitor GAI. Additionally, GAI-collected data lacks sex-disaggregation and gender sensitivity. EQ3: What other areas of interventions could fit under the SOW beyond what is already planned? GAI’s existing SOW is well-defined and fully aligned with EU accession governance priorities in Serbia and the needs of the partner institutions. The partners recognized the importance of their organizational development and improvement of core services, emphasizing “that GAI assistance and its tested services need to continue, with focus on root causes of challenges.”17 However, there are some potential areas for additional GAI involvement at the sectoral level (for example, the overall anti-corruption strategy), institutional level (for example, strengthening local anti-corruption councils’ capacity to monitor plan implementation or strengthening mechanisms for private sector whistle-blowers), and individual level (for example, addressing the needs of youth or enhancing understanding of participation benefits for citizens) to complement already planned and implemented activities. EQ4: What are the interventions with greatest traction, unintended outcomes, and/or lessons learned from USAID assistance that the Mission can use in decision making for future programming? GAI has consistently responded to Serbia’s national capacity development needs and is steadily strengthening national and local level capacity to improve government accountability. Future progress and its sustainability, however, rests on the continued commitment and capacity of Serbia’s governance system, institutions, and citizens. GAI has produced unintended results in different areas of its intervention. For example, the training programs favored horizontal knowledge sharing among the employees from different LSGs, and among IOIs. With support from GAI, stakeholders have been exposed to international best practices and models tested in the United States. This experience, together with established partnerships, generated additional readiness and willingness among participating institutions to propose reviews of both policy and legal documents to better reflect principles of transparency and accountability. The evaluation team identified lessons learned from GAI implementation. The need to provide focused expertise and policy advice, together with long-term support, remain crucial to ensuring sustainable organizational development of IOIs. Still, without follow-up on findings and recommendations, primarily 15 Prepared with the purpose of harmonizing ACA’s filings with the courts’ consideration of plea agreements. 16 The Practicum provided guidance on the investigation and prosecution of corruption cases. 17 The participants in the survey gave priority to capacity development (30 percent), inter-institutional cooperation for anti-corruption (25 percent), and the improvement of delivery of services at the local level (20 percent). vii | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV through the involvement of NA, these efforts do not contribute to systemic change and establishment of good governance. At the local level, authorities could fail in their efforts to enhance participation if they cannot ensure citizens’ trust through genuine program implementation and trust promotion. In this context, the need to ensure local ownership and commitment of LSGs and their leaders are prerequisites to sustainable reform efforts. Continued results-focused monitoring should be in place to ensure that targets are met and results reported to citizens. GAI also generated a number of best practices, such as the use of the LTI, engaging youth in budgeting processes in Sombor, supporting SAI to introduce performance audits, and strengthening cooperation between CPI and LSGs.18 EQ5: Did the activity affect gender equality within the area(s) of focus? To date, GAI has not affected gender equality. This can largely be attributed to a lack of consideration of gender issues during program design and insufficient attention to gender mainstreaming throughout implementation. The Activity’s gender integration efforts were primarily a formality, notably in the documentation of the sex of the Activity’s training participants. RECOMMENDATIONS Through analysis of primary and secondary data, the evaluation team identified achievements and challenges during the implementation of GAI and explored possible responses to these findings. The evaluation team formulated the following main recommendations for USAID and GAI’s consideration: Recommendation 1. The evaluation team recommends that GAI focus on the next stage of strengthening government accountability at national and local levels, which should include helping SAI, CPI, ACA, specialized anti-corruption court units, and LSGs implement organizational strategic priorities and follow up on improvement of services. This could involve helping to build capacities, systems, and connections among the key partners and institutions. Recommendation 2. Development of research and analytical capacities of ACA staff remains a dominant priority for this organization and GAI is well-positioned to continue building on the existing achievements and extend its technical assistance. However, if ACA's leadership continues to resist implementing its oversight mandate and the new procedures and policies that GAI supports, GAI should reconsider supporting ACA. Recommendation 3. GAI should remain flexible and responsive to the needs and priorities of the good governance framework. Concerning responsiveness, emerging priorities include: ● Systemic Level. To engage more actively in the policy making process related to anti-corruption, transparency, and accountability, GAI should codify and summarize knowledge in the core areas of its work and share it with the government as the basis for or evidence to create a more favorable legal and policy environment in this area. GAI also should provide specific support to IOIs to respond to the corruption risks and challenges posed by COVID-19. ● Institutional Level. GAI should: 1) continue support to local authorities – priorities could include: a) designing sound and realistic monitoring system for LAPs; b) enhancing the capacities of local independent anti-corruption bodies, in charge of LAPs monitoring; c) enhancing the capacities to implement recommendations to improve budgeting process and local finances; and d) enhancing 18 Best practices reflected several criteria such as the effects on participating institutions, the durability of established mechanisms, and the partnerships. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | viii strategic communication skills of local officials in senior positions; 2) explore opportunities to engage with public enterprises, including public utility companies, and enhance their transparency, in line with the internal rules, legal provisions and requirements; 3) expand the reach of and embrace private sector interest, their roles and responsibilities in advancing good governance, and to fight corruption efforts in the country; 4) strengthen and establish whistleblower protection mechanisms for the private sector, enabling them to report on corruption; 5) strengthen capacities of the NA and its relevant committees to read, understand and comment on the oversight institution reports, and present findings and recommendations; 6) strengthen the technical capacity of prosecutors; and 7) explore opportunities to support the network of DPOs.19 ● Individual Level. GAI should: 1) invest in additional efforts to reach youth, offering capacity development programs to prevent corruption, including anti-corruption clinic models, on-line training programs, and the use of social networks and on-line discussion forums; 2) prepare and implement programs on anti-corruption framework for the journalists; and 3) increase “demand” for participation, by working with the public to create broader interest and willingness to take part in local decision-making processes. Recommendation 4. GAI should define appropriate qualitative and quantitative indicators to enable GAI to adequately measure progress under components and capture GAI’s progress towards its objective.20 The evaluation team recommends that GAI provide a well-balanced combination of qualitative and quantitative indicators to capture changes and results attributable to the Activity, using national indicators and targets to the extent possible. The evaluation team also recommends including gender-sensitive indicators with a focus on “gender transformation,” identifying assumptions and risks, and implementing a sound risk management system. Recommendation 5. In addition to including gender-sensitive indicators, GAI should consider options to better integrate gender priorities in its activities and the overall anti-corruption framework. Additionally, GAI should consider activities and approaches that go beyond gender participation. Recommendation 6. GAI should continue implementing its systemic approach to capacity development, following needs assessments of the main stakeholders and partners. Considering the current stage of the governance systems, structures, and capacities of partner institutions in Serbia, GAI should consider a long-term, needs-based capacity development approach. The evaluation team recommends that GAI design a comprehensive training evaluation approach based on the Kirkpatrick model, assessing four levels of learning: 1) participant satisfaction with the training; 2) immediate change in individual knowledge and skills; 3) change in individual performance back in the workplace; and 4) change in the overall performance of the institution. Finally, GAI should prepare a plan to hand-over training programs to the relevant national institutions and prioritize preparing them to assume these responsibilities. 19 GAI’s role has been to support analysis of the budgeting processes and prepare recommendations for improvements; however, the Action did not provide follow up support, to implement recommendations. 20 For example, indicators could not adequately measure GAI’s performance in the areas such as right to information, civic participation, and complaint mechanisms. GAI did not introduce appropriate indicators to measure development of capacities of the target groups or the effects on anti-corruption education programs on the selected youth. Also, GAI did not measure the effects of awareness campaigns and assess if the targeted audience has enhanced understanding of a specific topic. The MELP also did not include more specific indicators related to assistance to CPI and LSGs. 1 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV 1. INTRODUCTION 1.1 EVALUATION PURPOSE The United States Agency for International Development in Serbia (USAID/Serbia) recognized the challenges facing accountable and representative governance in Serbia, due in part to the lack of adequate checks on central authority and sufficient opportunities for citizens to hold their government and elected officials accountable, and, accordingly, funded the Government Accountability Initiative (GAI). GAI is a four-year activity that seeks to strengthen the capacities and connections of key Serbian government institutions and stakeholders, resulting in increased government accountability at the national and local levels. To achieve this objective, the Activity is structured according to three intermediary results (IRs): 1) increase public participation in local government decision-making and oversight; 2) increase follow-up to oversight institutions’ recommendations; and 3) increase adjudication of corruption cases. This $8.5 million, four-year Activity started in February 2018 and will end in February 2022. More background on GAI is included in Annex B. USAID/Serbia commissioned IMPAQ International, LLC (IMPAQ) to conduct a mid-term performance evaluation of GAI. The evaluation provides the Mission with a fact-based and independent assessment of the Activity’s performance, ensuring an in-depth analysis of the Activity’s progress in achieving established targets. The evaluation also presents the evaluation team’s analysis of the recognized gaps between expected and actual performance and identifies source(s) of these gaps by considering the relevance of the design, project management, changes in operating environment, changes in context, success factors, and the Activity’s achievement of intended results and goals. The evaluation team conducted a mixed-methods study to assess the Activity’s impact and the likely sustainability of results. In parallel, the evaluation team identified relevant lessons learned with a view to strengthen the Activity’s remaining period of performance and inform future programming. In conducting the mid-term performance evaluation, the team focused on the following five evaluation questions (EQs) specified in the evaluation scope of work (SOW): 1. To what extent is the program on track to accomplish the inputs, outputs, and likely outcomes expected under all three intermediate results of the activity?21 2. What links can be determined between progress toward the Activity’s objectives and the Activity’s results? 3. What other areas of interventions could fit under the SOW beyond what is already planned? 4. What are the interventions with the greatest traction, unintended outcomes and/or lessons learned from USAID assistance that the Mission can use in making decisions for future programming? 5. Did the Activity affect gender equality within the area(s) of focus? 1.2 EVALUATION METHODOLOGY To conduct the mid-term performance evaluation of GAI, the evaluation team used a theory-based and utilization-focused approach, incorporating complementary qualitative and quantitative methods of data collection and analysis utilizing online and in-person tools tailored for different stakeholder groups. The team’s methodology and evaluation quality standards drew on USAID Evaluation Policy and the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD DAC).22 Before collecting data, the evaluation team conducted a thorough review of 21 The evaluation team changed the original wording of EQ1 included in its statement of work and the work plan to reflect that GAI is structured according to three IRs to achieve an overall objective. 22 USAID Evaluation Policy (January 2011, updated October 2016). More details available at: https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 2 documentation related to the EQs to gain an in-depth understanding of GAI activities and to assess the availability and quality of secondary data to answer the EQs. DATA COLLECTION. From September 23 through October 13, 2020, the evaluation team conducted primary and secondary research in Serbia, collecting both qualitative and quantitative data, to inform each of the EQs. The primary research included Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs), for a combined total of 66 participants (30 male and 36 female) representing 24 organizations. KII and FGD participants were identified from the full list of partners received from GAI, based on their involvement, knowledge, and awareness of the Activity. The evaluation team summarizes its data collection methods below: ● Key Informant Interviews. The evaluation team conducted 34 online and 10 in-person KIIs with 1) USAID and GAI implementing partners and grantees; 2) central Government of Serbia representatives; 3) Local Self-Government (LSG) representatives; and, 4) Civil Society Organizations (CSOs), business entities, and other experts. ● Focus Group Discussions. The evaluation team conducted four online FGDs with a total of 22 participants from four LSGs with which GAI-USAID signed memoranda of cooperation: the municipalities of Vrnjacka Banja, and cities of Sombor, Šabac, and Novi Pazar. ● Online Survey. The evaluation team disseminated an online survey to Activity participants (collecting 72 responses), including LSG representatives who received technical assistance from GAI and representatives of Independent Oversight Institutions (IOIs) and Public Prosecutor’s Office staff who participated in GAI’s trainings. 23,24 DATA ANALYSIS. The evaluation team’s methodology was based on contribution analysis, enabling the team to evaluate credible causal claims about interventions and their results. While answering the main EQs, the evaluation team used the OECD DAC criteria, namely, relevance, coherence, effectiveness, efficiency, impact, and sustainability. Additionally, in addressing all EQs, and specifically EQ5, the evaluation team assessed the extent to which, if any, GAI has integrated gender mainstreaming within the intervention, looking across all GAI components and activities. KNOWN LIMITATIONS. The evaluation team identified several challenges and limitations in preparing for and conducting the evaluation, including COVID-19, subjectivity, and semi-structured KIIs, inconsistent accounts, the situation in Serbia after the 2020 local elections, and ethical considerations. Annex A presents the Evaluation Statement of Work, which includes the EQs. 2. FINDINGS 2.1 EVALUATION QUESTION 1: GAI ACCOMPLISHMENTS TO DATE To what extent is the program on track to accomplish the inputs, outputs, and likely outcomes expected under all three intermediate results of the activity? In response to EQ1, the evaluation team sought to assess the performance of GAI at its mid-point and determine the extent to which the program is on track to accomplish the inputs, outputs, and outcomes expected under each of its three IRs. Utilizing primary and secondary data, the evaluation team assessed the progress and performance that GAI has made toward increasing public participation in decision 23 The response rate has been 57.2 percent as a total of 126 representatives of the stakeholder institutions have been invited to participate in the survey. 24 Best practices reflected several criteria such as the effects on participating institutions, the durability of established mechanisms, and the partnerships. 3 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV making and oversight to boost transparency and accountability, strengthening the capacity of independent oversight institutions to increase the uptake of recommendations, and, increasing adjudication of corruption cases. The team assessed GAI’s progress in strengthening the capacities and connections of the key stakeholders to increase governmental accountability at both the national and local levels. The evaluation team discusses its findings related to the Activity’s objectives and its three IRs below, and provides a more detailed analysis of the related performance indicators in Annex C. 2.1.1.1 TO STRENGTHEN THE CAPACITIES AND CONNECTIONS OF THE KEY STAKEHOLDERS TO INCREASE GOVERNMENT ACCOUNTABILITY AT NATIONAL AND LOCAL LEVELS Despite GAI’s results to enhance the capacities of oversight institutions, these efforts have not led to any measurable improvements in government accountability at the national level.25 To date, GAI has made progress towards its indicators and has contributed to strengthening the capacities of participating oversight institutions to more effectively hold the government accountable. 26 The primary data sources indicated results were achieved for accountable and transparent governance at the local level, ensured by GAI’s assistance to prepare and adopt local anti-corruption plans and establish independent monitoring bodies, improve budgeting processes and delivery of services, and strengthen mechanisms to protect whistleblowers. Still, the oversight of the “executive remains rather formalistic and, thus, the democratic accountability of the executive branch remains weak.” 27 The following discussion analyzes GAI’s objective and its indicators. GAI’s objective has shown some challenges related to its scope and focus: it is neither results-focused nor “specific, measurable, adequate, realistic, and time-bound.” The objective is also compound, established by merging two independent components related to improved capacities and connections (of institutions) and government accountability (at national and local levels). Despite these weaknesses, the objective has been integrated in GAI intervention logic and its chain of results. 28 The indicators at the objective-level, OI-I – World Justice Project Open Government Index (WJP-OGI), OI-2 – Local Transparency Index (LTI), and O1-3 – Good Governance Training, have been partially relevant to capture and measure progress towards the objective. ● O1-1. WJP OPEN GOVERNMENT INDEX. WJP-OGI is a third-party indicator published by the World Justice Project to measure “government openness based on the general public’s experience and perceptions.”29 While the Index is focused on measuring government openness, a desired impact of GAI’s intervention, the evaluation team recognized the limits of WJP-OGI to adequately capture GAI’s direct contribution or measure GAI’s progress towards the stated objective. The scope of the WJP-OGI exceeds a single project, and only broad contribution links can be established. Yet, due to its complexity and ability to monitor desired change at the impact level, the evaluation team recommends using WJP-OGI for country-level assessment, as it promotes broader systematic thinking and collaboration with other interventions. In analyzing the WJP-OGI over the past five years, the evaluation team observed a negative trend, declining from 0.51 in 2015 (Serbia was ranked 61 out of 102 countries) to 0.47 in 2020 (ranked 76 out of 128 countries). Exhibit 1 displays the WJP Government Index for Serbia for the last five years. 25 The project reports and other project products, supported by evidence and sources indicated in the GAI MELP, and triangulated with the primary data collected through interviews with stakeholders created the basis to assess and validate progress. 26 For example, SAI and to some extent the specialized anti-corruption court units. 27 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 28 GAI objective, which accurately describes the “intended changes in development conditions resulting from the joint work of GAI and key stakeholders,” OECD DAC and KII_01 29 https://worldjusticeproject.org/our-work/research-and-data/wjp-open-government-index/wjp-open-government￾index-methodology USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 4 The WJP-OGI for Serbia and its declining value illustrates the governance challenges in Serbia, including the government’s lack of genuine commitment, and the lack of capacity of governance structures, especially at the national level, to work on challenges under the good governance framework. Exhibit 1: The WJP Open Government Index for Serbia, 2015-2020 ● OI-2. LOCAL TRANSPARENCY INDEX. The LTI, a third-party indicator published by Transparency Serbia, provides a “direct insight into the functioning of the service centers and premises of the local administrations” to measure transparency levels across the country.30 In analyzing the 2020 LTI, the evaluation team found higher LTI scores among the LSGs that participated in GAI’s intervention than those that did not.31 Twelve out of the 13 participating LSGs (92 percent) recorded an increase on the LTI scale compared to the previous year, meeting or exceeding GAI’s annual targets. The evaluation team identified the Municipality of Dimitrovgrad as the sole LSG that did not meet its target due to organizational changes that caused delays in implementation of planned activities. The first group of LSGs to receive GAI’s intervention scored an average of 59 out of 100 and the second 53. These values are considerably above the national average of 46.32 Exhibit 2 shows the LTI in 2019 and 2020 for GAI-supported LSGs. EXHIBIT 2: LTI FOR GAI-SUPPORTED LOCAL SELF-GOVERNMENTS GAI LOCAL-SELF GOVERNMENTS LTI 2019 LTI 2020 TARGET ACTUAL TARGET ACTUAL First Group Dimitrovgrad 49 38 52 51 Raška 65 44 68 47 Šabac 44 41 47 57 Sjenica 36 34 39 39 Sombor 55 52 58 80 Vranje 65 60 68 73 Vrnjačka Banja 45 62 48 63 30 https://www.transparentnost.org.rs/index.php/en/research-on-corruption/lti 31 The Local Transparency Index for 2020 was conducted in the first quarter of 2020, based on research conducted on 145 LSGs (cities and municipalities) and 25 city municipalities throughout Serbia. 32 The national average LTI for 2020 is 46 is an improvement in comparison to the average 2019 LTI score of 40 for 6 points. Source: WJP Rule of Law Index® 0.47 0.47 0.49 0.56 0.51 2020 2019 2017 - 2018 2016 2015 5 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV Second Group Stari Grad, City Municipality N/A 23 26 28 Kragujevac N/A 43 45 55 Niš N/A 34 37 46 Novi Pazar N/A 66 69 82 Sremska Mitrovica N/A 45 48 51 Žabalj N/A 36 39 55 Source: LTI-Transparency Serbia and GAI MELP ● OI-3. GOOD GOVERNANCE PRINCIPLES AND ACCOUNTABILITY TRAINING. GAI monitors the number of public-sector employees and representatives of CSOs trained in the core areas of good governance. The team’s analysis of available data shows that GAI has provided capacity development support for the planned number of trainings and participants.33 For example, 294 civil servants and local officials were trained on personal data protection from December 2019 to February 2020.34 However, the cumulative data on the number of employees and representatives trained on good governance have not been reported regularly in GAI quarterly reports 35 and available data are not disaggregated by sex, limiting the team’s ability to analyze and validate achievements against targets. 36 2.1.2 IR1. INCREASED PUBLIC PARTICIPATION IN LOCAL GOVERNMENT DECISION-MAKING AND OVERSIGHT Authorities from participating municipalities recognized that support from GAI has been effective in enhancing public participation, especially concerning the preparation of municipal budgets. 37 Namely, GAI’s efforts to develop capacities and create a culture of openness, engagement, and public information sharing contributed to an enabling environment for public participation in GAI municipalities.38 Interviews with representatives of LSGs confirmed the team’s initial findings from the desk review that GAI has provided support in the preparation, adoption, and implementation of local anti-corruption plans (LAPs).39 GAI has initiated capacity development of local independent bodies to monitor LAPs, through a basic training program that presented their role within the local anti-corruption framework. However, the members of these independent bodies highlighted the need to “enhance their capacities 33 KII_02, KII_15 34 GAI Quarterly Report, February 6- April 30, 2020 35 The Q3 2019 report highlighted that “the target value for trainings in FY 2019 was set at 690 trainees and the number was surpassed, as GAI trained over 1,400 persons in 57 different sessions” The evaluation team was not able to review the Q3 2020 report, which included similar data. 36 GAI provides this data to the overall the US Mission Performance Plan and Report for Serbia. However, the evaluation team did not receive these documents during the desk review and was not able to include them in its analysis. 37 KII_08, KII_14 38 1 st Group included seven local governments (from November 2018): Sombor, Vranje, Vrnjacka Banja, Šabac, Dimitrovgrad, Raška and Sjenica; 2nd included six local governments (November 2019): Niš, Kragujevac, Novi Pazar, Sremska Mitrovica, Žabalj and Belgrade City Municipality of Stari Grad. 39 KII_010, KII_014, KII_016, KII_020 and GAI Quarterly Reports USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 6 for effective monitoring and reporting on LAP implementation,” which is beyond GAI’s current and planned scope.40,41 The evaluation team found that GAI’s systemic support for enhancing transparency of LSG units has been well-defined and effectively implemented, achieving planned results. As indicated in the Theory of Change (TOC), GAI identified specific issues relevant to these LSGs to increase their transparency and accountability (as measured by the LTI). Website improvements in Raška and Vranje, along with the preparation of a strategic communication strategy for the city of Niš, are some of the most notable results.42 In the context of enhanced transparency and integrity, the partners recognized that “GAI played a key role in establishing systems and mechanisms for the protection of whistleblowers.”43 The partners from the City of Šabac, for example, recognized that GAI assistance resulted in the creation of a city-level whistleblower protection system with a designated, trained person in the city’s administration.44 Representatives of the participating LSGs stated that GAI has provided technical support, which contributed to improved public engagement in the local decision-making process. Stakeholders from the cities of Šabac and Sombor recognized that support from GAI enabled them to develop and adopt a model for youth participation in budgeting processes, increasing self-confidence, and resulting in youth becoming more actively involved in local affairs.45 The municipalities of Vrnjacka Banja and Sremska Mitrovica recognized that GAI assistance enhanced their abilities for the participatory planning and the budgeting processes. In this context, stakeholders stated that GAI has promoted the culture of openness, engagement, and information sharing on public spending.46 The municipal representatives recognized that support from GAI “enabled them to establish oversight and control mechanisms over public spending, meeting their needs to establish more transparent and effective functioning of local authorities.”47 2.1.3 IR2: INCREASED FOLLOW-UP TO OVERSIGHT INSTITUTIONS’ RECOMMENDATIONS Under IR2, GAI’s interventions are aimed, in part, to increase the likelihood that oversight institutions’ recommendations to public bodies and the NA lead to concrete, meaningful change in the Serbian governance environment. However, the evaluation finds that IR2 has been formulated narrowly and inadequately to capture the actual GAI efforts. By prioritizing only the follow-up to oversight institutions’ recommendations, IR2 did not include a large area of organizational development and strengthening the capacity of oversight institutions as the central part of GAI assistance. 48 Analyzed from a broader perspective, GAI is on track to successfully enhance oversight institutions, most notably, the evaluation team observed these achievements in relation to planned support to the 40 KII_010, FGD_03 41 KII_010, FGD_03, KII_23 42 KII_24 43 KII_9, KII_24 44 The Rulebook on Internal Whistleblower Protection in the City Administration of the City of Šabac https://sabac.rs/wp-content/uploads/2020/06/pravilnik-o-postupku-unutrasnjeg-uzbunjivanja.doc 45 KII_14, KII_9 and KII_19 and FGD_04 46 KII_08, KII_19, KII_14 47 FGD_01 48 Under institutional development, it is recommended to answer, “development for what,” “development for why,” and “development for whom.” “Capacity for development- new solutions to old problems,” Edited by: Sakiko Fukuda-Parr, Carlos Lopes and Khalid Malik, United Nations Development Programme, Earthscan Publications Ltd, London and Sterling, Virginia, 2004 7 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV State Audit Institution (SAI) and Commissioner for Information of Public Importance and Personal Data Protection (CPI). Implementation of SAI recommendations has enhanced, with support from GAI. The last SAI activity report, together with performance audit reports, was discussed in the relevant parliamentary committee and also in the plenary in June 2019. 49 Following these sessions, the National Assembly (NA) adopted a conclusion recommending the government to act within its powers to ensure the implementation of SAI recommendations.50 In this context, SAI has recognized the assistance from GAI and enhanced its services, primarily performance audit capacities and sound risk assessment system to select public institutions (including LSGs) for audit. SAI management stated that support from GAI enabled them to assess whether public entities are achieving economic, efficient, and effective use of available resources. The number of external audit reports in 2019 reached a total of 444, exceeding by more than ten percent the number of reports that SAI released in 2018, which was slightly above 400.51,52 GAI also helped SAI establish 13 teams to conduct performance audits and followed sound risk assessment criteria for selecting institutions for audits (including LSGs).53,54 GAI also facilitates interaction between SAI and LSGs, ensuring appropriate post-audit assistance and implementation of recommendations on the most challenging topics (for example, public procurement process and implementation of program budgeting). Similarly, the evaluation team found that GAI facilitated communication between SAI and LSGs, including assistance to address the findings from the post-audit reports. SAI representatives stated that as a result of GAI assistance, some of the most common issues from audit reports have been clarified, and challenges solved, concluding that municipalities have been more effective in implementing recommendations from the audit reports.55 The Anti-Corruption Agency (ACA) recorded some improvements regarding the follow up on its recommendations. The ACA submitted in 2019 a total of 76 (compared to 45 in 2018) requests related to a failure to promptly submit asset declarations for misdemeanor proceedings. Based on the ACA’s work, misdemeanor courts issued 65 convictions in 2019,compared to 49 in 2018.56 Assistance from GAI has been relevant, resulting in prioritization criteria for selecting declarations for annual review and a proposed methodology to review assets and income declaration statements, thus prioritizing the 49 http://www.parlament.gov.rs/aktivnosti/kalendar.998.html 50 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf, page 118 51 SAI 2019- Annual Activity Report, http://www.dri.rs/documents/annual-activity-reports.181.html 52 SAI 2018 Annual Report- available at http://www.dri.rs/php/document/download/1549/3. In 2014, it was only 135 (SAI 2015 Annual Report- available at http://www.dri.rs/php/document/download/724/3) 53 The risk assessments fall in three key areas: i) the calculation of salaries; ii) public procurement and iii) assumed obligations above approved appropriations. 54 SAI representatives stated that GAI technical assistance to introduce and implement performance audits, as well as the implementation of recommendations in the targeted institutions, generated financial benefit to the public budget. 55 KII_07. The additional reference could be SAI reports: http://www.dri.rs/revizije/izvestaji-o-reviziji.136.html and http://www.dri.rs/dokumenti/godisnji-izvestaji-o-radu.93.html. 56 The ACA filed 25 criminal charges in 2019 (2018: 16) due to reasonable suspicion that a public official had not reported 27 assets or had given false information about assets intending to conceal the facts. Based on the agency’s previous work, 19 final judgments were issued. “We have received critical support from GAI to develop capacities for performance audits. These capacities have contributed to ensure more than 800 million dinars of financial benefits through SAI audits.” Interview with the representatives of SAI USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 8 public officials that submitted assets declarations.57 GAI also assisted in developing a prioritization framework scheme that ACA implemented in a limited fashion. Namely, ACA officials declared they would not be able to continue with the process due to “inadequate software support.”58 GAI has been working to improve cooperation between ACA and misdemeanor courts, focusing on harmonizing the review of plea agreements in misdemeanor filing. Although necessary, this effort has had a limited impact on ACA's service delivery capacities. In addition, GAI supported ACA to prepare and adopt its mid￾term strategic plan together with the operational plan for 2020. The results of the assistance to CPI could be challenging to interpret. The results of GAI efforts could be associated with increased number of citizens’ complaints. In 2019, a total of 4,321 complaints had been submitted, compared to 3,444 in 2018, most of them concerning administrative silence. Most were well-founded and the authorities should have provided the information requested by citizens according to the Law on Access to Information of Public Importance. The Activity delivered two-fold assistance. First, GAI supported the organizational development of CPI through exposure to international best practices regarding access to information and personal data protection.59 Second, in partnership with CPI, GAI delivered awareness and educational programs to different stakeholders. These programs included, for example, training for local administrations and entities (for example, utility companies or museums) directly or indirectly funded by public budgets to implement and follow up on the new requirements deriving from the Law on Personal Data Protection. However, concerning the implementation of CPI’s recommendations, improvements have not been recorded and the majority of public authorities do not comply with the obligation and recommendations to provide data to CPI recommendations regarding citizens’ requests for information. 60 In 2019, for 1,786, or 41 percent (compared to 1,889, or 55 percent, in 2018), of these well-founded cases, the authorities reported that they acted on CPI's request to provide the information to the complainants. Administrative enforcement of the decisions taken by CPI has yet to be ensured. 61 2.1.4 IR3: INCREASED ADJUDICATION OF CORRUPTION CASES The changes brought by the Law on the Organization and Competences of State Organs in the Fight Against Organized Crime, Terrorism, and Corruption produced some results concerning adjudication of corruption cases. Although corruption remains an issue of concern, the implementation of the Law has decreased the number of finalized high-level corruption cases, when compared with the previous years. 62 In this context, the key informants recognized that "GAI played a critical role in implementing the new Law from its adoption."63 According to the national 57 The Law on Prevention of Conflict of Interest in Discharge of Public Office proscribes an obligation to public officials to report their assets to the Republic Board for resolving the conflict of interest. Under "public officials", the Law includes all persons elected, nominated or appointed to organs of the Republic of Serbia, autonomous provinces, municipalities and towns, as well as to organs of public companies founded by the Republic, province, municipality or town. 58 KII_02 59 The workshop has been organized around the availability of and access to information of public importance during the time of COVID-19 pandemic, showing parallels between Serbia, the region and Europe. KII_015 60 The CPI recognized in 2019 that a total of 4,321 (in 2018: 3,444) of citizens’ complaints – most of them concerning administrative silence – were well-founded and the authorities should have provided the information requested by citizens according to the law on access to information of public importance. In 1,786 or 41 percent (2018: 1,889 or 55 percent) of these well-founded cases, the authorities reported that they acted the CPI's request to provide the information to the complainants. Administrative enforcement of the decisions taken by the CPI has yet to be ensured. Ref to EU Progress Report, 2020 and the CPI Progress Reports 61 Ref to EU Progress Report, 2020 and the CPI Progress Reports 62 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf, page 25 63 KII_06 9 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV partners, "GAI provided capacity development support, including tools for judges, public prosecutors, and other legal specialists to implement the legal framework." The GAI-produced Practicum for Investigation and Adjudication of Corruption Cases and its accompanying training program were referenced by the partners. The partners stated that support from GAI enabled them to "integrate and use financial forensic expertise in [the] adjudication of corruption cases, misfeasance in public procurement, and case law harmonization.”64 All these capacity development efforts have contributed, according to the national partners, to more effective adjudication of corruption cases."65 GAI responded to the need for a comprehensive approach for the analysis of corruption-related data through the establishment of the Electronic Register of Corruption Cases (ERCC). The ERCC should “enable a unified interface for specialized anti-corruption court unit and PPO data.” 66 The procurement process for the ERCC has been launched. Once implemented, it is expected that this tool will improve the analysis of information from existing case management systems and the monitoring and adjudication of corruption cases.67 GAI successfully tested the model to develop the capacities of the students/young professionals on the main anti-corruption related topics. Under IR3, GAI implemented the Anti￾Corruption Legal Clinic at the Faculty of Law in Belgrade, as a single semester program for a select group of 20 students completing their final year of studies. The model tested through the Anti￾Corruption Legal Clinic successfully affected youth engagement in anti-corruption activities. Although it only included a limited number of participants, the program was highly effective because a competent group of future professionals completed it.68 2.2 EVALUATION QUESTION 2: LINKING THE ACTIVITY’S OBJECTIVES AND RESULTS What links can be determined between progress towards the Activity’s objectives and the Activity’s results? EQ2 seeks to identify the links between GAI’s progress towards its objectives and its results at its mid￾way point. Drawing on the evaluation team’s desk review and interviews with key informants, the evaluation team found key links between GAI’s main achievements and its possible effects at the policy (systemic) and institutional levels. GAI has been successful in ensuring progress toward the Activity’s objectives, specifically regarding its interventions to improve sector policies, enhance administrative cultures, and strengthen management practices. 2.2.1 IMPROVING SECTOR POLICIES GAI successfully addressed accountability issues by setting the basis for policies that ultimately will increase government accountability at national and local levels. The evaluation team used the policy cycle model as an analytical tool to assess and justify the 64 KII_06 65 KII 66 KII_029, KII_06 67 KII_029 68 KII_011 The Practicum for Investigation and Adjudication of Corruption Offences represents an initial step in promoting efficient and innovative methods in fighting corruption and adopting good practices through implementation. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 10 links between the results of the Activity and its objective at the policy level.69 ● POLICY DECISIONS. The evaluation team found that GAI, by codifying its knowledge and experience, was able to influence “definition and agenda setting” in the policy cycle.70 GAI confirmed that the achievement of a more accountable governance system is a complex, slow process, with challenges at different levels. While GAI has helped set the policy agenda, it has not resulted in any new policy decisions. A brief reflection on the capacities of stakeholders, for example, showed that participating representatives of LSGs and public enterprises have considerably different levels of skills and knowledge.71 This indicated the need to prioritize systematic follow-up and continued capacity development. While GAI offered some tools to address these differences, the stakeholders expressed a need for additional technical assistance. 72 GAI also raised to the sectoral policy agenda the need to further strengthen and institutionalize protection mechanism for whistleblowers, especially in LSGs and public enterprises.73 However, some of identified challenges that affect whistleblower protections have been poorly understood.74 Additionally, there are limited capacities to implement the Law on whistleblower protection at the local level, a lack of appropriate models to establish internal whistleblowing systems, and a lack of complaints management applications. ● POLICY DEVELOPMENT. The primary and secondary data collected by the evaluation team indicated that GAI used some solutions that were useful to enhance transparency and accountability. These tested and positive solutions could be codified and used to prepare specific sectoral policies. For example, GAI supported LSGs in Serbia to prepare internal acts, procedures, and regulations, to implement the new Law on Public Procurement. The Activity published recommendations based on international best practices to improve the internal rulebook on whistleblowing in LSGs.75 GAI also developed a draft model for cooperation between high school student parliaments and the city administration, describing roles, responsibilities, processes, and a timeframe to facilitate regular annual consultation with high schools.76 GAI supported ACA with the development of three model acts regulating donations, conflict of interest, and complaints. GAI also prepared two analytical documents as the basis for the unified and integrative ERCC.77 These analytical documents provided an overview of the existing case management systems (AVP and SAPO) to help members of the Working Group for the ERCC 69 Reference to Guy Peters “Policy Making Cycle- steps and approaches,” from “Capacity for Policy- how can we do it better” (prepared by Tomislav Novovic, UNDP Serbia, 2010) 70 Priority setting is inherently complex and involves making choices about what issues are most pertinent and time sensitive. After identifying priority issues, government and other governance actors must set the agenda to address the issues. "Framing" priority issues involves defining a problem in terms that can mobilize political support to find a solution. More available at Jon Pierre, B. Guy Peters “Governance, Politics and the State,” Macmillan, USA, 2016 71 The interviews with different LSGs additionally confirmed some differences among LSGs regarding their capacities. 72 KII with the national stakeholders KII_07, KII_06, KII_14. Also, details are provided under the part Decision on instruments and implementation. 73 KII notes KII_15 74 KII notes KII_15, FGD_03 75 Recommendations for Rulebooks on Internal Whistleblowing Procedures for Local self-governments, https://www.odgovornavlast.rs/wp-content/uploads/2019/09/USAID-GAI-and-Pistaljka_Recommendations-for￾rulebooks-on-Whistleblowing-Framework-for-LSGs-1.pdf 76 KII notes KII_15 77 These documents are the Automated Case Management System for Courts (AVP) and the Software Application to the Prosecutor’s Office (SAPO) Database model and Data Dictionary along with a report on the compatibility of catalogues in both case management systems. 11 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV determine and select the appropriate set of data to be used in the ERCC database.78 The evaluation team found that the existence of a functional ERCC is inextricably linked with evidence-based policymaking. The partners stated that the ERCC “provides solid and reliable information on the effectiveness of implemented measures, while also enabling monitoring of activities.”79 Regarding the upcoming amendment of the Law on Free Access to Information, GAI prepared two analytical documents with recommendations for the government’s consideration. GAI prepared the Post-Mortem Analysis of Corruption Cases for judges and advisors from specialized anti-corruption court units.80 The team’s analysis of primary and secondary data confirmed that this document provided a comprehensive learning-oriented analysis of select corruption cases and a basis for evidence-based policy on capacity building and improvement for specialized anti-corruption court units and public prosecutor’s offices.81,82 ● DECISION ON INSTRUMENTS AND IMPLEMENTATION. GAI has designed tools and instruments that, according to the stakeholders, facilitated policy implementation, including the following: a) The Performance Audit Policies and Procedures Manual outlines the methodology to conduct a performance audit, including audit design, creation and use of sampling, application of audit criteria, questionnaire design, content analysis, and data reliability during audits. b) The Plea Agreement Guidelines harmonizes ACA’s filings with the courts’ consideration of plea agreements.83 c) The Practicum for Investigation and Adjudication of Corruption Offences provides guidance and practical advice to investigate and prosecute criminal offenses that include elements of corruption.84 This publication represents an initial step in promoting efficient and innovative methods to fight corruption and adopting good practices through implementation.85 In addition, GAI prepared a Practical Guide and FAQs for LAPs.86 d) The Conflict of Interest Manual serves as a guide for officials to use in managing ethical standards and conflicts of interest in the workplace. 87 78 Members of the group included representatives of the Ministry of Justice (including representatives of the Justice Division of the MOJ), Public Prosecutor’s Office, the Court of Cassation; GAI experts facilitated meetings of the WG). 79 KII notes, especially KII_29, KII_06 80 Details are available at https://www.4digitsconsulting.com/post-mortem-analysis-of-anti-corruption-cases/ 81 GAI Quarterly Report- Q1 2020, also KII_06, KII_12 82 Key informant interviews- KII_01 and KII_06, KII_12 83 These guidelines should enable agreement between ACA and Misdemeanor Courts on the use of plea agreements to adjudicate misdemeanor case filings ACA presents to the courts. 84 Practicum for Investigation and Adjudication of Corruption Offences, prepared by Biljana Sinanović, Brankica Marić, Vlatko Božović, Dragana Stojadinović; Goran Nikolić; Jelena T. Jakovljević and Jelena Kostić, PhD￾https://www.odgovornavlast.rs/wp-content/uploads/2019/11/Practicum-for-Investigation-and-Adjudication-of￾Corruption-Offences-ENG.pdf 85 Some of the topics include: initial information and access to open data; financial investigation; analysis of money flow and accounts; and drafting and execution of an investigation plan (indictment and asset recovery). 86 http://www.birodi.rs/wp-content/uploads/2019/08/LAP-cesto-postavljana-pitanja-i-odgovori-2.pdf 87 Details available in GAI Quarterly Reports- for example, Q2-Year 3 (May 1 – July 31, 2020) USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 12 e) The Guide for Persons Authorized for Receiving and Investigating Whistleblowers Reports in Local Self￾Government includes the necessary information and instructions to organize an efficient internal whistle-blowing system in LSGs.88 f) Local Transparency and Accountability Instruments. GAI provided instruments to enhance transparency and accountability at the local level, such as a guide to improve the communication of LSGs on social networks or the Local Government Model Website.89 g) Tools for Implementing the Public Procurement Law. GAI is working to provide tools to implement the new Public Procurement Law and develop model internal acts for public procurement to align with this new law. h) Tools for Financing Youth-Selected Projects. GAI prepared tools to facilitate student engagement in the planning of the local budget, and for budget allocation and transfer of funds to finance the selected projects (cases from Sombor and Šabac).90 2.2.2 EFFECTS ON PARTICIPATING INSTITUTIONS According to the key informants, GAI has influenced the overall administrative culture among the partner institutions through the application of tools and methods to enhance transparency and accountability. ● GAI guided and supported participating LSGs to identify priority anti-corruption actions in their LAPs and establish Local Anti-Corruption Councils as independent public monitoring mechanisms, according to key project participants.91 GAI-supported LSGs have also established internal capacities for personal data protection and mechanisms for whistleblower protection. In terms of delivery of services, LSGs recognized they “enhanced their capacities (e.g., clarified issues, prepared templates and practiced on real-alike situations) to implement the new Law on Public Procurement.”92 Two GAI-supported LSGs (City of Sombor and City of Šabac) have established single administrative points, integrating several administrative procedures into one, thus, working towards more efficient service delivery. ● SAI management and staff recognized critical assistance from GAI to identify strategic priorities and prepare a plan to improve the delivery of core services.93 The key informants recognized that support from GAI contributed to SAI’s organizational development primarily by introducing performance audits.94 ● The Activity is working to strengthen the organizational capacities of ACA. However, political influences and the newly appointed leadership’s lack of willingness to implement ACA’s core mandate limits the evaluation team’s ability to assess the effects of GAI assistance on the 88 Prirucnik za postupanje lica ovlašćenih za prijem informacija od uzbunjivaca u jedinicama lokalne samouprave, prepared by Milan Pavlica, Dragana Matović, Tom Divajn, Duško Kovacević, Vladimir Radomirović, Belgrade, 2019 89 https://www.transparentnost.org.rs/images/dokumenti_uz_vesti/LTI_2020_final_report_ENG.pdf 90 KII notes KII_14, KII_19, FGD_04 91 KII notes- LSGs and FGD_01, FGD_02, FGD_03 92 KII_014, KII_019, FGD_01, FGD_03 93 KII_07 94 KII_07 13 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV organizational development of ACA.95 Specifically, the evaluation team was not able to assess whether trained ACA employees contributed to improved ACA services. ● The media’s increased attention to and reporting on subjects under the transparency and accountability framework is one of GAI’s indirect results. For example, GAI has brought the topics related to LAPs or performance audit-reports to the attention of the media in some participating LSGs.96 2.2.3 MANAGEMENT PRACTICES The evaluation team’s desk review and primary data analysis indicates that GAI has been implemented efficiently and that GAI has exercised adaptive management. Partners stated that GAI has demonstrated flexibility throughout implementation. 97 Management mechanisms, including implementation modalities, organizational structure, and the role of each partner under GAI’s components have facilitated GAI’s ability to achieve planned results.98 Also, the timely implementation of activities and the availability of financial and human resources to implement these activities have contributed to overall efficiency. Implementing partners appreciate the GAI team’s flexibility and limited bureaucracy.99 Partners indicated that GAI has dealt with their requests in a timely manner, and the relationship with GAI management has been based on mutual trust and respect.100 GAI has established a clear work plan and is on target to complete the Activity on time and within the approved budget.101 GAI’s work plans included results-oriented annual plans (and more detailed quarterly plans), providing a sound basis for scheduling, resource allocation, budget control, and IR attainment. The evaluation team found that the work plan development process has been participatory, through the joint work of the GAI team. The partners stated that the GAI work plan also served to coordinate activities, especially highlighting its importance as a tool for coordinating efforts related to capacity development and training.102 The main factor that contributed to GAI’s flexible and agile response during the implementation is the proactive role that GAI management and technical staff played in responding to the changes, challenges, and emerging priorities under the Activity’s three IRs. This, combined with the established partnerships with the main stakeholders, created an efficient implementation process. The partners highlighted that GAI’s responsiveness and flexibility during implementation is attributed to its “proactive and swift decision making based on needs of” participants.103 The financial and human resources for the implementation of GAI have been well-planned and available, according to the work plans. At the operational level, GAI established an Activity 95 Different analytical reports from the Freedom House- Nations in Transit 2020, https://freedomhouse.org/country/serbia/nations-transit/2020 to EU Progress Report for 2020, indicate the challenging situation of ACA, with no autonomy in performance of ACA. 96 KII_10 97 KII_02, and other KIIs with national partners 98 KII notes KII_01. 99 KII notes 100 KII notes 101 KII notes KII_01 102 KII notes with the national stakeholders 103 The training program was modified to online delivery during the COVID-19 pandemic. Following protection measures, GAI prepared online programs for all subjects planned during the last quarters of the project. Delivering online training programs is associated with numerous challenges, from the availability of quality internet connections to ensuring active participation. Still, the participants have been active, expressing positive opinions about the approach, and have received trainings. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 14 team with the Chief of Party (COP) as its head. The COP's prime responsibility has been to plan efficiently and deliver results, including to ensure that deliverables are of a high quality, and delivered within the approved work plan and budget. The COP also serves as the technical expert for the Activity’s component under IR2, related to supporting IOIs. The team included component managers to oversee progress towards intermediate results. Each component has its own staff to provide technical and on-demand support during implementation. The Activity, through the COP, facilitates constant communication and information exchange regarding strategic priorities with the partners and authorities at the national and local level. GAI did not originally plan to establish a steering committee or advisory structure. However, the existence of a high-level steering mechanism could be beneficial to set institutional and sectoral priorities, while also facilitating commitment and ownership over results. GAI established a monitoring system and a team responsible for data-collection and analysis. This monitoring system served, to a certain extent, as the reference during reporting. The Monitoring, Evaluation, and Learning Plan (MELP) set the performance monitoring framework. It includes tailor-made data collection protocols and instruments. The evaluation team found that the adopted indicators and the overall MELP have served to facilitate tracking of performance under all Activity components and overall progress reporting, despite some weaknesses (elaborated under EQ1). The indicators presented in the MELP have been mostly satisfactory; still, the evaluation team found gaps within the existing indicators. The existing indicators were not gender sensitive. Also, they could not capture progress under some important GAI areas. For example, indicators could not adequately measure GAI’s performance in the areas, such as right to information, civic participation, and complaint mechanisms. GAI did not introduce appropriate indicators to measure development of capacities of the target groups or the effects on anti-corruption education programs on the selected youth (e.g., Anti-Corruption Legal Clinic and its participants). Also, GAI did not measure the effects of awareness campaigns and assess if the targeted audience has enhanced understanding of a specific topic. The MELP also did not include more specific indicators related to CPI’s assistance to LSGs. 104 The proposed means of verification and data sources provided were appropriate to validate progress and achievements. GAI has established a regular reporting practice, but results-oriented reporting has been limited. Concerning reporting frequency, GAI prepares regular quarterly reports, providing a substantive presentation of implemented activities, and presenting plans for the next quarter. However, results-oriented reporting and references to the MELP indicators to measure the progress towards the achievement of targets in the quarterly reports are limited.105 2.3 EVALUATION QUESTION 3: SCOPE OF WORK EXPANSION What other areas of interventions could fit under the SOW beyond what is already planned? Building on the evaluation team’s assessment of GAI’s current SOW, through the findings under EQ3 and the desk review and interviews with key informants, the team sought to identify additional areas of interest that would complement GAI’s intervention to further achieve the Activity’s desired results and promote its sustainability and scalability. 2.3.1 ASSESSMENT OF THE CURRENT SOW GAI fully aligns with governance priorities related to transparency, accountability, and the prevention of corruption in Serbia. GAI is well-aligned with the 2013-2018 National Anti￾104 The MELP included indicator IR2.1.5. Number of USG-supported anti-corruption campaigns and did not provide other indicators to measure effects of the assistance to CPI and LSG. Still, the LTI could serve to some extent as proxy indicator to measure results at this level 105 Based on GAI quarterly reports 15 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV Corruption Strategy (NACS), its Action Plan (AP), and its general objective “to eliminate corruption, as an obstacle to the economic, social and democratic development of the Republic of Serbia.” GAI's focus on enhancing government accountability at the local and national levels through the capacity development of key stakeholders and strengthening their connections is instrumental in implementing the NACS objective to “ensure good governance through the inter-institutional partnerships as the vehicle for its proper and effective implementation.” There are clear links between GAI's areas of intervention and the NACS AP, with its focus on the most vulnerable areas, including strengthened participation, oversight mechanisms, and improvement of investigation capacity, among others.106 GAI is relevant for the implementation of Chapter 23 of the AP, concerning justice, fundamental rights, and the fight against corruption. It reflects all mitigation measures recommended through the European Commission (EC) Screening Report.107 The relevant priorities for GAI are: i) clarify the cooperation between the stakeholders in charge of implementing anti-corruption measures; ii) clarify the mandate of ACA (including on a legal basis) to ensure that the staffing level matches the tasks; iii) improve free access to information rules and their practical implementation; iv) strengthen transparency and integrity of governing structures, including through strengthening internal control and audit bodies; v) adopt and implement the law on whistle-blowing and take the necessary steps to make the whistleblower protection system more effective in practice; and, vi) ensure independent, effective, and specialized investigation/prosecution bodies.108 GAI is aligned with the action point under Chapter 23 that envisaged “preparation of LAPs,” to identify corruption-prone areas and processes and define preventative and sanctioning mechanisms.109 Chapter 23 stated that “by implementing LAPs and establishing adequate mechanisms for their enforcement, local governments will enhance their adherence to good governance principles.”110 The partners were positive about the Activity’s responsiveness and involvement from design and planning to implementation, while re-stating that GAI provided the requested technical assistance. The Activity design process was consistent, based on a sound programming and formulation approach.111 The Activity identified problems using a bottom-up process of collaboration, and involved policy makers and representatives for the Activity’s main partners.112 GAI’s design of technical assistance and transfer of knowledge addressed the major challenges facing the independent regulatory bodies, including their limited technical capacities to deliver some of their core services, and 106 National Strategy for Fight Against Corruption for the period 2013-2018, more at: http://www.acas.rs/wp￾content/uploads/2010/06/Nacionalna_strategija_za_borbu_protiv_korupcije.pdf 107 https://www.mpravde.gov.rs/tekst/30402/revidirani-akcioni-plan-za-poglavlje-23-i-strategija-razvoja-pravosudja￾za-period-2020-2025-22072020.php 108 Ref to Chapter 23 - https://www.mpravde.gov.rs/tekst/30402/revidirani-akcioni-plan-za-poglavlje-23-i-strategija￾razvoja-pravosudja-za-period-2020-2025-22072020.php 109 Ibid. 110 Ibid. 111 KII notes KII_01, KII_02 112 KII_01 KII notes, national partners USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 16 exercise monitoring and oversight proactively and effectively.113,114,115 These challenges have continuously undermined mechanisms to hold the government accountable for the delivery of services.116 The core areas of GAI assistance respond to the needs for adoption, implementation, and monitoring of LAPs; improved transparency in local governance processes; enhanced civic participation; and provided the prerequisites for whistleblower protection and improvement of service delivery. The GAI objective, as well as its activities, are relevant to the needs of local authorities: the interviewed representatives stated that “GAI helps them to respond to citizens’ needs more effectively in line with law and assigned services.”117 The Activity adopted a tailor￾made approach, responding to the needs of each LSG and taking into consideration their unique characteristics.118 All municipality representatives who participated in the online survey administered as part of the evaluation recognized that GAI assistance is well-aligned with local governance reform priorities in Serbia: 50 percent of survey respondents reported a complete alignment and 50 percent reported a sufficient alignment. The survey participants also indicated that the Activity addressed the needs of IOIs: the majority (95 percent) indicated that they positively regarded GAI’s efforts to address the needs of key Serbian government institutions to ensure greater accountability. The focus of GAI is to enhance the capacities of government institutions to deliver services that ensure oversight of the operations of other government bodies and hold them accountable for their work. However, despite improvements and support from GAI, these institutions lack many of the tools and technical capacities to fully and competently fulfil their mandates.119 The IOIs report annually to the NA on challenges, successes, and opportunities to improve governmental operations; however, a lack of follow up by the NA and the Government has prevented the enforcement of the oversight bodies’ findings. This situation is undermining the public’s perception of the oversight institutions’ work and is discrediting citizens' faith in solving corruption in the long run. The absence of follow-up on findings and recommendations contributed to a general lack of understanding of oversight institutions’ mandates. The representatives of the judiciary recognized that “GAI played a pivotal role in responding to the immediate need to enhance capacities of specialized anti-corruption prosecutors and court units, established with the adoption of the new Law on Organization and Jurisdictions of State Bodies in Charge of the Prevention of Organized Crime, Terrorism, and Corruption.”120 The Activity also 113 For example, implementation of the new Law on Private Data Protection, implementation of performance audits for the State Audit Institution and follow up on recommendations, mandate of ACA. 114 For example, ACA needed to improve risk-based oversight of asset and income declarations and build the institution’s analytical and research skills to enhance its performance. 115 The EC Progress report for 2018 highlighted that the Parliament’s practice not to discuss any annual reports by independent bodies in plenary sessions has continued from 2015. Very limited discussion and follow up to the SAI’s audit reports and a lack of parliamentary discussion on the implementation of major laws and policies have been also recorded. 116 More details in the EC Progress Report 117 KII_12, KII_21, KII_20 and focus group discussions 118 KII_01 119 The recent EC Progress report highlighted these challenges, also emphasizing other challenges: to ensure more effective cooperation and channels of communication (with government institutions) and enhance credibility and understanding of mandates and roles of these institutions. 120 KII_06 17 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV addressed the need to establish an integrative electronic system to register corruption cases and increase follow-up proceedings.121 GAI has remained well-aligned with governance related strategic priorities during its implementation thus far. The evaluation team assessed GAI’s relevance at all points during its lifecycle to test and analyze whether the intervention logic remained true in the context of possible changes on the demand side, and, if not, to assess the response made to adjust and fine-tune the intervention as set out in GAI’s intervention logic and its logical framework. The evaluation team has focused on whether the IRs and Activity objective, as the main references under the intervention logic, remained valid and achievable or whether there was a case for revision to take account of changes in the external environment, demand, or speed of delivery. The “real time” assessment of GAI’s relevance confirmed that the intervention logic remained true throughout its implementation, as justified in some of the key governance-related strategic and policy documents. The 2020 Serbia Progress Report issued by the EC recognized limited progress with the implementation of anti-corruption measures.122 At the same time, the report highlighted that “corruption remains a pressing problem for the country,” together with insufficient transparency and accountability of public institutions.123 GAI under its components is addressing the identified needs and priorities of Activity stakeholders. The overall priority has been to work on “establishing an effective coordination mechanism to operationalize prevention policy goals.”124 At the institutional level, ACA is the key institution in the fight against corruption, and requires development of its capacities to “effectively implement the law on the prevention of corruption in line with the European Union (EU) acquis, international agreements and Council of Europe’s Group of States against Corruption (GRECO) recommendations.”125 In addition, IOIs require “systematic support for their work, to regularly monitor the implementation of their recommendations and hold the executive bodies accountable.” Their priority is to “improve data management, including registration of corruption cases, considering different units and stages.”126 At the local level, the need remains to ensure tangible improvements in relation to anti-corruption efforts and enforcement of implementation of the LAPs.127 The revised Action Plan for Chapter 23 also restated these priorities, proposing more actionable measures and indicators of success.128 The stakeholders and online survey participants that the evaluation team interviewed recognized USAID’s “substantive insight and adequate approach to development needs and challenges in the governance area” that contributed to the lasting relevance of GAI.129 Furthermore, achieved results and 121 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 122 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 123 European Commission- Serbia 2020 Report, more at https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 124 Ibid., European Commission- Serbia 2020 Report 125 Ibid., European Commission- Serbia 2020 Report, and also https://www.coe.int/en/web/greco/evaluations/serbia 126 https://www.coe.int/en/web/greco/evaluations/serbia 127 The report recognized that “out of the 102 local self-governments that adopted anti-corruption plans, 28 established a body to monitor their implementation, mostly in line with the Anti-corruption Agency’s model.” However, the actual benefits achieved through implementation of LAPs could not be assessed. 128 https://www.mpravde.gov.rs/en/tekst/30881/booklet-on-project-achievements-eu-for-justice-support-for￾chapter-23.php 129 KII notes KII_15 and KII_07 USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 18 progress in the specific areas of the intervention are inherently linked to the “enhanced relevance of GAI and its intervention.”130 GAI has accurately identified components and clearly established areas of intervention under these components. GAI’s already tested services need to continue, with a focus on root causes of challenges. The scope of GAI is well defined, with adequately identified priorities and areas of intervention. The prevailing opinion among the stakeholders is that GAI should “continue with the assistance/activities;” stakeholders state that “models and packages have already been developed and tested, effectively producing visible and concrete results.”131 The interviewed partners, namely those holding senior level positions, and survey participants, recognized that “GAI has confirmed its validity to the needs of respective institutions and LSGs in Serbia,” also proposing to “further fine-tune and focus support to address root causes of challenges.”132 The participants in the online survey confirmed the relevance of GAI and the need to continue with its activities, prioritizing capacity development, inter-institutional cooperation for anti-corruption, and the improvement of delivery of services at the local level.133 Survey participants suggested the following areas of proposed support, among others: capacity development or training, mentoring assistance aimed at improving service delivery, and developing specific anti-corruption guidebooks and other tools. The broad area of capacity development included proposals for “organization of priority training programs and workshops to improve good governance related capacities for holding the government accountable at the local and national level” and “further strengthen the capacity of specialized units, prosecutors, and judges in the areas of anti-corruption.” Survey participants recognized mentoring support as an important capacity development tool that should be used for the preparation of LAPs, establishment of public participation practices, and more transparent budgeting processes at the local level.134 In addition to strengthening cooperation among IOIs, survey participants highlighted the need to increase cooperation with the civil sector and media on specific anti-corruption topics. Improvement of the delivery of services, especially at the local level, should continue, with advancement of e-government opportunities, simplification of procedures, and more citizens-oriented processes. 2.3.2 ADDITIONAL AREAS OF INTERVENTION THAT COULD FIT UNDER THE SOW The evaluation team used the data gathered from KIIs, FGDs, the online survey, GAI documentation, other sources, and analytical documents to identify possible areas of interventions that could fit under GAI’s SOW, beyond already planned activities.135,136 The evaluation team presents these additional areas by levels of intervention: systemic, institutional, and individual. SYSTEMIC LEVEL ● ENGAGE IN POLICY MAKING. Stakeholders from the oversight institutions recognized that GAI could engage more actively in the policy making process related to anti-corruption, transparency, and accountability. According to these key informants, the expertise and experience gained from 130 KII_07 131 KII notes KII_16, KII_15, KII_07 132 KII notes- partners 133 Results from the online survey, prepared and analyzed by the evaluation team. 134 Results from the online survey 135 The revised Action Plan for Chapter 23, the annual reports of the National Assembly of Serbia, and IOIs. 136 The main references are the Serbia 2020 Progress Report of the European Commission, The Freedom House Report on Serbia, 2020, and the Council of Europe analytical reports on Serbia. 19 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV GAI could be beneficial to prepare and adopt a “new anti-corruption strategy together with a credible and realistic action plan.”137 ● SUPPORT OVERSIGHT INSTITUTIONS DURING AND AFTER COVID-19 PANDEMIC.138 Stakeholders recognized that GAI could provide important support to IOIs to respond to the corruption risks and challenges posed by COVID-19.139 Oversight is an issue of relevance, including access to information of public interest, protection of personal data, and also the protection of whistleblowers (e.g., if issues arise related to the procurement of medical equipment and supplies).140 INSTITUTIONAL LEVEL ● EXPAND SUPPORT TO LOCAL AUTHORITIES. GAI’s support to local authorities should continue and expand. For example, GAI provides specific assistance to LSGs in designing sound and realistic monitoring systems for their LAPs. In parallel, there is a need to enhance capacities of local independent anti-corruption bodies in charge of LAP monitoring. GAI could further support these bodies to assess progress, analyze results, highlight challenges, propose areas for additional attention, or promote best practices.141 The local level partners and FGD participants recognized GAI as an important support for preparation of local communication strategies and to improve communication. At the same time, they observed insufficient capacities for strategic communication especially at the level of its leadership and senior level officials. Considering GAI’s achieved results, GAI is well-positioned, according to them, to address these needs and facilitate development of communication skills at the local level.142 In some supported LSGs, GAI assisted with the analysis of finances and the budget implementation process, preparing plans with recommendations for improvement. However, there is no evidence that LSGs are progressing with implementation of these plans. GAI could build on existing support to budgeting activities and provide next-level assistance, focusing on implementation of recommendations and monitoring the entire improvement process. ● ENGAGE WITH PUBLIC AND PRIVATE COMPANIES. The need for oversight agency engagement with the public and private companies is highlighted as an issue in the most-recent EU country progress report.143 The mentioned areas of potential support include strengthening capacities to manage public interest data and building mechanisms and capacities to protect whistleblowers, among others.144 The partners recognized that GAI could support inclusion of private sector interests into good governance and anti-corruption priorities for the country. According to partners, GAI could also engage with public companies, including public utility 137 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 138 The COVID-19 outbreak increases corruption risks. The health sector is particularly exposed, because of the simplified procurement rules as a response to the immediate need for medical supplies, non-transparent spending of public funds, overcrowded medical facilities, and overburdened medical staff. As such, potential corruption and procurement issues, including bribery in medical-related services, conflicts of interest, and lobbying are possible. 139 KII_12, KII_07 and KII_15 140 Corruption Risks and Useful Legal References in the context of COVID-19, issued by Mr. Marin Mrčela, GRECO President, Strasbourg, 15 April 2020, GRECO (2020) 141 Based on the analysis from the EC Progress Report for Serbia, 2020- https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 142 FGD_01, FGD_02, FGD_03, KII_25 143 EC Progress Report for Serbia, 2020. https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 144 KII_15 and EC analysis USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 20 enterprises and other direct public budget users. Initial activities could include establishing and strengthening whistleblower protection mechanisms for the private sector, which would enable employees to report corruption. ● STRENGTHEN NATIONAL ASSEMBLY. The main analytical documents and information from the independent institutions showed the need to strengthen the capacities of the NA and its relevant committees to read, understand, and comment on the oversight institution reports, and present findings and recommendations during plenary sessions.145 The newly constituted Serbian parliament is dominated by an overwhelming majority of the ruling coalition and the absence of a viable opposition, a situation which is not conducive to political pluralism in the country. Nevertheless, members of the NA and especially members of the main committees would benefit from capacity development in the core areas of their work to exercise their functions more competently.146 ● STRENGTHEN PROSECUTORS. The recent EC Progress Report for Serbia and other references on good governance in Serbia show the need to strengthen the technical capacity of prosecutors.147 GAI could build on the positive results so far, including the Manual for Whistleblower Protection, the exposure to best practices, and experience exchange, by facilitating the visit of a career prosecutor from the U.S. GAI could consider intensifying these efforts, by presenting, for example, the American experience on following up with and acting on whistleblower reports. Particularly, GAI could provide technical assistance to prosecutors regarding implementation of the relevant anti￾corruption laws. ● STRENGTHEN DATA PROTECTION OFFICERS. GAI could explore opportunities to strengthen the network of data protection officers (DPOs) according to CPI.148 A total of 3,000 DPOs from public institutions and private enterprises have been engaged in Serbia.149 There is a need to provide sound capacity development support to the DPOs. GAI could work with CPI to prepare an interactive online platform with basic and advanced training for DPOs. INDIVIDUAL LEVEL ● OFFER LEARNING OPPORTUNITIES TO YOUTH. The partners observed that youth activism, through participation in local planning and decision-making processes, has been one of GAI’s important areas of support, and according to them, the experience from the City of Sombor could be beneficial to other participating LSGs.150 The partners also shared that the implementation of the Anti-Corruption Legal Clinic is a format that could be built upon.151 ● OFFER LEARNING OPPORTUNITIES TO JOURNALISTS. GAI could explore opportunities to develop the capacity of journalists to better understand the anti-corruption framework and audit reports, including performance audit reports and clarify issues of personal data protection, especially in media. There is a need for local media to oversee the implementation of LAPs as well. 145 EC Progress Report for Serbia, 2020. https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 146 The main committees relevant to GAI oversight are the Committee on Constitutional and Legislative Issues, the Committee on Finance, State Budget and Control of Public Spending, and the Committee on the Judiciary, Public Administration and Local Self-Government. 147 https://www.coe.int/en/web/greco/evaluations/serbia 148 KII_15 149 All large companies must appoint a representative for Serbia for personal data protection. Some of the large companies such as Google, Viber, now Facebook have started this process. 150 KII_14, KII_08, FGD_04 151 KII_011; FGD_04 21 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV ● RAISE AWARENESS. Citizens’ participation remains critical for transparent and accountable local governance. GAI’s primary focus has been on the capacities of local authorities to establish systems and mechanisms for citizens’ participation. However, GAI could encourage citizens to take a broader interest, and increase their willingness to participate in local decision-making processes. It will be especially important to engage the public in anti-corruption efforts and activities at the local level. Youth activation remains highly important. GAI could work to build awareness and interest by increasing youth’s understanding of their rights and encourage their participation in local processes. 2.3.3 FLEXIBILITY TO CHANGING NEEDS AND PRIORITIES GAI has been flexible in responding to challenges that have occurred during its implementation to date, particularly to the emerging priorities of the participating institutions. The GAI implementing environment is challenging and is characterized by limited progress in the fight against corruption. The analytical reports on the governance situation in Serbia highlight some of the systemic difficulties, including the weakening of institutions, and the executive branch’s interference with institutional mandates, which compromise the institutions’ fragile independence.152 There is an emphasis on the need to strengthen the capacities of ACA to implement its mandate. However, ACA’s implementation of the new anti-corruption law is lagging behind due to limited commitment of the new management and declining capacities of ACA.153 ACA must fully acknowledge the role of independent bodies and follow up on their recommendations promptly.154 The changes brought by the law on prevention of corruption (in force since March 2018), produced some results in increasing the number of closed cases. However, the 2020 EU progress report concludes that "corruption remains an issue of concern,” and there is still “no effective prevention coordination mechanism in place.”155 The GAI work plans are critical to ensuring timely and successful implementation of GAI activities.156 In addition, the Activity’s mitigation strategy focused on delivering technical level activities, ensuring that strategic level decisions are aligned with national priorities and identified needs. 157 Nevertheless, the Activity is responsive and flexible to the changes in the external environment within the agreed plans and implementation framework. For example, GAI demonstrated this flexibility in its response to the limited commitment of the leadership of ACA. GAI adjusted its capacity development approach, helping with strategic planning and development of analytical capacities.158 Also, GAI’s mechanism to assess the effectiveness of the training programs enabled GAI to “institutionalize” flexibility. Practically, GAI responded to the comments and reactions from trainees and adjusted its delivery approach to address needs.159 In addition, GAI responded to the COVID-19 pandemic effectively, adjusting its training programs to an e-learning format. Within a short period of time, GAI trainers and instructional designers reconfigured in-person classroom training programs for online learning.160 Implementing partners appreciated GAI’s level of flexibility and limited bureaucracy.161 Partners have indicated that they generally have good relations with GAI, the COP, and component heads. GAI responded to 152 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf and https://freedomhouse.org/country/serbia/nations-transit/2020 153 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 154 European Commission- Serbia 2020 Report, https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 155 Ibidem, European Commission- Serbia 2020 Report 156 Notes from KII_01 157 Notes from KII_ 01, project progress reports also reflected on these changes and reconfirmed the approach. 158 GAI Progress Reports and KII_01 159 KII_07, KII_15 and KII with representatives of LSGs. The results of the survey also indicated appropriateness. 160 KII notes- KII_01, KII_07, KII_15 161 KIIs notes USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 22 requests from partners in a timely manner and cultivated relationships based on mutual trust and respect.162 2.4 EVALUATION QUESTION 4: FUTURE PROGRAMMING What are the interventions with greatest traction, unintended outcomes, and/or lessons learned from USAID assistance that the Mission can use in decision making for future programming? In response to EQ4, the evaluation team analyzed primary and secondary data sources to identify GAI’s interventions that demonstrated the greatest traction to date and those that resulted in unintended outcomes, as well as to document lessons learned. 2.4.1 INTERVENTIONS WITH GREATEST TRACTION GAI has consistently responded to Serbia’s national capacity development needs and is steadily strengthening national and local level capacity to improve government accountability. Future progress and its sustainability, however, rests on the continued commitment and capacity of Serbia’s governance system, institutions, and citizens. GAI has provided capacity development assistance to employees and stakeholders involved in governing at the local and national levels.163 Of the surveyed employees and stakeholders who participated in GAI’s intervention, all stated they benefited from this assistance.164 The partners recognized that GAI has provided "timely, effective and highly demanded” technical assistance to the specialized anti-corruption court units, in line with the law on prevention of corruption.165 In addition to improvements in case management, monitoring, and adjudication of corruption cases, through the functioning of the GAI-supported ERCC system, these results and capacities should facilitate future evidence-based policy and decision making in the sectors of concern.166 KIIs with IOI stakeholders suggest that assistance available through GAI has been instrumental in providing demanded technical assistance to these institutions to establish capacities and systems to better deliver their core services.167 Still, the sustainability of progress relies on the continued genuine commitment of management teams to implement assigned functions. Authorities in Serbia must recognize the role of independent and regulatory bodies, strengthen their capacities, including their budgets and skilled staff, and guarantee their functional independence in practice.168 There are still challenges, such as dominance and influence of the executive branch over the work of IOIs, slow implementation of anti-corruption laws, politicized leadership of ACA, among others. These and other factors prevent Serbia from promoting and ensuring credible, professional institutions that follow the principles of transparency and accountability, which may present possible implications for sustainability.169 162 KIIs notes 163 A total of 81 percent of the participants in the survey participated in the trainings that GAI organized. 164 According to the results of the online survey, 93 percent of the participants stated they benefited from the capacity development support from GAI 165 The Law on the Organization and Competences of State Organs in Fight Against Organized Crime, Terrorism and Corruption, adopted by the National Assembly in 2016, and then came into force in March 2018. 166 KII_29 and KII_06 167 KII_15, KII_07, KII_06. These services include performance audits and follow up on recommendations (SAI), systematic protection of personal data (CPI) or prosecution of corruption cases 168 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf 169 Ref to progress under the Chapter 23, in the report https://ec.europa.eu/neighbourhood￾enlargement/sites/near/files/serbia_report_2020.pdf 23 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV GAI has been working to enhance policy processes by designing instruments and tools and implementing technical assistance to enhance transparency and accountability at the national and local levels. Further details are provided under Section 2.2.1. of this report. These efforts have been important and indicate that progress is likely to be achieved in some areas, including more transparent and accountable local governance, improved state audit practices, and strengthened frameworks for implementation of personal data protection. However, there are still challenges to ensuring the successful implementation of the policies, largely the coordination of horizontal and vertical policy. There are also remaining impediments to translate policies into actions, connecting adequately with public funds and ultimately monitoring and reporting on progress.170 The lack of country and sector specific targets and indicators under the anti-corruption framework and absence of regular monitoring and reporting mechanisms have potentially limited GAI’s progress to date and may affect the sustainability of GAI’s interventions. GAI has remained a valuable partner in supporting the achievement of transparent and accountable governance in Serbia, establishing strong and effective partnerships with the national institutions. Stakeholders from partner organizations recognized “strong comparative advantages” of GAI, stating that GAI (and USAID) remain “an important, credible and widely accepted partner in supporting transparency and accountability of the governing institutions” and contributing to “achieving priorities for the establishment of functional system to prevent and punish corruption in Serbia.”171 The backbone of GAI’s comparative advantage is its technical capacities and profound understanding of the organization-specific constraints and development needs, as well as its proven focus on the needs and priorities of national institutions.172 In addition, “USAID through GAI and previous interventions in the area of building transparent and accountable institutions in Serbia demonstrated strong abilities to establish and maintain effective partnerships based on trust, responsiveness, and mutual respect.”173 Stakeholders from participating organizations recognized the value of GAI’s technical support and enhanced efforts to ensure compliance with anti-corruption laws in Serbia, EU accession requirements (derived from Chapter 23), and other international standards.174 GAI’s demand-driven approach and strong national participation in all GAI activities has additionally contributed to a strong sense of stakeholders’ ownership of the achieved results.175 National partners have been involved in conceptualizing and implementing activities under GAI, through forums for example.176,177 GAI has employed a consultative and inclusive approach to formulating work plans and Activity design, ensuring stakeholders' involvement, contributing to an increased sense of local ownership and a foundation to promote the sustainability of results. The favorable environment for the Activity’s implementation can be attributed to long-lasting and mostly positive experience with USAID’s support to the establishment of good governance in Serbia. Likewise, GAI followed a participative partnership-building approach throughout the implementation of activities. National partners have been involved in the activities, either as participants, implementers, or members of different forums, to create awareness of the achievements and results in their respective areas of work. At the same time, the evaluation team found, through KIIs with key stakeholders, that 170 OECD-SIGMA “Monitoring Report: The Principles of Public Administration-Serbia,” May 2019, more http://www.sigmaweb.org/publications/Monitoring-Report-2019-Serbia.pdf 171 KII notes- national partners 172 KII notes- national partners 173 Notes from KII_02, KII_03, KII_07 and KII_08 174 Notes from KII_07, KII_14, and KII_08. 175 Notes from KII_02 and the results presented under the previous part of this report. 176KII_15. CPI and GAI have jointly developed and implemented a program on personal data protection for local governments 177 SAI Management has been participating in different activities for its organizational strengthening; at the same time, SAI was involved in delivery of mentoring support to LSGs and budget users. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 24 some stakeholders have been familiar with GAI components only directly related to them, with limited awareness of the other areas of the Activity’s intervention.178 This situation has been somewhat different among the senior-level representatives of partner organizations, who confirmed a more in-depth understanding of the Activity and its scope and results.179 2.4.2 UNINTENDED RESULTS GAI has produced important, yet unintended, results in different areas of its intervention, including increased communication, coordination, and cooperation among LSGs and national governing institutions. The evaluation team lists these unintended results below: ● Delivery of training programs addressed knowledge gaps and enhanced capacities and skills among the staff from participating LSGs. In addition, the trainings favored horizontal knowledge sharing amongst the employees from different LSGs, helping employees to connect and unify their understanding and application of procedures. ● Similarly, GAI has helped SAI to establish effective, although informal, communication with participating LSGs. During GAI events, the SAI management team participated as guest speakers and agreed on regular communication channels with LSGs.180 According to SAI representatives, this channel of communication has enabled LSGs to resolve some of the critical audit-related findings and recommendations and improve the internal management processes.181 ● GAI provided comments and interpretations on the relevant policies and laws, for example, the Public Procurement Law, the Law on Personal Data Protection, and the Law on Whistleblower Protection. GAI prepared guidelines or manuals that were broadly disseminated and shared with other LSGs. ● Stakeholders, with support from GAI, have been exposed to international best practices and models tested in the United States. Besides contributing to the improvement of professional capacities, these events created additional positive attitudes and enhanced Serbian public institutions’ commitment and readiness to implement practices for greater transparency and accountability.182 ● The Activity has been active in advising and advocating for transparency and accountability at different levels. The active role of GAI staff in implementing activities and building partnerships generated additional readiness and willingness among participating institutions to propose reviews of legal and policy documents to better reflect principles of transparency and accountability. Partners recognized that GAI provided important support to introduce participatory budgeting in the city of Sombor. These efforts included increasing youth participation to select priority initiatives for their schools and communities. According to the partners, this support resulted in improved youth civil activism in the City of Sombor, creating a favorable environment and developing the capacities of youth to explain their needs, to make questions and demands, and to become more socially active.183 178 Refer to KII notes with the national partners 179 The senior level representatives of the partner organizations have participated in GAI programming phase and have been involved in discussions during the conceptualization of GAI. They have also been involved in activities under other components. For example, SAI management took part in the workshops with LSGs, took part in GAI workshops and joined other events that enabled them to grasp the full scope of the Activity. 180 KII_07 181 KII_07 182 FGD_01, FGD_02, FGD_03, KII_08, KII_14 183 FGD_04 25 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV 2.4.3 BEST PRACTICES While the team’s priority in conducting the evaluation was to analyze GAI’s expected impact and sustainability prospects, the team identified best practices from GAI’s implementation based on several criteria, such as the effects on participating institutions, the durability of established mechanisms, and the partnerships. This section presents the following best practices: using the LTI, engaging youth in budgeting processes, supporting SAI, and strengthening cooperation between CPI and LSGs. USING THE LTI. According to the representatives of participating LSGs and based on primary data sources, the evaluation team found that the LTI, together with the scoring system and GAI’s technical assistance, have contributed to long-lasting improvements and greater transparency.184,185,186 Based on LTI, the average score for 2020 reached a total of 46 out of 100, improving from 40 in 2019. The main advances are in the areas of publishing budget information and developing and publishing local anti￾corruption action plans.187 The representatives of the participating LSGs stated that GAI provided needs￾based support to them and directly contributed to their overall improvement of the LTI score.188 GAI prepared a detailed analysis of the LTI score for each of the participating LSGs and continued with recommendations for improvement with specific measures that LSGs might implement. BEST PRACTICE: USING THE LTI Participation: Representatives of all 13 GAI-selected LSGs (heads of administration, communication managers, and IT experts) participated in GAI workshops and follow-up activities. Effects (and Impact): All GAI partners among the LSGs have increased transparency as measured by the LTI.189 Furthermore, GAI partners, the City of Novi Pazar and the City of Sombor, recorded the most significant improvements, reaching second and third position among all LSG units.190 Longer-term effects could be recognized at the level of developed systems, as they generated awareness and commitment among the LSGs to work on and ensure more transparent functioning. Replicability: The LTI scoring involves all municipalities and cities in Serbia; GAI’s step-by-step approach for the improvement of transparency is available to all LSGs. Sustainability: The achieved results likely will be sustained by GAI-selected LSGs, as the leadership and professionals have been strongly engaged to achieve and sustain transparency.191 ENGAGING YOUTH IN BUDGETING PROCESSES.192 Youth engagement is an added value of GAI’s support to budgeting in Sombor. GAI designed activities enabled young people to provide input to budget planning by suggesting projects to include in the city budget. The Activity, together with the administration of the City of Sombor and the high school student parliaments, enabled voting on 184 The LTI is a sum of positive points obtained from the analysis of web sites, documents, responses to requests for free access to information and direct insight into the functioning of the service centers and premises of the local administrations. https://www.transparentnost.org.rs/index.php/en/research-on-corruption/lti 185 The officials from participating LSGs expressed their commitment to continue working for the improved position on the LTI list, stating numerous activities beyond the scope of the LTI-assessment: publishing of all procurement data and contracts, meeting minutes from the local assembly meetings; decisions from the municipal council’s meetings, etc. 186 FGD_01, FGD_02, FGD_03, KII_08, KII_14, KII_21, KII_22 187 https://www.transparentnost.org.rs/LTI2020/ and https://www.transparentnost.org.rs/LTI2019/ 188 FGD_01, FGD_02, FGD_03, KII_08, KII_14 189 https://www.transparentnost.org.rs/index.php/sr/istraivanja-o-korupciji/lti 190 https://www.transparentnost.org.rs/images/dokumenti_uz_vesti/LTI_2020_final_report_ENG.pdf 191 KII notes LSGs and FGD_01, FGD_02, FGD_03 192 With GAI's support, the City of Sombor successfully implemented an innovative public engagement action plan for budget public hearings, organizing discussions in all local communities on topics specifically related to these communities. These events enabled citizens to discuss the proposed budget and provide suggestions directly to city representatives. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 26 potential projects to take place. GAI created a campaign and developed voting materials for schools to raise youth awareness about the importance of participation in local decision making. BEST PRACTICE: ENGAGING YOUTH IN BUDGETING PROCESSES Participation: More than 2,700 students from 5 high schools participated, representing more than 80 percent of Sombor's high school student population.193 Effects (and Impact): The approved and implemented projects have improved educational infrastructure, such as the renovation of school toilets, purchase of air conditioners, and acquisition of equipment for classroom training and student leisure activities. These projects have generated additional trust and commitment of the youth to participate in municipal affairs and solve immediate problems.194 Replicability: The model from Sombor has high potential for replication. GAI has begun similar activities in the City of Šabac, facilitating sustainable cooperation between the city administration and high schools.195 Sustainability: The City of Sombor has adopted student participation in budgeting as a regular practice and will continue to support students in formulating project proposals to be funded from the local budget.196 SUPPORTING SAI. Support to SAI to introduce and implement performance audits included a comprehensive capacity development support, training for specialized performance audit teams, and mentoring assistance. The objective is to assess if public system organizations and their operations, programs and activities are operating in accordance with the principles of economy, efficiency, and effectiveness. GAI supported SAI’s 13 performance audit teams to finalize initial performance audit designs for the 2020 performance audits. GAI also facilitated the teams’ preparatory work for the presentation of their audit objectives, research questions, selection of relevant methodology, and selected auditees during the preliminary design summits. GAI supported SAI in codifying the knowledge by releasing the Performance Audit Policies and Procedures Manual.197 BEST PRACTICE: SUPPORTING SAI Participation: GAI has provided comprehensive support to SAI related to performance audits, especially for the special performance audit teams.198 Effects (and Impact): The approved and implemented activities have demonstrated an impact on improved capacities of SAI to independently implement performance audits, bringing financial benefits to the country.199 Replicability: The capacity development program could be replicated and shared with SAI public auditors. Sustainability: SAI has adopted performance audits as its regular practice and will continue.200 STRENGTHENING COOPERATION BETWEEN CPI AND LSGs. GAI, together with CPI, trained LSGs to implement the Law on Personal Data Protection, with the focus to encourage local authorities to be more open in response to public, CSO, and media requests for public information. These capacity development efforts have been especially relevant, considering that local administrations, including their direct and indirect budget users, handle large amounts of personal data, while simultaneously facing 193 KII_08 194 FGD_04 195 KII_08 and KII_14 and FGD_03 196 KII_08 197 The manual incorporated performance audit design and message agreement practices, as well as methodology guides on specific aspects of performance audits. This included topics such as design, creation and use of sampling, audit criteria, questionnaire design, case studies, content analysis, and data reliability during audits. https://www.odgovornavlast.rs/knowledge-sharing/ 198 KII_07 199 According to unofficial data from SAI, around 800 mil dinars have been saved. KII_07 200 KII_07 27 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV implementation challenges.201 For example, during the COVID-19 pandemic, all private data was protected, although there was media pressure to report on the identity of infected persons.202 2.4.4 LESSONS LEARNED The evaluation team identified the following lessons learned at GAI’s mid-term: ● Providing focused expertise and policy advice is an added value of GAI. Providing focused expertise and policy advice on specific priorities, GAI added value and contributed to building institutions to ensure more transparent and accountable governance in Serbia. The real sustainability of these results at the level of institutions and the overall legal and public management systems depends on the government’s commitment to accountable and transparent governance. However, interference of the executive and legislative branches with the mandates of IOIs could undermine the credibility of their performance, especially of their reported findings and recommendations. ● Long-term support is crucial to ensure positive institutional development. The long-term USAID support to good governance in Serbia that has continued through GAI, together with technical assistance, has been instrumental for the institutional development of IOIs and local authorities in Serbia. It is critical USAID to work closely and collaboratively with its counterparts in Serbian institutions to achieve national ownership and, ultimately, sustainable change. Focusing on strengthening the technical capacity of analytical staff in government institutions is a strong approach to improve the delivery of its functions. However, the commitment of senior management in IOIs remains critical to ensure that developed capacities are embedded in organizational structure and contribute to the broader good governance agenda. The example of ACA shows that newly appointed senior management remained reluctant to utilize established capacities or participate in organizational development efforts, despite a declared commitment.203 The capacity building efforts of international development partners may legitimize senior management without achieving institutional impact or improved organizational performance. ● Incorporating technology-based efforts allows for greater participation. GAI tested innovative approaches to deliver training programs, benefiting from web-based opportunities. This experience demonstrates that the combination of traditional development interventions with the introduction and use of tailored new technologies can enhance the effectiveness of development efforts, increase results, and contribute to greater involvement of targeted groups. 201 The Law became effective in August 2019, but its successful implementation requires training, especially for administration and public bodies who are unaware of the new requirements and, according to the new CPI Commissioner, LSGs are not prepared to implement the Law. 202 KII_15 203 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf and https://freedomhouse.org/country/serbia/nations-transit/2020 BEST PRACTICE: STRENGTHENING COOPERATION BETWEEN CPI AND LSGs Participation: GAI, in partnership with CPI, provided training to selected LSGs. Effects (and Impact): Local authorities have an enhanced understanding of the Law on Personal Data Protection and its importance. GAI LSGs followed the law during the COVID-19 pandemic, although some LSGs have been disclosing the number of people that have been infected. Replicability: CPI continued with the delivery of training programs to other LSGs in Serbia, recognizing financial constraints. Sustainability: CPI adopted a training program as part of its regular services, however, sustainability will depend on participating LSGs’ readiness and commitment to implement the Law on Personal Data Protection. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 28 ● It is necessary to focus on LSG ownership. Selecting LSGs through a nation-wide open call generated commitment among these LSGs to participate in the Activity and to work and implement activities for enhanced transparency and accountability. These initial results together with the GAI partnership-building approach and demand-driven assistance, provided a platform for LSGs on which to build a strategy to progressively devolve responsibility to local authorities to lead and manage anti-corruption activities. ● Two-way communication is essential to establish trust between local authorities and Serbian citizens. Communication by the local authorities is crucial to increase transparency, and successful two-way communication remains critical for citizens and their meaningful involvement. If LSGs consider communication as a strategic tool, it will increase transparency and trust among its citizens. It is critical that LSGs continue to focus on unifying internal communication among employees and engaging in external communication with citizens to ensure sustainable transparency. ● Monitoring systems should incorporate results-based management principles. Although narrative reporting provides an overview of processes, it is time consuming and often misses links with broader changes at organizational or systemic levels. A comprehensive monitoring system must exist to guarantee rigorous control of activities and assessment of progress, while also contributing to effective reporting of results. Results-based management principles enabled greater transparency and partner awareness of achieved progress. The use of well-known and respected think-tanks and CSOs have been important factors to support LSGs. For example, Transparency Serbia and Pistaljka’s capacities and experience have been instrumental in ensuring progress and enhancing understanding of local governance structures towards greater transparency and accountability. 2.5 EVALUATION QUESTION 5: GENDER EQUALITY Did the Activity affect gender equality within the area(s) of focus? Most stakeholders perceived USAID as steadily promoting gender equality in Serbia, particularly through assistance to LSGs.204 However, the evaluation team found that GAI has given less attention to gender equality and gender mainstreaming, although corruption “presents a barrier for women to gain full access to their civic, social and economic rights."205 The evaluation team found that the GAI MELP does not adequately integrate gender considerations. The MELP is void of gender sensitive indicators and minimally draws on the use of sex-disaggregated data. Furthermore, the GAI team did not design data collection methods to systemically disaggregate information by gender and monitor the meaningful engagement of women across activities. 204 An example of a reference is the USAID-funded Community Development through Democratic Action (CRDA) Program implemented from 2001 – 2006. CRDA programs prescribed to include at least 30 percent of women in local commune councils, bringing change in the gender patterns in participation. Programs also enabled higher activism of women in areas with a long tradition of women’s grassroots initiatives (i.e. Vojvodina).- ref to “The Role of Community Development and Citizen Engagement Activities in Strengthening Civic Engagement and Government Responsiveness in Serbia,” https://secons.net/publications.php?p=75 205 Gender, Equality and Corruption: What are the Linkages? Policy Brief 01/2014, Transparency international https://images.transparencycdn.org/images/2014_Policybrief1_GenderEqualityCorruption_EN.pdf 29 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV For example, under the capacity development components, the focus has been only on gender participation (e.g., number of women participating in capacity development events). Although the opportunity existed to identify perceptions, understanding, and challenges to the achievement of gender equality in the broader anti-corruption framework (for example, through post-training evaluation questionnaires or through surveys on government’s effectiveness to tackle corruption), feedback questionnaires did not include options to receive gender-sensitive information, as there were no gender-specific questions.206 GAI still needs to work on and measure “lasting changes in the power and choices of women over their own lives, while tackling the root causes of inequality.”207 Exhibit 3 displays survey responses related to opinions about GAI’s support for gender equality during its implementation. Most survey participants stated either that GAI insufficiently (25 percent) or partially (25 percent) addressed gender, while many (31 percent) did not have that information. However, a total of 10 percent declared that GAI did not address gender equality at all; opposite to this, 10 percent of participants in the survey stated that GAI addressed gender in a satisfactory manner. Unfortunately, participants in the survey who expressed negative or relatively negative opinions did not justify or explain these views and perceptions. In addition, interviewed partners highlighted the importance of the technical support that GAI provided; at the same time, the partners were unaware of the links between gender equality and anti-corruption, expressing only limited knowledge of gender-sensitive practices.208 This lack of justification makes it difficult for the evaluation team to make specific recommendations or elaborate on the lack of gender inclusion. 3. CONCLUSIONS AND RECOMMENDATIONS 3.1 CONCLUSIONS EVALUATION QUESTION 1: GAI ACCOMPLISHMENTS Conclusion 1. GAI has been effective in delivering results and meeting targets, creating a solid basis to continue activities at the national and local levels to further strengthen good governance. GAI has achieved concrete and visible results during this period of implementation. Most notably, the results related to enhancing transparency at the local level and developing technical capacities of SAI. The evaluation team found that GAI has strengthened the capacities of the critical institutions responsible for improving government accountability, particularly at the local level and for participating institutions, such as SAI and LSGs. Despite GAI’s efforts and results, the indicators showed a decline in government accountability at the national level due mainly to lack of commitment.209 EVALUATION QUESTION 2: LINKING THE ACTIVITY’S OBJECTIVES AND RESULTS 206 The analysis of the MELP. 207 https://www.undp.org/content/undp/en/home/sustainable-development-goals/goal-5-gender-equality.html 208 KII notes- national partners 209 The analytical reports such as Nations in Transit 2020 highlighted the “country’s increasing slide towards authoritarianism,” with raising concerns over media freedom, political control, and the free and fair conduct of elections, among other issues. The European Commission’s 2020 Report on Serbia recognized limited efforts or absence of any progress in the core governance areas. Specifically, the country is lagging behind with the reform of the judiciary and fight against corruption, with numerous challenges to ensure respect of fundamental rights and freedom of expression- https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf EXHIBIT 3: PERCEPTIONS OF GAI’S GENDER INTEGRATION (N=72) HAS GAI EFFECTIVELY PROMOTED AND ADDRESSED GENDER EQUALITY DURING IMPLEMENTATION OF ACTIVITIES? Not at all 10% Insufficiently 25% Partially 25% Completely 10% Don’t have information 31% USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 30 Conclusion 2. GAI’s results show that it is meeting its objective, with notable results at the policy and institutional levels. GAI’s results have contributed to sectoral policies. The policy cycle model confirmed that positive results from GAI have influenced the designing of policy tools. GAI results also contributed to organizational developments, particularly of SAI and participating LSGs. Despite this, GAI has faced challenges in cooperating with ACA and strengthening its capacities and delivery of services. Conclusion 3. GAI has been implemented efficiently, in line with the work plans, and demonstrated flexibility to stakeholders’ needs. GAI established a clear system of work planning, to complete the Activity on time, within the approved budget and standard of quality. GAI also established an effective monitoring, evaluation, and learning system, with a team responsible for data￾collection and analysis. The proposed indicators are relevant to measure progress, but important elements of GAI remain outside of the monitoring system without appropriate indicators to measure progress and results.210 The reporting practice has been solid, but a specific results-oriented and gender sensitive approach, including references to adopted indicators, has been limited. Conclusion 4. The GAI team, including its senior management, contributed to the effective implementation and achievement of results. GAI has been a flexible and reliable partner, accountable for the achievements of results. These elements, together with the strong partnership and technical capacities of GAI staff, have been critical factors contributing to GAI’s effectiveness and its flexible and responsive implementation. Implementing partners commented positively about GAI’s responsiveness and involvement with design, planning, and implementation, while restating the relevance of the technical assistance that is available through the Activity. Yet, GAI did not have a steering or advisory mechanism to provide strategic guidance and coordinate development interventions in specific areas. This body could be an effective forum for greater involvement of the high-level national partners to genuinely drive development efforts, ensuring strong national leadership in the future. EVALUATION QUESTION 3: SCOPE OF WORK EXPANSION Conclusion 5. GAI’s scope is well-defined and already tested services need to continue. In Section 2.3, the evaluation team describes potential areas for additional GAI involvement at the sectoral level (for example, the overall anti-corruption strategy), institutional level (for example, strengthening local anti-corruption councils’ capacity to monitor plan implementation or strengthening mechanisms for private sector whistle-blowers), and individual level (for example, addressing the needs of youth or enhancing understanding of participation benefits for citizens) to complement already planned and implemented activities. EVALUATION QUESTION 4: FUTURE PROGRAMMING Conclusion 6. GAI’s intervention logic has been well-established, based on a sound TOC, ensuring its alignment with national strategic priorities under the good governance framework. GAI effectively responded to the needs to establish more transparent and accountable institutions. GAI remained relevant during the entire period of implementation thus far (2018-2020) and has aligned itself with the needs of the country. The responsiveness of GAI to the changing environment in Serbia and the needs of the IOIs, specialized anti-corruption court units, and LSGs (as the main partners) are critical factors to its relevance. 210 For example, indicators could not adequately measure GAI’s performance in the areas, such as right to information, civic participation, complaint mechanisms. GAI did not introduce appropriate indicators to measure development of capacities of the target groups or the effects on anti-corruption education programs on the selected youth (e.g. Anti-corruption Legal Clinic and its participants). Also, GAI did not measure the effects of awareness campaigns and assess if the targeted audience has enhanced understanding of a specific topic. The MELP also did not include more specific indicators related to assistance to CPI and LSGs. 31 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV Conclusion 7. GAI has been effective in responding to national capacity development needs in areas of intervention. GAI has steadily and successfully developed the capacities of national partners to improve functioning and delivery of services (under all components). The actual sustainability of these results depends upon the continuing availability of these capacities and that they will continue to be requested upon the completion of GAI. Conclusion 8. The sense of national ownership over the achievements under GAI is due to effective partnerships and the involvement of the national stakeholders in its design and implementation. The participatory approach for the design and implementation of GAI has contributed to an enhanced sense of ownership, setting the ground for sustainability of results. The sustainability of GAI’s achievements, however, require additional attention and further efforts. Several external factors may undermine the sustainability of GAI interventions, such as the Government’s continued lack of commitment to implement reforms and anti-corruption measures, the deteriorating role of independent oversight institutions; unpredictable political developments; and the recent economic decline largely due to the COVID-19 pandemic.211 EVALUATION QUESTION 5: GENDER EQUALITY Conclusion 9. GAI has given relatively less attention to gender equality and gender mainstreaming. The MELP is not gender sensitive. The use of sex-disaggregated indicators has been limited, and GAI team did not design methods of data collection and analysis for systematization of gender-disaggregated information. 3.2 RECOMMENDATIONS Through its analysis of primary and secondary data, the evaluation team identified challenges to the implementation of GAI. The evaluation team proposes the following main recommendations to USAID and GAI to resolve these problems: Recommendation 1. The evaluation team recommends that GAI focus on the next stage of strengthening government accountability at national and local levels. The next stage should focus on implementation of identified organizational strategic priorities for the SAI, CPI, ACA, specialized anti￾corruption court units, and LSGs and follow up on improvement of services, by helping to build capacities, systems, and connections among the key partners/institutions. Recommendation 2. Development of research and analytical capacities of ACA staff remains a dominant priority for this organization and GAI is well-positioned to continue building on the existing achievements and extend its technical assistance. However, if the ACA's leadership continues to resist implementing its oversight mandate and the new procedures and policies that GAI supports, GAI should reconsider supporting ACA. Recommendation 3. GAI should remain flexible and responsive to the needs and priorities of the good governance framework. Concerning responsiveness, emerging priorities include: ● Systemic Level. To engage more actively in the policy making process related to anti-corruption, transparency, and accountability, GAI should codify and summarize knowledge in the core areas of its work and share it with the government as the basis for or evidence to create a more favorable legal and policy environment in this area. GAI also should provide specific support to IOIs to respond to the corruption risks and challenges posed by COVID-19. ● Institutional Level. GAI should: 1) continue support to local authorities – priorities could include: a) designing sound and realistic monitoring system for LAPs; b) enhancing the capacities of local 211 https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/serbia_report_2020.pdf and https://freedomhouse.org/country/serbia/nations-transit/2020 USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 32 independent anti-corruption bodies, in charge of LAPs monitoring; c) enhancing the capacities to implement recommendations to improve budgeting process and local finances; and d) enhancing strategic communication skills of local officials in senior positions; 2) explore opportunities to engage with public enterprises, including public utility companies, and enhance their transparency, in line with the internal rules, legal provisions, and requirements; 3) expand the reach of and embrace private sector interest, their roles and responsibilities in advancing good governance, and to fight corruption efforts in the country; 4) strengthen and establish whistleblower protection mechanisms for the private sector, enabling them to report on corruption; 5) strengthen capacities of the NA and its relevant committees to read, understand, and comment on the oversight institution reports, and present findings and recommendations; 6) strengthen the technical capacity of prosecutors; and 7) Explore opportunities to support the network of DPOs.212 ● Individual Level. GAI should: 1) invest in additional efforts to reach youth, offering capacity development programs to prevent corruption, including anti-corruption clinic models, on-line training programs, and the use of social networks and on-line discussion forums; 2) prepare and implement programs on anti-corruption framework for the journalists; and 3) increase “demand” for participation, by working with the public to create broader interest and willingness to take part in local decision-making processes. Recommendation 4. GAI should define appropriate qualitative and quantitative indicators to enable GAI to adequately measure progress under components and capture GAI’s progress towards its objective.213 The evaluation team recommends that GAI provide a well-balanced combination of qualitative and quantitative indicators to capture changes and results attributable to the Activity, using national indicators and targets to the extent possible. The evaluation team also recommends including gender-sensitive indicators with a focus on “gender transformation,” identify assumptions and risks, and implement a sound risk management system. Recommendation 5. In addition to including gender-sensitive indicators, GAI should consider options to better integrate gender priorities in its activities and the overall anti-corruption framework. Additionally, GAI should consider activities and approaches that go beyond gender participation. Recommendation 6. GAI should continue implementing its systemic approach to capacity development, following needs assessments of the main stakeholders and partners. Considering the current stage of the governance systems, structures, and capacities of partner institutions in Serbia, GAI should consider a long-term, needs-based capacity development approach. The evaluation team recommends that GAI design a comprehensive training evaluation approach based on the Kirkpatrick model, assessing four levels of learning: 1) participant satisfaction with the training; 2) immediate change in individual knowledge and skills; 3) change in individual performance back in the workplace; and 4) change in the overall performance of the institution. Finally, GAI should prepare a plan to hand-over training programs to the relevant national institutions and prioritize preparing them to assume these responsibilities. 212 GAI’s role has been to support analysis of the budgeting processes and prepare recommendations for improvements; however, the Action did not provide follow up support, to implement recommendations. 213 For example, indicators could not adequately measure GAI’s performance in the areas such as right to information, civic participation, and complaint mechanisms. GAI did not introduce appropriate indicators to measure development of capacities of the target groups or the effects on anti-corruption education programs on the selected youth. Also, GAI did not measure the effects of awareness campaigns and assess if the targeted audience has enhanced understanding of a specific topic. The MELP also did not include more specific indicators related to assistance to CPI and LSGs. 33 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV ANNEX A. EVALUATION STATEMENT OF WORK The United States Agency for International Development (USAID) in Serbia seeks the services of a Contractor to conduct a performance evaluation of the USAID-funded project “Government Accountability Initiative” (GAI). C.1. BACKGROUND The goal of this activity is to strengthen the capacities and connections of key Serbian stakeholders to increase government accountability at the national and local levels. The development hypothesis is that if the skills and connections of government institutions and oversight bodies are increased, then Serbia will have a stronger foundation for more responsive and accountable governance, and in order to achieve its purpose, this activity focuses on three objectives: local government is more accountable; oversight of government institutions strengthened; and adjudication of corruption cases increased. Within these three areas, the Contractor must: a) support efforts to increase the inclusiveness of public decision making regarding public resource allocations and to encourage and incentivize local governments to operate in a more transparent and responsive manner; b) provide assistance to oversight institutions to strengthen monitoring of government performance and public officials, and exercise mandates as defined per relevant legislation; c) provide assistance to specialized anti-corruption court units and help establish a system for the collection of unified statistics on corruption, distinguishing clearly between different types of criminal offences, the length of cases, outcome of the proceedings. Detailed background information on the SOW for this activity can be found in Attachment #1 of this RFTOP (“GAI SOW”). C.2. PURPOSE OF THE EVALUATION USAID/Serbia requires a full, evidence-based and independent mid-term performance evaluation of the USAID/Serbia Government Accountability Initiative (GAI) Activity, and a report that provides a qualitative as well as quantitative evaluation of the progress made towards achieving expected results per the contract. Specifically, the purpose of the evaluation is to: • Examine the effectiveness of project activities implemented to date; • Determine whether the project has achieved planned results; • Identify gaps in performance against targets; and • Provide recommendations on closing these gaps in the upcoming years of the project. The Contractor must not simply provide an accounting of performance against targets but provide independent analysis on why targets were realized, met, or significantly surpassed. The Contractor must also provide an analysis of the reasons behind gaps between expected and actual performance, including an identification of the likely source(s) of these gaps (design, project management, changes in operating environment, change in context, success factors, if contractor’s approach has led to intended results and goals, etc.). C.3. EVALUATION QUESTIONS, METHODOLOGY AND DATA SOURCES Evaluation Questions: Evaluation questions that Contractor must address are as follows: 1. To what extent is the program on track to accomplish the inputs, outputs and likely outcomes expected under all three objectives of the activity? 2. What link can be determined between progress toward the activity's objectives and the activity's results? USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 34 3. What other areas of interventions could fit under the SOW beyond what is already planned? 4. What are the interventions with greatest traction, unintended outcomes and/or lessons learned from USAID assistance that the Mission can use in decision making for future programming? Did the activity affect gender equality within the area(s) of focus? Evaluation Methodology: In developing an appropriate methodology, the contractor must consider potential limitations and incorporate best-practice mitigation measures. The evaluation must be supported by relevant data and information gathered from meetings with beneficiaries and all the relevant stakeholders. The evaluation team should clearly demonstrate links between the data collected and the subsequent findings, conclusions, and recommendations drawn. Where applicable, responses to the questions above and resulting data must be dis-aggregated by sex and gender. Available Data Sources: Suggested data sources include, but are not limited to: (a) secondary data/background documents, (b) activity plans, outputs, and reports, (c) key informant interviews, (d) focus group discussions and (e) survey(s) of project stakeholders and participants. Contractor must place emphasis on the collection of reliable empirical data and/or objectively verifiable evidence, as opposed to anecdotal evidence. Where surveys or interviews are used, appropriate sampling and questioning techniques must be utilized to ensure representative results; where references are made to data generated by GAI and their partners, it must be complemented by references to independent data sources and any significant data differences must be explained. The contractor will have access to all performance data, M&E database, surveys, and other deliverables, that have been collected and reported by the implementing partner as part of their Performance Monitoring and Evaluation plan for GAI. The contractor must also review the following documents in preparation for the GAI performance evaluation: • Relevant project reports, assessments, annual and life of project work plans. • Full data set in their formats available and collected by the implementing partner. • Annual performance reports of GAI partners and sub-partners. • Anti Corruption Assessment conducted prior to the GAI design. • USAID/Serbia and Montenegro’s 2011-2015 Amended Strategy and 2013-2017 CDCS. In addition, the evaluation team must consult with key informants and stakeholders, including: • Meetings with USAID/Serbia. • Meeting with Checchi’s home office key staff and Belgrade office GAI staff. • Meetings with key current/former GAI implementing partners, sub-contractors, and sub￾grantees in Serbia. • Meeting with relevant officials of the Government of Serbia (such as the Ministry of Justice), oversight institutions (such as SAI, ACA), and anti-corruption court units. • Meetings with other donors, including the EU Delegation, and those donors engaged in this area. C.4. GENERAL CONTRACTOR REQUIREMENTS – STATEMENT OF WORK The Contractor must conduct a methodologically sound evaluation which must be fully compliant with USAID’s Evaluation Policy and Automated Directives System (https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf), and must utilize tested and scientific evaluation methodologies. 35 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV C.5. TEAM COMPOSITION The Contractor must utilize a team with the experience and skills necessary to complete this task as specified in Section F.4. C.6. SCHEDULE AND LOGISTICS The schedule (with precise due dates for deliverables) will be defined in the work plan before the start of the in-country field work. Prior to commencing the field work, all team members will review background program documents to gain a firm understanding of the situation in Serbia and the activity. The Contractor will be responsible for logistical arrangements for conducting the evaluation, including scheduling meetings, making copies of documents and drafts, making travel arrangements, etc. Field visit logistics will be organized and managed by the Contractor. This includes travel in-country transportation. No international travel is anticipated. USAID/Serbia will provide the team with input and guidance in setting up a schedule of interviews and site visits, but the responsibility for the schedule for the evaluation resides with the Evaluation Team. The Contractor is responsible for ensuring the safety and well-being of Contractor’s staff and shall apply accepted practices in accordance with the local laws and health authorities’ directives. USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 36 ANNEX B. ACTIVITY BACKGROUND USAID/Serbia established “Accountability of Key Democratic Institutions Strengthened” as the first of its two Development Objectives (DOs) in its Country Development Cooperation Strategy (Fiscal Years 2013 – 2020), to address the challenges in sustaining accountable and representative governance in Serbia. These challenges result in part from the lack of adequate checks on central authority and sufficient opportunities for citizens to hold the Serbian government and elected officials accountable. Under this DO, USAID/Serbia provided on-going support to strengthen the accountability and representativeness of key democratic institutions in Serbia, with the goal of improving governance and furthering Serbia’s efforts towards EU accession. Directly tied to this DO, USAID/Serbia funded the GAI, a four-year activity with the overall objective to strengthen the capacities and connections of key Serbian governmental institutions and stakeholders, resulting in increased government accountability at the national and local levels. To achieve this objective, the Activity is structured according to three IRs: 1) increase public participation in local government decision-making and oversight; 2) increase follow-up to oversight institutions’ recommendations; and 3) increase adjudication of corruption cases. This $8.5 million, four-year activity started in February 2018 and will end in February 2022. In this section, the evaluation team gives an overview of the Activity’s key components, organized according to each IR. Corruption is principally a governance issue which indicates failure of institutions and a lack of institutional capacity to manage society by means of a framework of social, judicial, political and economic checks and balances.214 When these formal and informal systems break down, it becomes harder to implement and enforce laws and policies that ensure accountability and transparency. From an institutional perspective, corruption arises when public officials have wide authority, little accountability, and perverse incentives, or when their accountability responds to informal rather than formal forms of regulation. Considering this, the focus of GAI has been to strengthen capacities and organizational development of the main IOIs, as well as strengthen the operational and functional links between them. Furthermore, GAI assistance at the local level strives to enable targeted LSGs to fulfil their commitments and obligations by promoting constructive interaction among local authorities and citizens. These activities have been infused with the principles of anti-corruption and good governance. The GAI TOC provided an overview of these and other challenges and issues, enabling the evaluation team to analyze systemically the formulation of GAI’s intervention logic and establishment of its chain of results.215 We present the GAI TOC in the box below. GAI Theory of Change216 “The GAI TOC is that if select branches and levels of Serbian government are provided tools for good governance, they will have the will and ability to apply them constructively to result in improved governance for all Serbian residents. ● Specifically, we believe that sound examples of good governance at the local level, applied in ways that show clear benefit to the governments themselves, will prove appealing and durable, so that counterpart LSGs continue to apply them, and non-counterparts are interested in adopting them. This in turn will increase opportunities for members of the public to participate in and evaluate decisions by LSGs. 214 The principles at the heart of good governance are the same as those at the heart of democracy: Equity; Participation; Pluralism; Partnership; Subsidiarity; Transparency; Accountability; Rule of law; Effectiveness; Efficiency; Responsiveness; and Sustainability. 215 OECD DAC definition: “Chain of results is the causal sequence for a development intervention that stipulates the necessary sequence to achieve desired objectives beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts, and feedback” 216 USAID/Serbia GAI MELP. 6 February 2018 – 5 February 2022 (October 19, 2018) 37 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV ● We believe that support to independent oversight bodies can improve the capacities of some, and slow or arrest the declining role of others, such that the oversight bodies continue to act as meaningful checks on government corruption. ● We believe that building the capacity of new anti-corruption courts and PPOs and identifying bottlenecks in corruption case processing will give counterparts the ability to increase the adjudication of corruption cases if they are motivated to do so. Overall, these results will lead to an increased ability of independent and judicial bodies and of the public to check malfeasance by government actors.” IR1: INCREASED PUBLIC PARTICIPATION IN LOCAL GOVERNMENT DECISION-MAKING AND OVERSIGHT Under IR1, the Activity is supporting efforts to increase the inclusiveness of public decision making regarding public resource allocations and to encourage and incentivize LSGs to operate in a more transparent and responsive manner. GAI anticipates that the adoption of good governance principles will offer members of the public a better understanding of how government works and enable them to participate in government decision-making processes, which will in turn reduce opportunities for corruption, while increasing transparency, accountability, and inclusiveness.217 At the Activity mid-point, GAI has iteratively provided technical assistance to 13 LSGs in areas in which they show interest and willingness to implement changes and new policies.218 Notably, the Activity provided assistance in developing LAPs and establishing of independent monitoring bodies (IMBs), and, in 2020, is transitioning to supporting LAP implementation and capacity building of IMBs.219 The evaluation team summarizes the key activities under IR1, as described in the GAI work plans for project years 1-3: LAP DEVELOPMENT AND OVERSIGHT. GAI provided technical assistance to 13 LSGs to develop and/or revise LAPs and establish independent monitoring bodies. GAI also developed and proposed mechanisms and tools to strengthen oversight of LAP implementation to be used primarily by IMBs. Together, this work will be used to raise public awareness in the LSG communities about the importance of creating and implementing sound local anti-corruption policies. GAI also focused on identifying and documenting experiences and successful practices to share with other LSGs, as well as with relevant stakeholders to raise awareness and provide guidelines to others. In 2020, GAI began to focus its technical assistance on LAP implementation and capacity building of IMBs and also began providing capacity building workshops for GAI-supported IMBs to share experiences with other IMBs that do not receive donor-funded technical assistance. TRANSPARENCY. In addition to supporting the development of LAPs, GAI focused on specific topics such as transparency, public participation, budgets and spending, public services, and whistleblower protection. GAI work closely with counterpart LSGs to make clear how and why these interventions can have both long and short-term benefits for the LSGs and the public they serve. Specifically, GAI developed a model LSG website aligned with legal requirements and good governance standards, as well as an open budget portal in line with international best practices. GAI also recommended necessary improvements of individual LSG websites, to ensure relevant information is made available to the public in a transparent and user-friendly manner. Supporting this work, GAI also promoted LSGs use of social media platforms by suggesting content and training them in their use. In partnership with Transparency Serbia (TS), the Activity also provided LSGs with action plans that included concrete steps required to improve the LTI survey scores. GAI is also assisting LSGs in aligning their internal procedures with the 217 USAID/Serbia GAI MELP. 6 February 2018 – 5 February 2022 (October 19, 2018) 218 LSGs were added to the Activity in two groups: Group One – Cities of Vranje, Šabac, and Sombor; and the municipalities of Raška, Vrnjačka Banja, Dimitrovgrad, and Sjenica; Group Two – Cities of Niš, Kragujevac, Sremska Mitrovica, Novi Pazar, the municipality of Žabalj, and the city municipality of Stari Grad. The combined population of the 13 LSGs is 1,046,000, representing 14.8 percent of the total population in Serbia. 219 USAID/Serbia GAI Work Plan: Year 3 – February 06, 2020 – February 05, 2021 (December 20, 2019) USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 38 new law, partnering with the Public Procurement Office to provide all counterpart LSGs training on the use of its procurement portal, and providing tailor-made technical assistance to interested LSGs to develop and implement communication strategies for better interaction with the public. PUBLIC PARTICIPATION. GAI is supporting LSGs’ efforts to increase public involvement in government decision making, particularly public policy development and resource allocation. Specifically, to improve the supply side of public participation, the Activity aims to build capacity of LSGs to improve initiatives or develop new initiatives to improve communication strategies, use appropriate channels to facilitate public participation, and provide adequate feedback to citizens. This includes LSG capacity to engage the public in formulating, monitoring, and measuring the impact of public policies. To strengthen the demand side, GAI seeks to create interest and demand among citizens, youth organizations, businesses, and CSOs to take an active part in making decisions on locally important topics, notably through developing and implementing public awareness campaigns. PUBLIC SERVICES. GAI is implementing technical assistance targeting specific services in each LSG participating in GAI, responding to local needs, which will provide opportunities to significantly reduce the prevalence of corruption and improve governments’ efficiency and the quality of services provided to citizens and businesses. Specifically, GAI is assisting LSGs in optimizing procedures and introducing public quality monitoring for selected services in each LSG, as well as developing new tools and procedures to enable LSGs to receive information and feedback on the quality of services and better respond to the legitimate public needs and expected level of quality of delivered services. WHISTLEBLOWER PROTECTION. GAI is providing both basic and more practical and focused trainings to designated persons in LSGs on protecting whistleblowers, as well as creating a network for sharing experiences and cooperation among the trained individuals. In addition, GAI supports counterpart LSGs with direct technical assistance for defining the most appropriate systems for whistleblowers protection. IR2: INCREASED FOLLOW-UP TO OVERSIGHT INSTITUTIONS’ RECOMMENDATIONS Under IR2, the Activity is working with the Serbian government oversight institutions to strengthen the ability of the ACA and SAI to monitor the performance of government and public officials, and to exercise their mandates as defined in relevant legislation. GAI expects that, by increasing the focus of IOIs’ resources on risk-based prevention and detection of corruption and improving communications and coordination with the NA, the public, and counterparts, IOIs will achieve more successful outcomes and greater public visibility and respect. Improved interface among the institutions and stakeholders will, in turn, contribute to more significant response by the NA and enforcement bodies to IOI recommendations.220 The evaluation team summarizes the key activities under IR2, as described in the GAI work plans for project years 1-3: ASSISTANCE TO SAI. GAI is helping SAI to enhance strategic planning and management, performance auditing, reporting and oversight processes, with the goal of increase SAI’s ability to execute its mandate. Notably, GAI staff are mentoring performance audit teams to enhance audit design and implementation, integrate financial and compliance audit expertise, and enhance reporting and recommendation follow-up policies and processes. ASSISTANCE TO ACA. GAI is providing ACA with assistance in strategic planning, risk-based monitoring of government officials’ conduct, and reporting with effective follow-up from action agents, with the goal of increasing ACA’s ability to execute its mandate. Particularly, GAI has focused on helping ACA to implement strategies to prevent and detect corruption; strengthen ACA’s oversight of asset and 220 USAID/Serbia GAI MELP. 6 February 2018 – 5 February 2022 (October 19, 2018) 39 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV income declaration statements; and support ACA’s training for government institutions to implement codes of ethical conduct, among other areas. SUPPORT TO THE COMMISSIONER FOR FREE ACCESS TO INFORMATION OF PUBLIC IMPORTANCE AND PERSONAL DATA PROTECTION (CPI). GAI is implementing activities to support CPI’s two key mandates regarding access to information of public importance and personal data protection. GAI is also working to raise awareness of citizens’ rights to access information from the government, including by collaborating with CSOs to identify strategies to improve their access to information. GAI also partnered with CPI to provide training at the local level on the requirements in the new Law on Personal Data Protection. Overall, the focus of GAI’s support to CPI has been sustaining the attention of all institutions on freedom of access to public information, particularly by ministries that propose laws related to freedom of information. IR3: INCREASED ADJUDICATION OF CORRUPTION CASES Finally, under IR3, GAI is working through the Serbian Judiciary’s structures to provide technical assistance to specialized anti-corruption court units and PPOs. The Activity’s work is designed to facilitate the implementation of the Law on Organization and Jurisdiction of State Organs in the Fight against Organized Crime, Corruption, and Terrorism Financing, which is also known as the Law on Organized Crime and Corruption. Specifically, GAI committed to providing technical assistance to anti￾corruption specialized court units to help them to implement the new law. This assistance will help them establish a system to collect unified statistics on corruption, distinguishing clearly between different types of criminal offences, the length of cases, and outcome of the court proceedings. This in in turn should enable justice system leaders to pinpoint obstacles and bottlenecks that impeded the successful adjudication of corruption cases.221 The evaluation team summarizes the key activities under IR3, as described in the GAI work plans for project years 1-3: SUPPORT TO SPECIALIZED ANTI-CORRUPTION COURT UNITS. To support the court units, GAI conducted an initial needs assessment which examined the institutional capacities, IT infrastructure, and utilization of existing case management systems of the PPOs and court units, given the required cooperation of state organs as stated in the Law on Organized Crime and existing interdependencies among the institutions. Informed by this assessment, among other activities, GAI organized two roundtables on the adjudication of corruption cases, which were facilitated by a U.S. District Court judge, to introduce U.S. best practices in adjudicating financial crimes and corruption. The Activity also organized two roundtables with judges of specialized anti-corruption court units to harmonize procedures. GAI also is exposing judiciary and prosecutorial communities to practices used in other countries, such as the use of forensic experts to reinforce the benefits of such interventions newly introduced to Serbia by the 2018 legislation. ELECTRONIC REGISTER OF CORRUPTION CASES. GAI is supporting the development of a new ERCC that will enable the anti-corruption court units to track and report on cases, with the goal to improve information sharing among institutions pursuing prosecution and adjudication of corruption cases. GAI completed an initial needs assessment and mapped key stakeholders and best practices in case management systems. The Activity also identified the main interlocutors who are currently amending regulations to establish standardized methodology for data collection, record keeping, and statistical reporting on criminal offences of corruption. GAI is now finalizing its database model and database dictionary to ensure the compatibility of catalogues with existing case management systems, including AVP and SAPO. GAI is identifying an IT company to develop the defined hardware and software architecture solutions for the ERCC. 221 USAID/Serbia GAI MELP. 6 February 2018 – 5 February 2022 (October 19, 2018) USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 40 ANNEX C. ANALYSIS OF KEY PERFORMANCE INDICATORS INDICATOR DEFINITION FINDINGS OBJECTIVE: TO STRENGTHEN THE CAPACITIES AND CONNECTIONS OF THE KEY STAKEHOLDERS TO INCREASE GOVERNMENT ACCOUNTABILITY AT NATIONAL AND LOCAL LEVELS OI-1. Open Government – WJP Baseline: 0.49 (2017/18) Target: 0.51 (2020) The World Justice Project (WJP) Open Government Index (OGI) uses 4 dimensions to measure government openness, publicized laws and government data, right to information, civic participation and complaint mechanisms. A score from 0 to 1, with 1 indicating the highest level of openness, based on household surveys and in-country expert questionnaires collected for the WJP’s Rule of Law Index.222 The issue remains if and to what extent the Objective 1ndicator (OI)-1 can help measure attainment of GAI’s objective. GAI has been working in some of the WJP OGI related areas (e.g., right to information, civic participation, complaint mechanisms). The evaluation team found that this indicator could serve to satisfy a broad contribution claim, without clear justification between the results of GAI and progress measured by this indicator. OI-2. LTI (TS) Baseline: 40.35 (2015) Target: 44 (2020) Average numerical score, on a scale from 0 to 100, on the LTI developed by Transparency Serbia, for Local Governments assisted by GAI. The initial baseline is the average score of all LGs covered by the 2015 Index (the LTI 2015 is more complete than the 2017 Index). As the project progresses, GAI will report on progress of LSGs GAI selects to support as well as the entire universe of LSGs covered in each year’s Index. The OI-2 LTI measures “government accountability at local levels,” and it is relevant to measure progress under the Objective. Reflecting different dimensions such as the analysis of web sites, documents, responses to requests for free access to information and direct insight into the functioning of services, this indicator directly correlates with the core areas of GAI support at the local level. OI-3. Number of public-sector employees and representatives of CSOs trained on good governance principles and accountability Baseline: 0 Target: 1,865 (2020) Training is defined as in-class or in-service technical training and assistance to impart knowledge and skills, as well as an exchange of ideas, values or culture, for public-sector employees and representatives of CSOs. Good governance principles are understood as defined by UNDP governance principles, and include accountability, transparency, rule of law, responsiveness, equity and inclusiveness, effectiveness and efficiency, and participation. Training intends to assist in the implementation of existing laws, enhance counterpart communication when performing mandates and tasks, strengthen inclusiveness and transparency in processes, and creating more accountable government on all levels. The training topics may include, but are not limited to, performance auditing, whistleblower protection, strategic communication, access to information, citizen/public participation and inclusiveness, and use of information system. The methodology will be adjusted to the topics of trainings and may include analysis and application of relevant professional standards and legislation, review and analysis of practical cases, presentation of basic concepts, roles, responsibilities, and international practices. The OI-3 indicator measures the dimension “strengthen capacities of the key stakeholders.” However, counting “number of public-sector employees and representatives of CSOs trained on good governance principles and accountability” is not a valid reference to strengthen capacities. This indicator could be useful at the result-level, but not at the objective level. 222 The World Justice Project® (WJP) is an independent, multidisciplinary organization working to advance the rule of law worldwide. The World Justice Project has three primary goals: i) Increased understanding of the rule of law and its foundational importance; ii) Greater rule of law adherence by governments; and iii) Multidisciplinary, home-grown cultures of the rule of law. More at: https://worldjusticeproject.org/ The effective rule of law reduces corruption, combats poverty and disease, and protects people from injustices large and small. It is the foundation for communities of justice, opportunity, and peace— underpinning development, accountable government, and respect for fundamental rights. 41 | MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA USAID.GOV INDICATOR DEFINITION FINDINGS IR1. INCREASED PUBLIC PARTICIPATION IN LOCAL GOVERNMENT DECISION-MAKING AND OVERSIGHT Number of policies or mechanisms adopted in line with procedures to increase public engagement in selected LSGs Baseline: 0 (2017/18) Target: 16 (2020) The number of policies or mechanisms that have been adopted in counterpart LSGs to increase citizen engagement. Policy is defined as a principle or rule, or set of thereof, adopted by counterpart institution to guide decisions and achieve rational outcome(s) in the area of good governance. Mechanisms includes citizen advisory committees, open public hearings, formal consultations such as comment periods, and other strategies that solicit and respond to public input. The evaluation team found that the proposed indicator for IR1 is valid and useful to measure and communicate the Activity’s progress in developing the capacities of government to engage citizens. The indicator is quantitative, measuring the number of policies or mechanisms that have been adopted in counterpart LSGs to increase citizen engagement. However, the indicator did not capture qualitative dimensions of these policies and mechanisms. Some of the examples of these dimensions could be the capacities for preparation and implementation of policies, functional aspects, changes or improvements that have occurred through implementation of polices and mechanisms. IR2: INCREASED FOLLOW-UP TO OVERSIGHT INSTITUTIONS’ RECOMMENDATIONS IR2.1.1. Financial benefit provided to the Government of Serbia and its citizens as a result of SAI audits Baseline: 671 million (2018) Target: 738 million (2020) The measure here is the amount of financial benefits identified through SAI audits, presented in Serbian Dinars. Indicators I IR2.1.1. and I IR2.1.2. refer to the improvement of functioning of SAI, are highly relevant to measure progress under GAI. Indicator I IR2.1.2 reflects on the improved institutional and technical capacities of SAI to oversee the performance of government institutions at all levels, measured by the number of mechanisms and practices adopted to implement policies (e.g., methods for conducting audits and communication with external stakeholders). Complementary to this, indicator I IR2.1.1. could be at the “outcome” level, measuring the effects of improved capacities through the amount of financial benefits identified through SAI audits. IR2.1.2. Number of mechanisms adopted by SAI to improve its capability to oversee government performance Baseline: 0 Target: 12 (2020) The measure here is the number of mechanisms SAI adopts, thus improving its capacities to oversee the performance of government institutions at all levels. “Mechanisms” are defined as practices adopted to implement policies and may include, for example, methods for conducting audits and communication with external stakeholders. IR2.1.3. Number of ACA referrals to PPO and Misdemeanor courts Baseline: 414 (2017) Target 525 (2020) The measure here is the number of ACA referrals that are made to PPOs and misdemeanor courts. Indicators IR2.1.3. and IR2.1.4. are relevant to measure GAI efforts to enhance capacities of ACA, by measuring its organizational development (e.g., established mechanisms to improve oversight capability of ACA), while also assessing its performance and functioning within the anti-corruption system. IR2.1.4. Number of mechanisms adopted by ACA to improve its capability to conduct oversight Baseline: N/A Target: 5 (2019); 5 (2020) The measure here is the number of mechanisms ACA adopts to improve its capacity to conduct oversight. “Mechanisms” are defined as practices adopted to implement policies and may include, for example, processes for reviewing asset disclosure statements and controlling conduct of officials and civil servants. IR2.1.5. Number of USG-supported anti￾corruption campaigns Baseline: 0 Target: 14 (2020), with 3 in 2018, 4 in 2019 and 7 in 2020. This indicator measures the number of campaigns supported by GAI. A campaign is defined as an activity aimed to raise awareness of the importance of accountability of government on all levels, which can be delivered through, but is not limited to supporting CPI and LSGs in media events, promotional material distribution and media Indicator I IR2.1.5. is relevant for GAI, measuring the number of campaigns it supports. The quantitative side of campaigns is important but without measuring a campaign’s qualitative components it is difficult to assess or quantify the effects of a campaign. It would be, for example, important to know if the targeted audience has an enhanced USAID.GOV MID-TERM EVALUATION OF THE USAID’S GAI ACTIVITY IN SERBIA | 42 INDICATOR DEFINITION FINDINGS appearances (TV, radio, newspaper) and online media. The campaign methodology and medium of delivery will be adjusted to suit the selected target audience. Key messages will be formed to enhance target audience reception and understanding. understanding of a specific topic. The MELP also did not include more specific indicators related to assistance to CPI and LSGs. IR3: INCREASED ADJUDICATION OF CORRUPTION CASES IR3.1.1. Number of corruption cases registered by the ERCC Baseline: N/A Target: 780 courts and 4725 PPP (2020 total) The number of corruption cases for which a case number is assigned by the prosecutor or judiciary and is registered by the ERCC, annually. For judicial cases, the figure counts original filings in the four special anti￾corruption units of higher courts (Novi Sad, Belgrade, Kraljevo, and Niš), but not appeals. “Corruption” is defined as cases falling under Article 2, clauses 3) and 4) of the 2016 Law on the Organization and Competencies of State Authorities in Combating Organized Crime, Terrorism, and Corruption. To avoid duplication of cases counted, the data will be reported separately for the PPOs and the courts. ERCC is expected to be available to users in FY2019. The proposed indicators are useful to measure some of the dimensions of IR3, as defined here. This is particularly the case to measure if and to what extent the ERCC has proven functional and delivered results. The assumptions under these indicators have been that the “ERCC is expected to be available to users in FY2019;” however, the ERCC was not established at the time of this evaluation (October 2020). GAI did not introduce indicators to measure other dimensions and results under IR3, such as enhanced capacities of the target groups or the effects on anti-corruption education programs on the selected youth (e.g. Anti￾IR3.1.2. Number of user corruption Legal Clinic and its participants). requests of ERCC data Baseline: 0 Target: 70 (2020 total) The number of user requests for corruption case related data from ERCC. ERCC is expected to be available to users in FY2019.