PERFORMANCE EVALUATION OF USAID’S JUSTICE ACTIVITY IN BOSNIA AND HERZEGOVINA Final Report March 22, 2019 This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of authors and do not necessarily reflect the views of USAID or the United States Government. ii TABLE OF CONTENTS TABLE OF CONTENTS ................................................................................................................................................................... ii ACKNOWLEDGEMENTS .............................................................................................................................................................. iii ACRONYMS ....................................................................................................................................................................................... iv LIST OF EXHIBITS ............................................................................................................................................................................ vi EXECUTIVE SUMMARY ................................................................................................................................................................... 1 PERFORMANCE EVALUATION OF THE USAID/BiH JUSTICE ACTIVITY ..................................................................... 6 EVALUATION PURPOSE, QUESTIONS, DESIGN, AND LIMITATIONS.............................................................................. 6 BACKGROUND INFORMATION ON JUSTICE ACTIVITY (JA).............................................................................................. 8 CONTEXT IN WHICH JA OPERATES................................................................................................................................................. 10 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS................................................................................................... 14 Evaluation Question 1 (EQ1) ...............................................................................................................................................14 Evaluation Question 2 (EQ2) ...............................................................................................................................................18 Evaluation Question 3 (EQ3) ...............................................................................................................................................45 ANNEXES ...........................................................................................................................................................................................49 ANNEX I: DETAILED PRESENTATION OF THE JA MEL INDICATORS (ACTUALS AND TARGETS) SOURCED IN HJPC ADMINISTRATIVE DATA................................................................................................... 50 ANNEX II: DETAILED PRESENTATION OF HJPC ADMINISTRATIVE DATA ON PROCESSING CORRUPTION CASES IN 2014 – 2017 IN BIH AND SELECTED NUMBER OF POs..................... 63 ANNEX III: HJPC DATA IN PROCESSING CORRUPTION RELATED CASES IN 2015-2017 BY KTK REGISTRY BOOK................................................................................................................................................................. 83 ANNEX IV: EVALUATION PURPOSE, DESIGN, AND LIMITATIONS .............................................................................. 84 ANNEX V: EVALUATION AND BRIEF ASSESSMENT STATEMENT OF WORK....................................................... 87 ANNEX VI: DOCUMENTS REVIEWED............................................................................................................................................104 ANNEX VII: RECOMMENDATIONS RELATED TO CRIMINAL PROCEDURE (INCLUDING ASSET FORFEITURE) STEMMING FROM THE JA ANALYSES AND ASSESSMENTS WHICH WERE ALSO REFLECTED IN THE EU ADOPTED DOCUMENTS (PEER REVIEWS AND TAIEX RECOMMENDATIONS).................................................................................................................................................109 ANNEX VIII: SUMMARY OF CORRUPTION-RELATED LITERATURE REVIEW........................................................119 ANNEX IX: DATA COLLECTION INSTRUMENTS...................................................................................................................121 ANNEX X: LIST OF INTERVIEWEES.................................................................................................................................................141 ANNEX XI: DISCLOSURE OF ANY CONFLICT OF INTEREST..........................................................................................146 ANNEX XII: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM USAID/BIH MISSION AND THE IMPLEMENTING PARTNER (IP) .........................................................................................................151 iii ACKNOWLEDGEMENTS USAID/Bosnia and Herzegovina (BiH) commissioned IMPAQ International (IMPAQ), through USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH), to design and conduct a performance evaluation of USAID/BiH’s Justice Activity (the JA) and a brief assessment of the BiH justice sector. Although the same team conducted both the JA evaluation and BiH justice sector assessment simultaneously, in this report MEASURE-BiH presents the results of the JA performance evaluation only. The evaluation team employed a rigorous research design and methodological approach to answer the evaluation and assessment questions, by examining the JA’s achievements in strengthening the professional status and performance of prosecutors and justice sector institutions to uphold public integrity and combat corruption in BiH. MEASURE-BiH’s evaluation team members who worked on this evaluation and prepared this report include Edis Brkić, evaluation co-lead and MEASURE-BiH Deputy Chief of Party; Davorin Pavelić, evaluation co-lead and MEASURE-BiH Chief of Party; Emina Ćosić Puljić, evaluation team member and senior research analyst at MEASURE-BiH; Salminka Vižin, evaluation team member and research analyst at MEASURE-BiH; Amila Mujčin, evaluation team member, MEASURE-BiH external consultant, and MEASURE-BiH research fellow; Boris Badža, evaluation team member and MEASURE-BiH GIS specialist; and Sanel Huskić, evaluation team member and MEASURE-BiH research analyst. The authors thank the institutions, organizations, and individuals that contributed to this evaluation. At USAID/BiH, Elma Bukvic Jusic, MEASURE-BiH Contracting Officer’s Representative, has been instrumental in the design of the evaluation. Jasna Kilalić, Justice Activity Contracting Officer’s Representative from USAID/BiH, contributed to the report by helping us understand the design of the Justice Activity and its management. We also thank Justice Activity implementation team members for their prompt assistance with the Justice Activity documentation. All of these organizations and individuals have contributed tremendously to this report. iv ACRONYMS ACCOUNT Anti-Corruption Civic Organizations' Unified Network AEA Association of Expert Associates and Advisors in Courts and Prosecutor’s Offices in BiH APIK Agency for the Prevention of Corruption and Coordination of the Fight against Corruption ARC OSCE “Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases” project, financed by INL BD Brčko District BiH Bosnia and Herzegovina BoR Book of Rules CDCS USAID’s Country Development Cooperation Strategy CEC Corruption and Economic Crime CEELI Central and Easter European Initiative Institute in Prague CMS/TCMS Case Management System/ Prosecutors Case Management System CSO Civil Society Organization DA Diagnostic Analysis of the Integrity of the Judicial Sector in BiH and Potential Risks of Corruption or Unethical Conduct in Judiciary DAD Diagnostic Assessment Document on the Enforcement Regime of Civil Claims in Bosnia and Herzegovina DCMS ODC Case Management System DO Development Objective DSG Disciplinary Sanctioning Guidelines EC European Commission EU European Union FBiH Federation of Bosnia and Herzegovina FPOSD Special Department of the PO FBiH GIS Geographic Information Systems HJPC High Judicial and Prosecutorial Council of Bosnia and Herzegovina ICITAP The International Criminal Investigative Training Assistance Program IMEP Independent Media Empowerment Program INL The United States Department of State, Bureau of International Narcotics and Law Enforcement Affairs IP Implementing Partner IR Intermediate Result JA USAID/BIH’s Justice Activity in Bosnia and Herzegovina JEI-BIH Judicial Effectiveness Index of Bosnia and Herzegovina JPTC Judicial and Prosecutorial Training Centers KI Key Informant KII Key Informant Interview MEASURE-BiH USAID/BiH’s Monitoring and Evaluation Support Activity MEL Monitoring, Evaluation and Learning Plan MoJ Ministry of Justice MoU Memorandum of Understanding NSCP-BiH MEASURE-BiH National Survey of Citizen Perceptions NIT Nations in Transit ODC Office of Disciplinary Counsel OPDAT Department of Justice Office of Overseas Prosecutorial Development Assistance and Training v OSCE Organization for Security and Cooperation in Europe PGI Prosecutor guided investigation PPP Prosecutor Partnership Program PO Prosecutor Office POAP Prosecutor Office Assistance Plan RAI Regional Anti-corruption Initiative RS Republic of Srpska RSSPO The Republic of Srpska Special Department for Suppression of Corruption, Organized and Serious Economic Crimes SAI Supreme Audit Institution SC HJPC Standing Committee SCL HJPC Standing Committee for Legislation SDC Swiss Development Cooperation SGIP Strengthening Governing Institutions and Processes SIPA State Investigation and Protection Agency SoW Statement of Work SPO PO Banja Luka the Special Department for Suppression of Corruption, Organized and Serious Economic Crimes TA Technical Assistance TO Task Order UNDP United Nation Development Programme USAID United States Agency for International Development USKOK The Croatian State Prosecutor's Office for the Suppression of Organized Crime and Corruption WGIA Working Group on the Integrity and Accountability of Judges and Prosecutors vi LIST OF EXHIBITS Exhibit 1: Number of indictments for corruption related crime in POs that implement PPPs (the JA MEL Indicator 1.1.2) ..................................................................................................................................................................19 Exhibit 2: Number of convictions for corruption-related crime in POs that implement PPP (the JA MEL Indicator 1.1.1) ..................................................................................................................................................................20 Exhibit 3: Aggregate results in processing corruption cases by all prosecutor offices in Bosnia and Herzegovina, 2014-2017.................................................................................................................................................23 Exhibit 4: Aggregate results in processing both corruption and economic crime cases by all POs in BiH in 2014 – 2017 .......................................................................................................................................................................24 Exhibit 5: The judiciary is effective in combating corruption, percentage of respondents agreeing with the statement ............................................................................................................................................................................30 Exhibit 6: Public officials who violate the law are generally identified and punished, percentage of respondents agreeing with the statement..................................................................................................................30 Exhibit 7: Judges are able to make decisions without direct or indirect interference by governments, politicians, the international community or other interest groups and individuals, percentage of respondents agreeing with the statement..................................................................................................................32 Exhibit 8: The JA input for EC Peer Assessment ........................................................................................................................34 Exhibit 9: Do you agree that disciplinary procedures against judges/prosecutors are…?...............................................35 Exhibit 10: Judges can be trusted to conduct court procedures and adjudicate cases impartially and in accordance with the law / Prosecutors can be trusted to perform their duties impartially and in accordance with the law, percentage of respondents agreeing with the statements ...................................36 Exhibit 11: Bribery among Judges: How much do you agree or disagree with statement that judges do not take bribes? ......................................................................................................................................................................37 Exhibit 12: Bribery among Prosecutors: How much do you agree or disagree that prosecutors do not take bribes? ...............................................................................................................................................................................37 Exhibit 13: 1.1.3 Conviction rate for corruption related crime in POs that implement PPPs ........................................51 Exhibit 14: 1.2.1 Number of unresolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs .........................................................................................................................................52 Exhibit 15: 1.2.2 Percentage of resolved criminal reports for corruption and economic crime versus total number of cases in prosecuting offices that implement PPPs .............................................................................53 Exhibit 16: 1.2.3 Number of resolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs .........................................................................................................................................54 Exhibit 17: 1.3.1 Percentage of investigations for corruption and economic crime completed within the one calendar year in prosecuting offices that implement PPPs ..................................................................................55 Exhibit 18: 1.3.2 Number of investigations for corruption and economic crime completed in one calendar year in prosecuting offices that implement PPPs ...................................................................................................56 Exhibit 19: 1.3.3 Number of ordered investigations for corruption and economic crime in one calendar year in prosecuting offices that implement PPPs .............................................................................................................57 Exhibit 20: 1.3.4 Number of unresolved investigations for corruption and economic crime in one calendar year in prosecutor offices that implement PPPs ....................................................................................................58 Exhibit 21: Aggregate results in processing corruption cases by all prosecutor offices in Bosnia and Herzegovina, 2014-2017...............................................................................................................................................64 Exhibit 22: Full Prosecutors’ Office names and corresponding abbreviations used in mapping .....................................65 Exhibit 23: Processing corruption criminal reports in 2014 – 2017 by cantonal/district POs ........................................66 Exhibit 24: Number of solved and rejected criminal reports in 2017 for 11 POs with big and medium caseload ............................................................................................................................................................................67 Exhibit 25: Number of newly opened investigations in 2017 for 11 POs with big and medium caseload ...................67 Exhibit 26: Processing of investigations for corruption cases in 2014 – 2017 by cantonal/district POs ......................68 vii Exhibit 27: Indictments (filed and confirmed) for corruption cases in 2014–2017 by POs .............................................70 Exhibit 28: Graphical output of processing corruption cases by PO Sarajevo ...................................................................71 Exhibit 29: Graphical output of processing corruption cases by PO Banja Luka ...............................................................72 Exhibit 30: Graphical output of processing corruption cases by PO Tuzla .........................................................................73 Exhibit 31: Graphical output of processing corruption cases by PO Zenica .......................................................................74 Exhibit 32: Graphical output of processing corruption cases by PO Bihac .........................................................................75 Exhibit 33: Graphical output of processing corruption cases by PO Mostar ......................................................................76 Exhibit 34: Graphical output of processing corruption cases by PO Travnik .....................................................................77 Exhibit 35: Graphical output of processing corruption cases by PO Siroki Brijeg.............................................................78 Exhibit 36: Graphical output of processing corruption cases by PO Bijeljina .....................................................................79 Exhibit 37: Graphical output of processing corruption cases by PO East Sarajevo ........................................................... 80 Exhibit 38: Graphical output of processing corruption cases by PO Doboj .......................................................................81 Exhibit 39: The JA Activity LogFrame ............................................................................................................................................94 1 EXECUTIVE SUMMARY PURPOSE, EVALUATION QUESTIONS AND PRIMARY AUDIENCE IMPAQ International (IMPAQ) was commissioned by USAID/Bosnia and Herzegovina (BiH) through the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct a performance evaluation of the USAID/BIH’s Justice Activity (JA) in Bosnia and Herzegovina, implemented by Millennium DPI Partners, LLC, under Task Order (TO) No. AID-168-TO-14-000001. The Activity started on September 23, 2014 and will continue until September 22, 2019. USAID/BiH’s JA is designed to support USAID’s broader democracy and governance goal of achieving more functional and accountable institutions that meet BiH’s citizen’s needs (Development Objective [DO] 1), as expressed in USAID’s Country Development Cooperation Strategy (CDCS). The JA’s goal is to contribute to USAID/BiH Intermediate Result (IR) and Project 1.1.—more effective judicial, executive, and legislative branches of government. Specifically, the Activity also is expected to further USAID’s IR of making government more responsive to citizens by strengthening the capabilities of the justice sector’s actors (IR1.1.1). To meet its goal, the JA delivers assistance to the BiH judicial branch through two principal components: Component 1 (C1): Strengthening the professional status and performance of prosecutors; and Component 2 (C2): Strengthening justice sector institutions to uphold public integrity and combat corruption. The JA performance evaluation examined the following issues: the appropriateness of the JA design; sequencing, flexibility, and resource allocation during implementation; beneficiary needs; achievement of expected results, with particular attention paid to results achieved to increase the overall quality of the work of prosecutors in BiH; and results of activities implemented directly with the High Judicial and Prosecutorial Council of Bosnia and Herzegovina (HJPC) related to integrity and discipline. In addition, the evaluation explored opportunities to complement ongoing JA interventions and lessons learned from implementation of the current contract. The primary audience for this evaluation is USAID/BiH, who will use the findings, conclusions, and recommendations to assess the achievements of the JA and inform further interventions in the BiH justice sector. This report reflects data collected by the evaluation team between June 13 and July 27, 2018. EVALUATION METHODS AND LIMITATIONS To conduct this evaluation, the evaluation team used a mixed-methods approach that included an extensive desk review of implementation documents, relevant reports of international organizations, and local government documentation. We also analyzed administrative data on processing corruption and economic crime cases from 2014 through 2017, MEASURE-BiH survey data on the publics’, judges’, and prosecutors’ perceptions of the judiciary from 2015 through 2017, and semi-structured interviews with 107 key informants (KIs) through 55 key informant interviews (KIIs). The main limitations of the evaluation include the possibility of social desirability bias, data contamination, lack of complete information, and recall bias. FINDINGS AND CONCLUSIONS The evaluation team analyzed context in which Justice Activity (JA) operated during its life cycle and found that JA operated in a highly challenging environment, in which a number of unforeseen complications arose (i.e., delays in establishing Prosecutor Office (PO) Special Departments, various personnel issues 2 encountered in PO BiH, establishment of a new PO in Prijedor, and decisions of the highest ranking courts on the unconstitutionality of provisions of the BIH Criminal Procedure Code and the jurisdiction of cantonal courts/POs in the absence of the establishment of the FBiH Supreme court/PO Special Departments). During implementation, individual POs were understaffed, underequipped, and underfinanced. POs were limited in their day-to-day activities, frequently lacking resources for their work and conducting investigations. In the period of the JA’s implementation there was no political support for work of POs in processing corruption cases (i.e., mainly manifested in allocation of inadequate budgets, unforeseen cuts of allocated budgets, and delays in paying expert witnesses and for services provided to POs). An absence of political support also has manifested in the low levels of support that other governmental agencies show for POs (i.e., law enforcement agencies, the Supreme Audit Institutions (SAIs), the Agency for the Prevention of Corruption and Coordination of the Fight Against Corruption (APIK), etc.). Given this context, the JA’s implementation required flexibility and adaptation, which was achieved through communication with stakeholders, and adjustments in planning and delivering technical assistance (TA). This adaptability was recognized by the stakeholders and confirmed by the evaluation team as further discussed in this report. The JA’s interventions have been appropriate and met beneficiaries’ needs, to the extent possible given available resources, in institutional strengthening for prosecuting and processing CEC cases. KIs confirmed that the JA’s design appropriately tailored implementing activities to meet the beneficiaries’ needs, while POs and other stakeholder KIs positively noted the JA’s ability to adapt and tailor assistance to the diverse demands, needs, and operational dynamics of the POs. The majority of beneficiaries, including representatives of the High Judicial and Prosecutorial Council’s (HJPC) Standing Committees (SC), stated that they valued the JA’s assistance. The evaluation team found that beneficiaries vetted all phases of the JA implementation (design, development, implementation, and completion). Beneficiaries and stakeholders also confirmed that the JA management was professional and flexible. Using Prosecutor Partnership Programs (PPPs) and tailored activities specified in Prosecutors’ Offices Assistance Plan (POAPs), the JA delivered the principal assistance directly to individual POs, strengthening organizational leadership, planning, balancing allocation of resources, improving performance in prosecutor offices, and upholding public trust and integrity through prosecution of corruption and other serious crime. PPPs have been established with 19 POs as formally requested by the JA contract. POs stated that JA assistance provided them with tools to better organize their work and manage cases more effectively. The team found that JA interventions and assistance related to strategic planning were important and useful. The team also found that the JA assistance to POs related to improving the quality and transparency of POs’ public relations (PR), including the development of POs’ PR strategic documents, monitoring, evaluation and learning (MEL) plans, monitoring of corresponding performance indicators, and upgrades of POs’ websites and improvements in content management, which were especially useful to those POs that did not have a PR officer. In addition, many POs confirmed that the JA did address some of their problems related to working conditions and lack of resources by providing direct material assistance, including furniture, IT and various specialized equipment. KIs identified the most useful elements of the JA to be the provision of experts in forensic accounting, specialized training, material assistance, the development of Integrity Plans, and the Guidelines for Preventing Conflict of Interest, including Asset Declaration Forms. The JA worked jointly with the Swiss/Norwegian “Strengthening Prosecutor Offices Capacities” project on several activities. The two projects coordinated well and complemented each other’s activities, which satisfied the beneficiaries who found that both projects were flexible and cooperative. The JA and the Organization for Security and Cooperation in Europe (OSCE) “Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases” (ARC) project, financed by the United States Department of State, Bureau of International Narcotics and Law Enforcement Affairs (INL) are working on similar tasks related to trial monitoring and analysis of judgments/verdicts in corruption cases. The 3 KIIs and document review revealed that there is little cooperation between the JA and OSCE ARC project in these activities. The evaluation team concludes that USAID/BiH and INL should further examine similarities and differences in their efforts, but in particular, they should identify areas for joint work of the JA and the OSCE ARC project on analysis of corruption cases and verdicts and possibilities for joint presentation of these results. By design, the JA lacked quantifiable expected results or milestones. The JA contract did not specify quantifiable results, allowing for flexible targets throughout implementation. This provided the JA flexibility in meeting the beneficiaries’ needs, avoiding potential duplication of interventions with the Swiss/Norwegian project, and adapting to newly emerging developments. However, this approach complicated the evaluation team’s process for monitoring and measuring achieved results against contractual expectations. Monitoring progress of the Activity implementation was a challenging task and resulted in four modifications to the Activity’s MEL Plan, including a revision of targets during the implementation period. In relation to the JA targets set in the JA MEL plan, the evaluation team identified two major groups of indicators. The first group is connected to direct TA delivered by the JA, while the second group consists of indicators sourced from the administrative data on processing CEC cases and perception data of public and judges/prosecutors. The JA largely achieved the expected results, or set the foundation to reach those results by the end of the contract, with respect to the first group of MEL indicators. These indicators have been more under the JA’s control and include the following: the number of beneficiaries trained; number of judges and prosecutors who earned a certificate of specialized (two-year long) training in the investigation, prosecution and adjudication of corruption and financial crimes; number of courts and POs that adopted Integrity Plans; number of codes of conduct revised for improved standards in judicial conduct and others. Also, for several indicators, 2019 (the life of Activity) targets have already been met. The team did not find observable changes resulting from the JA with respect to the second group of MEL indicators. These indicators were sourced from the administrative and perception data (the HJPC administrative data on processing CEC cases in all phases of criminal proceedings, and surveys of the public, and judges and prosecutors on ability of the justice system to fight corruption). Despite the positive perception of beneficiaries of the JA’s interventions, these indicators do not show that the JA’s implementation resulted in observable changes in processing corruption and economic crime cases or changes in poor public perception. In addition, the team could not estimate if the JA will reach its life of the Activity targets for this group of indicators because the indicators and the data used to calculate them, are highly volatile or show inconsistent trends and patterns. On the other hand, the introduction of administrative/perception data in the JA MEL plan and their monitoring are positive developments, as their usage allows USAID to review results of the entire judiciary in processing corruption and economic crime cases and gain valuable information to inform decision-making and future programs. In reviewing these data, a reader should be aware that these data show aggregate result of both donor interventions (i.e., JA, Swiss/Norwegian project, EU Twinning Light projects), and efforts, or lack thereof, of domestic stakeholders. Due to the complexity and interconnectedness of these efforts, the team could not distinguish the influence of these individual parts on the overall values. Over the period of JA implementation, the performance of POs and their capacity to track and monitor CEC cases also substantially increased. In collaboration with the Swiss/Norwegian project, the JA contributed to the following achievements in this area: initiation of the Collegium of Chief Prosecutors, implementation of structured discussion about the achievements of the POs Strategic Plans and Annual Plans, and defining the quantitative indicators set to accompany the annual and strategic plans including indicators related to PR of POs, as per methodologies set in MEL plans. Although the JA provided assistance to HJPC changing the bylaws that regulate prosecutors’ performance, these changes are insufficient to motivate prosecutors to work on the most complex CEC cases. In addition, some 4 stakeholders continue to resist prioritizing CEC cases for processing, claiming that this leads to further backlogs in other types of cases. Cooperation among state/entity and justice sector institutions initiated with the JA assistance remains primarily donor driven, and the cooperation may decline with JA withdrawal. The JA managed to initiate cooperation between POs, and different government agencies and institutions that could contribute to prosecuting CEC cases. This cooperation includes initial meetings and, in some cases, negotiations between POs and these agencies and institutions have led to the establishment of procedures for future cooperation. However, POs’ opinions about the usefulness of this cooperation are unfavorable, as government agencies and institutions have not provided substantial support for the POs’ work. The JA’s efforts to support the Office of Disciplinary Counsel (ODC) (i.e. development of the ODC’s strategic plans, MEL plan and annual reports; trainings that resulted from a training needs assessment; a functional review of the ODC and recommendations for upgrading the ODC Case Management System (DCMS)) were implemented successfully. Moreover, the HJPC formally adopted the Guidelines for Determining Disciplinary Measures developed with JA assistance and ODC is using them to handle complaints and propose sanctions related to judges’ and prosecutors’ performance. The HJPC’s formal adoption and application of these documents in processing disciplinary cases indicates a significant endorsement given the judicial sector’s strict regulation. The JA’s Diagnostic Analysis of the Integrity of the Judiciary Sector in BiH and the Possible Risks of Corruption or Unethical Conduct in the Judiciary System (DA) and consequent studies (i.e., Analysis of System of Use of Expert Witnesses in Cases of Corruption, Organized and Economic Crimes, Analysis of Court Verdicts, Analysis of Asset Forfeiture) are generally perceived by beneficiaries as providing high quality assessments of justice sector issues. The JA’s analytical work and diagnostic studies began before the European Commission (EC) peer review missions. The results of the JA’s analytical work and diagnostic study were incorporated into, or at least similarly identified by EC peer review recommendations. In addition, some JA interventions, such as developing diagnostic studies of and guidelines for integrity plans, conflict of interest, and disciplinary sanctioning, functional analysis of the ODC, and asset declaration reform, all created the necessary foundations for providing the JA amendments to the new Law on the HJPC. As the recommendations of the EC peer review missions will guide continued reforms in the BiH judiciary, JA’s work (through development of laws and bylaws) in this area will become even more important. While major diagnostic studies and consequent analysis were of high quality, the evaluation team found that these studies were not translated into English, which prevents non-local-language speakers from understanding the documents. Despite the extensive onsite and offsite training provided by the JA, which beneficiaries welcomed and appreciated, the team’s analysis of KIIs showed that prosecutors, judges, and law enforcement agencies still lack expertise related to processing economic crime and corruption cases, as well as cases related to cybercrime. Further training in CEC is needed for prosecutors, judges, and police to address issues that have emerged systematically when processing CEC cases. Given the lack of expertise for providing CEC training in BiH, continued donor engagement in this area may be necessary. To build upon the JA delivered training, upcoming training from INL in forensic accounting represents an opportunity for staff from POs to further improve their knowledge and skills in this area. Activities related to enforcement of judgements and implementation to improve courts’ efficiency in enforcing judgments were in the early stage of development when the former Component 2 was terminated by USAID/BiH. The evaluation of former Component 2 activities was not included in the SoW for this performance evaluation. Nevertheless, through a review of relevant documents, the evaluation team recognizes that a similar approach and implementation methodology were planned within the former 5 Component 2 as was the case for DA. The evaluation team concludes that initial activities within the former Component 2 were performed correctly. The evaluation team used semi-structured interview protocols for KIIs to ask about JA activities as described in the JA’s contract and work plans. Nevertheless, during KIIs, participants provided relevant information about the BiH justice sector that was not directly related to the JA's tasks and performance. We discuss this information here, given its relevance for USAID/BiH's understanding of the sector and to provide context on the environment in which JA is operating. Namely, the evaluation team identified a noticeable lack of communication among stakeholders in the judicial sector. Judges and prosecutors are unaware of ODC’s activities and practice, and POs are unaware of individual practices of some POs related to improvements in processing CEC cases within the existing regulatory framework. In addition, HJPC members rarely visit individual POs to disseminate information on ongoing HJPC activities and receive information on PO needs. KIs pointed out a number of issues that negatively influence the work of justice sector institutions, including disciplinary proceedings, appointments, performance appraisals, current legislative and regulatory initiatives, and knowledge of issues and needs among judicial institutions, judges, and prosecutors. Further, KIs pointed out that these issues created serious impediments for POs and courts in effectively processing cases. RECOMMENDATIONS Most of the activities associated with Component 2 of the JA complement the EC peer review recommendations and related ongoing efforts to change the HJPC law. Therefore, they should be continued. To improve processing CEC cases throughout the remainder of the JA implementation, in addition to planned assistance to POs in PPP group 4, USAID should offer additional technical assistance to POs, to the extent possible, that are most likely to prosecute high-profile cases, including Republika Srpska Special Prosecutors Office for Organized Crime and Corruption (RSSPO), and cantonal POs in Sarajevo, Zenica, Tuzla, and Bihac. Further improvements to the prosecutors’ performance system are needed to provide prosecutors the time to focus on complex corruption cases. To achieve this, USAID should consider reviewing prosecutors’ performance criteria to create incentives for prosecutors to work on these cases and providing technical assistance to establish procedures to determine which cases meet the criteria of a complex corruption case. USAID should consider providing initial technical assistance to ODC in working with CMS/TCMS data, analytical skills and data mining to identify suspiciously processed cases. ODC was recently granted independent access to CMS/TCMS records, which potentially will enable ODC to conduct data-driven ex￾officio investigations and improve ODC’s investigative work. Despite this potential, current ODC staff lack the technical and analytical skills necessary to conduct these data-driven investigations. USAID should continue trainings in POs’ budget preparation and delivering specialized training to prosecutors. In addition, as part of building upon the JA delivered specialized training, USAID should look for possibilities to involve representatives of POs in INL’s training program for forensic accounting. The USAID and INL projects should seek synergies and coordinate their corruption trial monitoring and verdict analysis activities. USAID should facilitate visits of HJPC members to individual POs to share information related to the JA assistance provided to HJPC, and facilitate discussions between HJPC members and prosecutors on issues of mutual concern. Overall, further improvement is needed in coordinating donor supported interventions aimed at assisting institutions in the justice sector. All USG agencies operating in BiH include the fight against corruption as a priority. USAID should identify opportunities for synergies and maximize return on investment by reinforcing cooperation across USG activities and projects. 6 PERFORMANCE EVALUATION OF THE USAID/BiH JUSTICE ACTIVITY EVALUATION PURPOSE, QUESTIONS, DESIGN, AND LIMITATIONS PURPOSE USAID/BiH commissioned IMPAQ International (IMPAQ) through the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct a performance evaluation of USAID/BiH’s Justice Activity (the JA) in Bosnia and Herzegovina (BiH). The JA is being implemented by Millennium DPI Partners, LLC under Task Order (TO) No. AID-168-TO-14-000001. The JA started on September 23, 2014 and will continue until September 22, 2019. This report reflects data collected by the evaluation team between June 13 and July 27, 2018. The date of the final evaluation report reflects the timeframe in which USAID/BiH and the IP submitted their comments, and the evaluation team provided its responses. The performance evaluation provides USAID/BiH with an evidence-based and independent review of USAID/BiH’s JA, and qualitative and quantitative analyses of the JA’s performance. RESEARCH QUESTIONS IMPAQ conducted this evaluation using a rigorous design and methodological approach, which addresses the specified evaluation questions and provides insights into the JA’s progress in achieving the expected results. The evaluation questions (EQs) below broadly addressed the JA’s design, results, and lessons learned. EQ1: To what extent has the JA’s design been appropriate in terms of chosen activities and the beneficiaries’ needs? Do beneficiaries and stakeholders vet the chosen activities? How was the JA implemented in terms of sequencing, timing, resource allocation, flexibility, and cooperation with beneficiaries and stakeholders? How do beneficiaries and stakeholders perceive and value the JA’s work? EQ2: To what extent has the JA achieved expected results under both components as outlined by the contract, and what are the prospects of meeting life-of-activity targets specified by the monitoring and evaluation plan? More specifically, was the individually-tailored assistance to each PO appropriate, and what results were achieved to improve the overall quality of the work of prosecutors in BiH? Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient, and can these be considered successes? EQ3: Within the current task order (TO) scope, what opportunities can be identified to additionally complement ongoing the JA interventions? What are the lessons learned from implementation of the current TO? DESIGN AND DATA SOURCES The evaluation team used a mixed-methods approach and triangulated data across different sources of information. Whenever possible, the evaluation team used data from the following sources: 1. the JA implementation documentation, data, and records; 2. donor, state, and HJPC/PO official documents; 3. HJPC administrative data on cases processed by courts and POs, including data on processing corruption and economic crime cases from 2014 through 2017; 4. selected survey data from MEASURE-BiH’s National Survey of Citizen Perceptions (NSCP-BiH), conducted annually from 2015 through 2017; 5. selected data from the Survey of Judges and Prosecutors in BiH, conducted from 2015 through 2017 by MEASURE-BiH; and 6. semi-structured interviews with 107 key informants (KIs) through 55 key informant 7 interviews (KIIs). Out of 5512 KIIs, 19 were conducted with direct beneficiaries (representatives of 15 POs, HJPC, HJPC Standing Committees3 and Disciplinary Panels, HJPC Secretariat4 and Office of the Disciplinary Counsel (ODC), and the remained with stakeholders. The full list of all beneficiaries and stakeholders is provided in Annex X. The team triangulated data from these sources to address the same questions and sub-questions from multiple perspectives when possible. Comparing data enabled the team to more fully understand the issues and provide more confident findings. LIMITATIONS The main limitations of the evaluation include the possibility of social desirability bias, data contamination, lack of complete information, and recall bias. Social desirability bias: This type of response bias exists when key informants answer questions in a way that they feel will be viewed positively and may make the Activity look more favorable to the evaluation team. Therefore, the KIs that had intensive cooperation with the Activity may overstate the positive effects of the interventions and understate its negative effects. Data contamination: As other donor interventions and international organizations have been supporting the justice sector in similar areas of work (for example, the Swiss/Norwegian project), it may be challenging for respondents to isolate the JA’s contribution. In addition, the administrative data on processing corruption and economic crime cases show the collective results of interventions by all donors, and the efforts, or lack thereof, of domestic stakeholders; thus, it is not possible to isolate the JA’s singular contribution. Lack of complete information: Any error in the report that is a result of KIs inability or unwillingness to provide accurate answers. In many instances during the interview session, some of the respondents were unwilling to provide an answer or had no detailed knowledge about the Activity due to their tasks and roles (e.g. Deputy Chief Prosecutors, Assistants, Advisors) and therefore unable to provide an accurate answer. Recall bias: The JA’s implementation began in 2014 and respondents may not clearly remember its early activities. To mitigate these biases, we selected a broad range of stakeholders, both direct JA beneficiaries and other stakeholders in and around the BIH judiciary who were not exposed to JA interventions, to interview. We triangulated the respondents’ answers with historical data and Implementing Partner’s (IP) records. More information on the evaluation purpose, design, and limitations is presented in Annex IV. 1 One KI provided written answers to the interview protocol questions instead of having an interview with the evaluation team 2 Interviews with USAID/BiH and the IP are included in total number of KIIs. 3 Tree interviewed HJPC members participate in the work of seven HJPC standing committees or work groups. 4 Nine representatives of HJPC Secretariat departments or projects participated in the HJPC Secretariat interview. 8 BACKGROUND INFORMATION ON JUSTICE ACTIVITY (JA) USAID/BiH’s JA is designed to support USAID’s broader democracy and governance goal of achieving more functional and accountable institutions that meet BiH’s citizen’s needs (Development Objective [DO] 1) as expressed in USAID’s Country Development Cooperation Strategy (CDCS). The JA’s goal is to contribute to the USAID/BiH Intermediate Result (IR) and Project 1.1.—more effective judicial, executive, and legislative branches of government—and to further USAID’s IR of making government more responsive to citizens by strengthening the capabilities of the justice sectors’ actors (IR1.1.1). The JA accounted for the constraints created by BiH’s constitutional and political system, as stated in the JA’s contract, 5 and was intended to respond to the following challenges and needs facing the judicial sector in BiH:  the signaled intention of the executive and legislative powers at the state and entity levels to politicize and curb the independence of the judiciary by introducing several draft laws;  the inability of POs to apply management skills and conduct prosecutor-guided investigations with the police hampering convictions in corruption and serious crime cases;  justice institutions’ lack of skills to effectively analyze data to inform management decisions that allocate resources strategically;  the lack of fairness and transparency in HJPC’s operations, particularly in following a neutral and merit-based appointment process of judges and prosecutors; and,  the Office of Disciplinary Counsel’s (ODC) lack of transparency and guidelines to handle complaints and propose sanctions related to judicial and prosecutorial performance. The JA attempts to address challenges faced by the justice sector in BiH by achieving the following key objectives and outcomes, as specified in the JA’s contract:6  preserving the independence of justice sector institutions through greater self-accountability (and external accountability mechanisms such as civil society organizations (CSOs);  moving key justice sector institutions from planning and developing standards, to improving their performance based upon analysis of results;  strengthening the management and decision-making practices of the HJPC and POs, so that they allocated resources strategically to fight corruption and other serious crimes;  supporting on-site knowledge and skills application by front-line prosecutors and police handling corruption cases so that they work as a team and network across jurisdictions;  building consensus for key reforms within the justice sector and standardizing cross-jurisdiction cooperation between state, entity, and local actors;  using diagnostic studies to inform major policy solutions that improve system-wide performance and lead to strategies that will effectively fight corruption; and  increasing public confidence in the justice sector through professional regulation, accountability, and transparency. The JA’s initial design was aligned with provisions of the contract, dated September 21, 2014, which included three Activity components. USAID/BiH requested Modification No.1 of the Contract7 refocused the JA’s resources on work with prosecutors and justice sector institutions to better combat corruption (Components I and 3 of the JA) and eliminated work related to improving the efficiency of enforcing judgments (Component 2). The original Component 3 was renamed Component 2 after the Modification No.1 of the contract. Thus, the JA now has two components: Component 1 (C1): Strengthening the 5 Contract No.: AID-168-TO-14-00001, p.2 6 Contract No.: AID-168-TO-14-00001, p.1 7 Contract AID-168-TO-14-000001, Modification No.1, Date 09/29/2015, p.3 9 professional status and performance of prosecutors; and Component 2 (C2): Strengthening justice sector institutions to uphold public integrity and combat corruption. In accordance with the contract, under C1, JA designed interventions to achieve the following five outcomes: 1.1 POs have strengthened organizational leadership, planning, and performance. 1.2 Prosecutors perform functions more efficiently through balanced allocation of resources. 1.3 Prosecutors uphold public trust and integrity through prosecution of corruption or other serious crimes. 1.4 Prosecutors provide appropriate and accurate information to citizens to strengthen transparency and responsiveness. 1.5 Prosecutors’ status is improved through performance appraisals, merit-based career advancement, or incentives to prosecute cases. In accordance with the contract, the JA implements the Prosecutor Partnership Program (PPP) directly in all 19 POs to improve management and help achieve strategic objectives (Activity 1.1). In addition, the JA provides intensive, on-site training and mentoring to senior managers and front-line prosecutors, involving the direct application of skills to actual management issues and criminal cases (Activity 1.2). Through the PPP, the JA introduces office guidelines on case weighting and performance awards to motivate prosecutors to prosecute cases involving major corruption and serious crimes. The JA also helps POs process criminal cases more efficiently by improving staffing and creating more efficient work-flow procedures (Activity 1.3). In accordance with the contract, under C2, the Activity designed interventions to achieve six outcomes: 2.1 Prosecutors investigate and prosecute high-profile corruption and economic crime cases free from political or improper influence. 2.2 There is increased cooperation among state, entity, and local justice and other sector actors. 2.3 ODC is properly resourced to manage complaints procedures, autonomously review conduct of judges and prosecutors, and recommend appropriate sanctions. 2.4 Disciplinary proceedings are processed expeditiously and fairly, and decisions are subject to independent and impartial review. 2.5 Public trust in and respect for justice sector institutions is increased because the institutions are better able to act independently and impartially and be held accountable. 2.6 Prosecutors and judges are trained to identify elements of corrupt activities and to investigate and prosecute corrupt practices. Programming under Activity 2.1 to strengthen prosecutorial capacity to investigate and prosecute high￾profile cases of corruption and other serious crimes is largely executed through the PPP. In particular, the PPP strengthens the capacity of the Republika Srpska Special Prosecutors Office for Organized Crime and Corruption (RSSPO), and the JA disseminates RSSPO best practices to other POs.8 The JA works closely with the HJPC Standing Committee (HJPC SC) to build further consensus around strategies to fight corruption across jurisdictions. Activity 2.2 features a comprehensive diagnostic to analyze sources of corruption in the justice sector and other government sectors alongside regional diagnostics that further advance the consensus process and cooperation on local levels. Activities 2.3–2.5 reinforce integrity in the judiciary through improved performance by ODC staff, more transparency in the disciplinary process, and effective application of codes of conduct to all personnel in courts and POs through the ODC and with linkages to BiH’s anti-corruption agency and other government agencies. Activity 2.6 reinforces anti￾corruption efforts under Activity 2.1, bringing judges and prosecutors together to develop legal specializations in corruption and other complex criminal matters, including developing practical manuals and sentencing guidelines applicable to processing these types of cases. 8 Contract No.: AID-168-TO-14-00001, p.17 10 CONTEXT IN WHICH JA OPERATES In accordance with the Constitutional organization of BiH, the prosecutors’ office (PO) system in BiH at the outset of the JA consisted of four independent PO systems. At the BiH level, the relevant structure included the Prosecutor's Office of BiH (PO BiH), while at the entity level there were two separate systems, one in FBiH and one in RS. In RS, the prosecutors’ office system consisted of the RS Prosecutor’s Office (RS PO) and five district prosecutors’ offices.9 In addition, the Special Department for Suppression of Corruption, Organized and Serious Economic Crimes (SPO) existed within the District PO Banja Luka. The SPO’s re-allocation to the RS PO was anticipated. In FBiH, the prosecutors’ office system consisted of the FBiH Prosecutor’s Office and 10 cantonal POs.10 In addition, in FBIH, the establishment of the Special Department for suppressing corruption and organized crime was anticipated within the FBIH PO with exclusive jurisdiction over a number of serious crimes, including serious corruption cases under the FBiH Criminal Code. Finally, the relevant structure in Brčko District (BD) included the Prosecutor’s Office of BD. During JA implementation (September 2014 – present), the initial structure of PO system was changed; planned PO Special Departments were not formed as forecasted in the JA contract, and a new PO was established. On May 18, 2016, during the implementation period, the Law on the Republic of Srpska Special Department for Suppression of Corruption, Organized and Serious Economic Crimes (RSSPO), was passed and published in the Official Gazette. Through this law, the RSSPO was established, although with 20 month delay compared to the JA contract forecast and set requirement (“Timeline for implementation of the PPP is as follows: Year 1 (seven prosecutor offices, including RSSPO)”). 11 In accordance with the approved work plan, in Year 2, the JA provided Technical Assistance (TA) to the PO Banja Luka, while the SPO was still a part of PO Banja Luka and/or in a process of transformation into the RSSPO as stipulated by the law on RSSPO. The approved work plan activities were aligned with the situation on the ground but not with the JA’s contract forecast and requirement that JA strengthen capacity of RSSPO and disseminate the best practices in investigating and prosecuting high profile cases of corruption and serious crime to other POs (as set in the Contract: “… the Contractor will spread RSSPO best practices to other prosecutor offices.12”). Furthermore, a new PO - PO Prijedor was established as per the new Law on RS Prosecutor Offices, which was passed and published on August 16, 2016. Establishment of PO Prijedor also was not anticipated in the JA Contract. The establishment of PO Prijedor has resulted in the transfer of some cases and prosecutors from PO Banja Luka to PO Prijedor (accompanied with additional appointments of prosecutors). All of these changes and delays were outside JA’s control and the JA adjusted to the new circumstances through realignments in the work plans (which were approved by the USAID/BIH). With establishment of PO Prijedor, the total number of POs in BiH increased to 20, while the JA Contract requested that JA TA be provided to “all 19 POs.” 13 The JA contract forecasts experienced various challenges in FBiH as well. The JA Contract forecasted that JA would work with the Special Department of the FBiH PO (FPOSD) and designed Special Activities feature to respond rapidly if FBiH establishes a special prosecutors’ office. 14 Although the Law on Suppressing Corruption and Organized crime in FBiH (by which establishment of FPOSD was stipulated) 9 District POs in RS (as per the PO seat): PO Banja Luka, PO Doboj, PO Bijeljina, PO East Sarajevo and PO Trebinje 10 Cantonal POs (as per the PO seat): PO Sarajevo, PO Tuzla, PO Zenica, PO Orasje, PO Gorazde, PO Mostar, PO Siroki Brijeg, PO Livno, PO Travnik and PO Bihac 11 Contract No.: AID-168-TO-14-00001, p.7 12 Contract No.: AID-168-TO-14-00001, p.17 13 Contract No.: AID-168-TO-14-00001, p.7 14 Contract No.: AID-168-TO-14-00001, p.3 and p.22 11 was passed and published in the Official Gazette of FBIH on July 23, 2014 (before design of the JA activity), FPOSD was not established during JA’s implementation period. In the absence of the establishment of special departments (in both the FBiH Supreme Court and FBiH PO), the Supreme Court of FBiH ruled15 in 2015 that adjudication of relevant crimes shall remain within the jurisdiction of the cantonal courts until the Special Departments are established, thus the competent POs (corresponding to their Cantonal courts) remained in charge of prosecuting corruption and other serious criminal cases. As a result of this decision, the JA had an opportunity to adjust its work activities and deliver TA to the cantonal POs, which by this ruling remained in charge of prosecuting the same crimes as would the proposed FPOSD. However, through KIIs, the evaluation team learned that these developments and the ruling of the FBiH Supreme Court produced split opinions among professionals. Namely, as noted in the OSCE report16 and confirmed by KIIs: “The failure to establish these authorities could have resulted in institutionalized impunity for the above-mentioned crimes since, with the entry into force of the Law, cantonal prosecutors and courts are no longer competent for their prosecution and adjudication.” The JA contract also requested that the JA partner with PO BiH in Year 1. Within the timeframe of JA’s implementation, PO BiH experienced a loss of key staff members. Namely, the Chief Prosecutor was initially suspended and, after prolonged disciplinary proceedings, removed from his position. Similarly, the head of the PO BiH Special Department for Organized, Economic Crime and Corruption was suspended initially, and recently indicted, further diminishing trust in the institution. In addition, the new acting management of PO BiH and top officials of the State Investigation and Protection Agency (SIPA) did not establish a good working relationship. The media covered this situation to further undermine the work of PO BiH. Based on these circumstances, and decisions of other donors to pull out from providing TA to PO BiH, JA was instructed not to work with PO BiH. Relevant KIs confirmed that JA and other donor projects were not providing TA to PO BiH for a substantial part of JA’s implementation. As the evaluation team learned through KIIs, in accordance with this development and subsequent USAID/BiH instructions, JA worked with 19 (including newly formed PO Prijedor in JA PPP POs) out of 20 POs in BiH. In addition to changes in the organizational structure of POs, and the FBiH Supreme Court’s decision on the competence of cantonal POs, in 2017, during JA implementation, the BiH constitutional court17 ruled that key aspects18 of the Criminal Procedure Code of BiH regulating special investigative measures were incompatible with constitutional and human rights and that the BiH parliament must make legislative changes. The constitutional deficiencies included the lack of specificity in the formulation of these measures and the excessive scope of their application. While the Constitutional Court of BiH issued a decision requiring harmonization of these measures, and the amendment procedure was initiated at the BiH level, no action has been taken at the entities and Brčko District levels. Given this, after amendments are adopted at the BiH level, they will not be applied by courts in FBiH, RS, and BD, which will deem them formally constitutional within their jurisdiction. The current uncertainty was noted in KIIs with POs, particularly as special investigative measures (such as the interception of communications and computers, the use of undercover agents, covert surveillance, and simulated bribery) are essential to investigate corruption and organized crime cases effectively. The absence of coordinated action by legislative bodies and the failure to pass adequate and harmonized amendments simultaneously created dilemmas for POs and prosecutors regarding the legal framework in which prosecution should take place. 15 https://www.pravosudje.ba/vstv/faces/vijesti.jsp?id=54616 16 Organization for Security and Cooperation in Europe (OSCE), Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases Project Report (February 2018) (OSCE 2018) 17 See the Constitutional Court Decision No. U 5/16, June 1, 2017. Available at http://www.ustavnisud.ba/dokumenti/_bs/U-5-16- 1076865.pdf Ibidem, paras. 77–79, 82–83 (accessed on August 10, 2018). 18 Specifically, the decision declared unconstitutional a number of provisions of, namely: Article 84(2)-(5), “Right of the Witness to Refuse to Respond,” Article 117(1d); “Criminal Offenses as to Which Undercover Investigative, Measures May Be Ordered,” Article 118(3); “Jurisdiction to Order the Measures and the Duration of the Measures.” Other provisions that were declared unconstitutional concerned the timeframe for completion of the investigation and the timeframe for filing the indictment with the preliminary hearing judge. 12 The Brief Assessment of the BiH Justice Sector identified and presented in detail additional contextual factors that were present during JA’s implementation including, but not limited to: a lack of adequate budgets and financing for POs’ work of POs, including availability of budgets for already approved new PO positions, and delays in payments for expert witnesses and other goods and services delivered to POs; filings of numerous poor-quality criminal reports for corruption cases; a discrepancy in expected standards for validity of evidence between POs and courts; lack of prioritization and expeditious processing of corruption cases by courts; negative trends in the harmonization of legislation among entities/BD and BiH; and differently defined subject matter and territorial jurisdictions of POs, which causes conflicts of jurisdiction (that lack established mechanisms for settlement). Finally, political parties and their representatives in the executive and legislative branches of government do not provide uncontested support for the work of POs, particularly with respect to their processing of corruption cases. This absence of political support also has manifested in the low levels of support that other governmental agencies show for POs. Law enforcement agencies, the Supreme Audit Institutions (SAIs), the Agency for the Prevention of the Fight Against Corruption (APIK), and others have limited engagement and lack proactivity in assisting POs in identifying corruption crimes and collecting evidence. Most of the KIs that the team interviewed confirmed this. KIs highlighted various issues, including political influence over the work of police, lack of commitment, and lack of number and capacity of police inspectors working on corruption cases, in interactions between POs and law enforcement agencies. In particular, POs feel their work is limited by the quantity and quality of work of law enforcement agencies. According to 16 KIs (10 beneficiaries and 6 stakeholders), enforcement (police) agencies have limited capacity to investigate economic and organized crime and corruption cases. Many POs (8) stated that police reports about organized crime, corruption, and economic crime cases are of an unsatisfactory quality and are not well substantiated with evidence. According to two POs, this is one of the reasons for the large number of decisions by POs to stop investigations that have been initiated. The JA contract correctly identified PO BiH and the Special Departments of PO RS and PO FBiH as lead POs in processing corruption cases, especially the most complex cases. As previously noted, during the JA’s implementation, these three POs experienced developments that were not anticipated in the JA contract and were outside of the JA’s control. Based on these findings, the evaluation team concludes that the JA operated in a highly challenging environment that was not conducive to fighting corruption. Given this context, the JA’s implementation required flexibility and adaptation. The evaluation team conducted a literature review of corruption-related documents and reports produced by international and domestic organizations, which, in combination with the evaluation team’s observed findings and conclusions, identified a number of environmental challenges that constrained the JA’s operations. While the team presents an extended summary of these obstacles in Annex VIII, we note a few key findings here: “None of the cases that are indicted were adjudicated in the last three and a half years, until a few days ago. Our courts, especially all over Sarajevo, do not keep up in terms of capacities. They have nine judges who are adjudicating cases … then they have 27 prosecutors just in this department. Simply put, there are people who are not up to the task.” -PO “Also, there is a huge discord inside, I am speaking of Ministry of Internal Affairs, there is no appointment, there is no Commissary. The appointment decision was overruled, political interference, etc. Professional depoliticized police was not created and that is goal that we should pursue.” “However, what is alarmingly troublesome is huge number of negative, so￾called negative decision that order stopping and not conducting investigations. In one hand is due to the shortcomings of indictments, and sometimes in collecting of the evidence.” -POs 13  Political parties have a major influence on all branches of government and the general public considers them inherently corrupt.  The legislature does not act in a harmonized or consistent way in response to corrupt acts by public officials and fails to enforce existing laws and regulations.  The complexity of the government and the overlapping jurisdictions and legislature lead to persistent corrupt practices in government institutions.  Governmental oversight agencies do not have the necessary authority or scope to perform their duties in an appropriate manner.  The pertinent judicial and prosecutorial bodies fail to adequately cooperate and coordinate efforts to prevent, detect, and prosecute high-level corruption, which often results in lengthy high-level corruption cases and unsatisfactory results.  Citizens have a low level of trust in the judiciary, as well as few incentives to get involved in anti￾corruption efforts or to report corruption. The European Union’s annual progress reports for BiH identify corruption as one of the country’s most pressing issues, and list a number of improvements needed in the fight against corruption 19. 19 “Significantly improve the track records in the areas of repression and prevention of corruption, including by imposing effective and deterrent sanctions.” European Commission – Staff working document Bosnia and Herzegovina 2018 Report, available at https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/20180417-bosnia-and-herzegovina-report.pdf (accessed on August 15, 2018). 14 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Evaluation Question 1 (EQ1) To what extent has the Activity design been appropriate in terms of chosen activities and beneficiary needs? Do beneficiaries and stakeholders vet the chosen activities? How was the Activity implemented in terms of sequencing, timing, resource allocation, flexibility, and cooperation with beneficiaries and stakeholders? How do beneficiaries and stakeholders perceive and value the JA’s work? (EQ1) FINDINGS Finding 1: A majority of beneficiary and stakeholder KIs (14), when discussing JA activities that they participated in, stated that the JA’s design20 was appropriate in terms of implementing activities that meet beneficiaries’ needs. Out of 15 interviewed beneficiary POs, 10 recalled that the JA conducted a capacity assessment of their institutions and assessed POs’ needs before implementing any intervention. Out of 15 interviewed beneficiary POs, five explicitly stated that the JA’s design reflected the needs of the justice sector and beneficiaries in prosecution and processing of corruption and economic crime (CEC) cases The interviewed POs recalled receiving direct technical assistance from financial and economic experts to assist with particular cases, specialized trainings for prosecutors, and roundtables related to improving management practices and public relations. KIs perceived that the JA’s Diagnostic Analysis of the Integrity of the Judiciary Sector in BiH and the Possible Risks of Corruption or Unethical Conduct in the Judiciary System (DA)21 and consequent studies (i.e., Analysis of System of Use of Expert Witnesses in Cases of Corruption, Organized and Economic Crimes, Analysis of Court Verdicts, Analysis of Asset Forfeiture) provided high quality assessments of justice sector issues. The JA selected interventions to implement based on the JA’s previous diagnostic and analytical work. A majority of beneficiaries (13) and some stakeholders (5) know and understand the contents of at least some of the studies and analyses that the JA has produced, and stated that they find them useful (1 interviewed PO disagreed). According to 12 beneficiary KIs, some of the most useful types of assistance provided to beneficiaries include technical assistance in developing Integrity Plans, and technical assistance in developing the Guidelines for Preventing Conflict of Interest including Asset Declaration Forms. Beneficiary and stakeholder KIs (7) identified the third key feature of the Activity implementation, consensus building forums, as important. In terms of material assistance, out of 15 interviewed beneficiary POs, seven confirmed that the JA helped them solve some of their problems related to working conditions and resources. During the field visits, the evaluation team confirmed that POs received assistance with IT, technical equipment, and furniture. Finding 2: The JA’s design anticipated that the JA would work closely with a related “Strengthening Prosecutor Offices Capacities” project (Swiss/Norwegian project), which is providing ongoing assistance to the prosecutorial system in varying capacities.22 The evaluation team found that JA worked jointly with the Swiss/Norwegian project on several activities. Those activities were mainly of prolonged duration and focused on PO operations, including strategic management, annual reporting, public relations, prosecutors’ appraisal processes and specialized trainings.23 According to KIIs with donors, the two projects 20 The JA’s contract specifies the following key features of the JA: Prosecutor Partnership Program (PPP), Diagnostic Assessments and Consensus Building Forums. In addition, the Activity design included a special fund for providing rapid response to unanticipated events and urgent requests. 21 During an interview with the implementing partner, their representatives confirmed that the Diagnostic study was used for design of the consequent JA’s interventions (assets forfeiture, analysis of verdicts for corruption cases, audit reports, improvement of POs web pages, etc.). 22 Contract No.: AID-168-TO-14-0001; September 22, 2014. p.7 23 In implementation of POs strategic plans and results presentation to Collegium of Chief Prosecutors the Swiss/Norway project and the JA split task. Swiss/Norway project was in charge of POs management related to administrative data indicators. The JA was in charge of PR indicators. Regarding technical assistance in organizing and delivering trainings for three groups of prosecutors 15 coordinated well and complemented each other’s activities. Out of 15 interviewed beneficiary POs, eight expressed satisfaction with coordination between the JA and Swiss/Norwegian project finding that both projects were flexible and cooperative. The JA and Swiss/Norwegian project avoided conducting overlapping interventions by communicating frequently during the planning processes of both activities. In some instances, beneficiaries were unable to distinguish between JA and Swiss/Norwegian activities due to the complementarity of these efforts, but a donor representative noted the JA was not required to exert significant effort to distinguish its’ individual efforts in joint activities. The evaluation team found that the JA contract does not mention specific cooperation with the Organization for Security and Cooperation in Europe (OSCE) on analyzing corruption cases and verdicts. Through information obtained from KIIs on other donor activities, the evaluation team learned that the OSCE “Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases” (ARC) project (financed by the United States Department of State, Bureau of International Narcotics and Law Enforcement Affairs (INL)) is monitoring and analyzing trials and judgments in corruption cases. The evaluation team found that the JA is working on a similar task, while KIIs and the document review revealed that there is little cooperation between the JA and OSCE ARC project in these activities. Finding 3: The JA’s design and the contract did not specify quantifiable expected results or milestones of implementation, which enabled the JA to be implemented flexibly. The JA Monitoring, Evaluation, and Learning (MEL) Plan, however, specifies quantifiable expected results. The JA MEL Plan had four amendments/adjustments, including revision of targets over the implementation period. Finding 4: Based on documents reviewed, the JA regularly informed the HJPC’s Standing Committees on planned activities, including PPP/POAP implementation, delivered support related to strengthening POs’ management capacities, asset forfeiture related activities, use of expert witnesses in criminal proceedings, use of SAI reports as the basis to initiate criminal proceedings, analysis of verdicts in corruption cases, and strengthening POs’ PR capacities. Furthermore, the JA informed HJPC and its respective Standing Committees of topics, including those related to disciplinary proceedings, career advancement, performance appraisal, and the JA recommendation for changes of the HJPC Law. As a result of these activities, the HJPC formally vetted and adopted the following documents:24 the Guidelines for Development and Implementation of the Integrity Plan in the Judicial Institutions of Bosnia and Herzegovina, the Guidelines for Prevention of Conflict of Interest in the Judiciary, and the Guidelines for Determining Disciplinary Measures. The evaluation team found that the HJPC’s formal adoption of these documents indicates a significant endorsement given the judicial sector’s strict regulation. At the level of individual POs, all Prosecutor Partnership Programs (PPPs)25 have been signed between Chief Prosecutors and the JA CoP. Finding 5: Interviewed beneficiary POs (7) and other stakeholders (12) that have collaborated with the JA stated in interviews that the JA’s management was professional and flexible. POs and other stakeholder KIs positively noted the JA’s ability to adapt and tailor assistance to the POs’ diverse demands, needs, and (the group on corruption, the group on economic and organized crime, and the group on cybercrime), the Swiss/Norway project alone delivered trainings to the group on cybercrime, while to the other two groups, trainings were delivered jointly by Swiss/Norway project and the JA in accordance to mutually agreed training plan and curricula. 24 Guidelines for development and implementation of the Integrity plan in the judicial institutions of Bosnia and Herzegovina, adopted by HJPC on July 8, 2016 decision no. 08-02-3-2080-1/2016 http://usaidjp.ba/assets/files/publication/1470212003- smjernice-za-izradu-i-provodenje-plana-integriteta.pdf Guidelines for prevention of conflict of interest in judiciary, adopted by HJPC on July 8, 2016 HJPC decision no. 08-02-3-2078/2016 https://pravosudje.ba/vstv/faces/docservlet?p_id_doc=35081 , http://apik.ba/Print.aspx?id=905 Guidelines for Determining Disciplinary Measures, adopted on July 8, 2016 https://usaidjp.ba/assets/files/publication/1470216400- high-judicial-and-prosecutorial-councils-guidelines-for-determining-disciplinary-measures.pdf, HJPC/ODC 2016 annual report quotes it https://www.pravosudje.ba/vstv/faces/docservlet?p_id_doc=40366 , p.24 25 PPPs are formal Memorandums of Understanding (MoUs) between the JA and POs where all technical assistance to be provided by the JA to POs is specified. 16 operational dynamics. One PO mentioned that the JA always consulted them before implementing any interventions and was always at their disposal. Finding 6: PPPs were signed26 with 19 POs.27 The implementation of Prosecutor Office Assistance Plans (POAP) was completed in three groups of PPP POs, while implementation in a fourth group is in progress. According to the contract, all 19 PPPs/POAPs were to be completed within the first three years of the JA’s implementation. However, the sequencing of assistance to specific PO groups was changed, and the JA started to work with the last group of POs in Year 4. These adjustments to the contract were approved in the annual Work Plans of the Activity, as well as by HJPC SC for Efficiency of POs and the chief prosecutors, both in Federation of Bosnia and Herzegovina (FBiH) and Republika Srpska (RS). Finding 7: All beneficiaries, including representatives of the HJPC’s Standing Committees, stated that they valued the JA’s assistance in activities they directly participated in. The JA’s beneficiaries and stakeholders identified the following as the most useful interventions: provision of experts in economics and forensic accounting; specialized trainings for prosecutors and trainings for PR personnel; material assistance with IT, other equipment, and furniture; technical assistance in developing Integrity Plans; and technical assistance in developing the Guidelines for Preventing Conflict of Interest. Furthermore, in the HJPC letter to all courts and POs in BiH informing them of the JA’s intervention, the HJPC emphasizes that “HJPC recognizes importance of USAID/BiH’s the JA as an initiative of extreme importance for strengthening judicial capacity in BiH.” 28 Finally, as mentioned in finding 4, HJPC expressed appreciation for and endorsed the JA’s work by formally adopting bylaws. The particular value of these guidelines (see finding 4) is that they are becoming a part of everyday operations of the BiH justice system, for example, guidelines for disciplinary sanctions are used and quoted in HJPC’s Disciplinary Committee decisions. 29 The 2017 HJPC Annual Report30 recognizes the JA’s technical assistance in a number of HJPC reported activities. Finally, as stated in finding 36, the value of JA’s work is recognized in the EC peer review mission recommendations. (EQ1) CONCLUSION The JA’s interventions have been appropriate and met beneficiaries’ needs, to the extent possible given available resources, in institutional strengthening for prosecuting and processing CEC cases. The JA selected activities to implement based on the JA’s prior high-quality diagnostic and analytical work and extensive consultations with beneficiaries. Interventions that establish consensus building forums between POs and government agencies are an important precondition for achieving the JA’s expected results. The evaluation team concludes that the interventions that have been most useful for the JA’s beneficiaries include providing experts in economics and forensic accounting, specialized trainings for prosecutors and PR personnel, material assistance with IT, other equipment, and furniture, technical assistance in developing Integrity Plans, and technical assistance in developing the Guidelines for Preventing Conflict of Interest including Asset Declaration Forms. The quality of the JA’s diagnostic and analytical work is evidenced by HJPC’s formal endorsement of these guidelines and that they are incorporated into the judicial recommendations of the EC peer reviews. Beneficiaries vetted all phases of JA’s implementation, design, development, implementation and completion. The HJPC acknowledged the quality and importance of the JA’s interventions, namely TA, in 26 PPP is a formal agreement between the JA and PO (MoU) specifying the scope and methods of the JA’s assistance, needs assessment analysis of the specific PO, development of the POAP, timeline and coordination of activities and duration of PPP itself. 27 See Section “Context in which the JA operates” 28 HJPC Letter No.: 01-50-237-2/2015; March 5, 2015. 29 Guidelines applied in the Decision of the HJPC first instance disciplinary commission for judges, no 04-07-6-2340-4/2018, date September 5, 2018. Available at https://vstv.pravosudje.ba/vstv/faces/docservlet?p_id_doc=48237 30 HJPC 2017 Annual Report. Available at: https://www.pravosudje.ba/vstv/faces/pdfservlet?p_id_doc=46562 17 a number of its official documents and letters. PPPs have been established with 19 POs as formally requested by the JA contract. During implementation, the JA’s staff effectively and appropriately managed the JA. The JA employed a flexible approach during the project’s implementation and accommodated the emerging needs of beneficiaries and new developments in the justice sector. The absence of quantifiable results in the JA’s contract enabled the implementation team to set moveable targets throughout implementation, adapt to beneficiaries’ needs, and adjust to prevent overlap with other similar activities. However, this approach limited the evaluation team’s ability to assess results and conclude whether the JA successfully achieved its objectives. The JA established and maintained prolonged cooperation and achieved synergies with the Swiss/Norwegian project, which should be seen as a very positive example of donor coordination efforts. The evaluation team concludes that USAID/BiH and INL should further examine similarities and differences in their efforts, but in particular, they should identify areas for joint work between the JA and the OSCE ARC project on analyzing corruption cases and verdicts and explore possibilities for joint presentation of these results. 18 Evaluation Question 2 (EQ2) To what extent has the JA achieved expected results under both components as outlined by the contract, and what are the prospects of meeting life-of-activity targets specified by the monitoring and evaluation plan? More specifically, was the individually-tailored assistance to each PO appropriate, and what results were achieved to improve the overall quality of the work of prosecutors in BiH? Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient, and can these be considered successes? As noted previously, the JA initially had three components. However, a JA contract modification removed the initial component 2 after the first year of implementation. The contract modification requested the JA to “Refocus the resources of USAID/BiH Justice Activity on the work with prosecutors and justice sector institutions to better combat corruption (Components 1 and 3 of the Justice Project) and suspend the work related to the improvement of efficiency in the enforcement of judgments (Component 2).31” Activities related to the enforcement of judgements (hereinafter: the former Component 2) were in the early stage of development and implementation when the former Component 2 was terminated by USAID/BiH in accordance with the contract modification. The following two components currently comprise the JA’s contract: Component 1 (C1): Strengthening the professional status and performance of prosecutors; and, Component 2 (C2): Strengthening justice sector institutions to uphold public integrity and combat corruption. The JA component which was removed and was not explicitly examined by this evaluation was: The former Component 2 (Frm-C2): Improving efficiency in the enforcement of judgments. JA MEL Plan The JA developed its Monitoring and Evaluation (MEL) Plan to measure the results of each Activity component and establish baseline measurements to assess the effects of proposed interventions.32 In accordance with changes in the JA contract and changes in the context in which the JA operates, the JA engaged in four modifications of its MEL plan. The evaluation team reviewed JA MEL Plan Modification 4, and found that the structure of the MEL Plan follows on major components and outputs/outcomes as set in the JA contract. The assessment team found that most indicators in C1 (namely, for output/outcome 1.1 Strengthened organizational leadership, planning, and performance in prosecutor offices; 1.2 Prosecutors perform functions more efficiently through balanced allocation of resources and 1.3 Prosecutors uphold public trust and integrity through prosecution of corruption and other serious crime) are sourced from HJPC administrative data on processing corruption and economic crime cases by POs, but also their corresponding courts in BiH. These indicators cover all major steps in processing these cases, including indicators on processing criminal reports, conducting investigations, filing indictments, and obtaining convictions. The team presents the indicator analysis, their actuals and targets in Annex I, and presents summary findings as follows: The evaluation team examined actual values and trends throughout the JA’s implementation, and found that both individual values for individual POs and groups of PPP POs, as well as aggregated data for all POs in BiH were volatile and demonstrated changing trends from year to year. For example, the trend in the 31 Contract AID-168-TO-14-000001, Modification No.1, Date 09/29/2015, p.3 32 Contract No.: AID-168-TO-14-00001, p.23 19 number of indictments for corruption cases, Indicator 1.1.2, was highly volatile between 2014 and 2017. While there was a large increase in indictments in 2015 (61 additional indictments than in 2014), there was a sizeable drop in 2016, when 72 fewer indictments were filed. Exhibit 1 provides the full name, definition, actuals, and targets for indicator 1.1.2 as stated in the JA MEL plan. In addition, this exhibit illustrates trends in the actuals and targets for this indicator from 2014 through 2019, and actuals from 2014 through 2017 based on HJPC administrative data.33 Exhibit 1: Number of indictments for corruption related crime in POs that implement PPPs (the JA MEL Indicator 1.1.2)                     Source: The JA MEL Plan (Modification 4) and HJPC administrative data Note: Actuals are presented in blue (2014-2016) and targets are presented in red (2017-2019) 33 Table is copied directly from JA MEL plan (modification no. 4). It should be noted that summary numbers in the last row of the table are incorrect. The evaluation team checked the numbers presented in the last row against raw datasets provided by HJPC, and found that consistently in all 10 JA’s MEL indicators sourced from HJPC administrative data, summary data present aggregate values for all POs in BiH (including BiH PO which is not beneficiary of the JA Activity). The evaluation team believes that this inconsistency in the JA MEL Plan is misspelled or that whole the JA MEL Plan and the last raw need thorough revision. Graphs of the data in JA MEL plan (as stated in the upper table) are provided on the left-hand side. The right-hand side graph contains additional data on actuals for 2017 sourced from HJPC administrative data which are calculated by the evaluation team. The notes made here equally apply for presentation of all indicators in JA Outcomes 1.1 – 1.3. 20 While POs in BiH generally improved their performance in processing corruption and economic crime cases during the first two years of the JA’s implementation, most indicators that track POs’ performances in processing criminal reports, conducting investigations, filing indictments, and obtaining convictions in corruption and economic crime cases in Year 3 experienced negative changes. Consequently, results of most of these indicators in Year 3 compared to their baseline values do not show observable improvements. For example, there is a very small increase in the number of convictions in corruption￾related cases between 2014 and 2017 (13 convictions or 6.1 percent). This trend is driven by the increase in convictions between 2014 and 2016. However, there were 10 fewer convictions in 2017 than in the previous year. Exhibit 2 provides the full name, definition, actual numbers, and targets for indicator 1.1.1 - Number of convictions for corruption-related crime in POs that implement PPPs as stated in the JA MEL plan. The Exhibit illustrates trends in actual numbers and targets for this indicator from 2014 through 2019 as stated in the JA MEL Plan, and actuals from 2014 through 2017 based on HJPC administrative data.34 Exhibit 2: Number of convictions for corruption-related crime in POs that implement PPP (the JA MEL Indicator 1.1.1) Source: The JA MEL Plan (Modification 4) and HJPC administrative data Note: Actuals are presented in blue (2014-2016) and targets are presented in red (2017-2019) 34 See footnote 33, supra. 21 Due to the volatility in trends of these indicators (based on the team’s review of all indicators coming from the administrative data as presented in Annex I), the evaluation team cannot estimate if the life-of￾activity targets for these indicators, as set in the MEL Plan, will be achieved. In addition to reviewing the JA MEL indicators sourced from the administrative data, the evaluation team conducted a spatial analysis of HJPC administrative data on processing corruption and economic crime cases from 2014 through 2017. Our analysis summarized POs’ processing these cases, from initiating the case after a criminal report was filed, through investigation, filing an indictment, and obtaining a conviction (see Annex II). The evaluation team found that from 2014 through 2017, POs did not make noticeable advances in processing corruption and economic crime cases. The number of criminal reports that police and individuals filed, rather than POs, varied from 2014 through 2017. From 2015 to 2017, the inflow of the cases increased, but it decreased in 2017. Other key variables experienced few changes, however. For example, Exhibit 3 shows that there was no substantial change in the patterns of rejecting criminal reports or terminating investigations for corruption cases from 2014 to 2017. For other variables, such as the number of indictments filed and number of convictions, when changes occurred, the direction varied from year to year. Our comparative analysis indicates that, based on all HJPC key variables (number of newly received criminal reports, number of investigations opened, number of indictments filled and number of conviction judgments), only PO Zenica showed steady improvement from 2014 through 2017. PO Tuzla had an exceptionally productive year in 2015. PO Bijeljina and PO East Sarajevo also had their most successful years in 2015. PO Bihac had its most successful year in 2016. In general, by either looking at individual JA MEL indicators or aggregate HJPC administrative data for the whole judiciary or individual POs, the evaluation team found that these data, which should indicate the overall progress of the BiH judiciary in processing corruption and economic crime cases (see Exhibit 4), showed not only the results of the JA intervention, but also the results of other donors’ interventions (i.e., Swiss/Norwegian project, EU Twinning Light projects) and domestic stakeholders’ efforts, or lack thereof. Because of the complexity and interconnectedness of these efforts, the evaluation team could not measure the influence of the individual interventions’ contributions to the overall values. The evaluation team further found, however, that introducing these data in the JA MEL plan and monitoring them were positive developments. As a result, USAID may review results of the entire judiciary’s efforts in processing corruption and economic crime cases. These indicators are a valuable source for decision makers regarding future programs. In addition to the indicators mentioned above (1.1 – 1.3), which are sourced from HJPC administrative data, the JA MEL Plan includes indicators (1.4.3, 2.4.2 and 2.5.1) that are sourced from surveys of the public, or surveys of judges and prosecutors conducted by the JA. When the evaluation team examined these indicators over the period of the JA’s implementation, the team found that the trends were volatile and unpredictable (See Annex I). The evaluation team found that out of these three indicators, one indicator’s actual value (indicator 1.4.3 - Percentage of public that agree that BiH POs adequately inform public about their work ) remained around its baseline level, the second indicator’s actual value (indicator 2.4.2 - Percentage of judges and prosecutors who strongly agree with the statement that the disciplinary process is fair and impartial) fell below its baseline value, while the third indicator’s actual value (indicator 2.5.1 - Percentage of public who agree with the statement that the judiciary is not vulnerable to corruption) improved over its baseline and even surpassed its life-of-activity target (but still showed poor public perception). 22 As with indicators sourced from HJPC administrative data, indicators based on the survey data are subject to the influence of the JA, local stakeholders’ efforts, and other donors’ interventions. The evaluation team could not isolate contributions of each of these efforts to the indicators. Thus, for the same reasons as in case of indicators sourced from HJPC data, the evaluation team cannot estimate if the life-of-activity targets for these indicators, as set in the MEL Plan, will be achieved. Unlike the indicators mentioned above (1.1 – 1.3 sourced in the HJPC administrative data and the three mentioned indicators sourced from survey data), the following indicators in the JA MEL Plan can be gauged against set outputs/outcomes in the JA contract, for example: 1.4.1 Number of POs that regularly disseminate information to the public through their websites, including appropriate information on corruption and economic cases in POs that implement PPPs; 1.4.2 Number of press releases (or other types of communication channels) issued to the public, including information on investigation and prosecution for corruption, economic crime cases in POs that implement PPPs; 2.2.1. Number of collaborative mechanisms established and/or improved among justice sector institutions and other parties involved in anti-corruption efforts; 2.3.1. Number of improvements implemented in ODC performance in disciplinary matters; 2.4.1. Judicial Discipline Bench book and Ethics Handbook that will enable ODC staff and disciplinary committees to conduct proceedings more expeditiously and with greater consistency (fairness) developed; 2.5.2 Number of codes of conduct revised for improved standards in judicial conduct; 2.5.3 Number of Courts and POs that adopted Integrity Plans; 2.6 Number of judges and prosecutors earned the certificate of specialized training in the investigation, prosecution, and adjudication of corruption and financial crime cases; DR 1.3-1 Number of judges and judicial personnel trained with USG assistance; and, DR 2.4-1 Number of government officials receiving USG-supported anti-corruption trainings. These indicators cover efforts that the JA directly controls, and results indicate that there has been consistent improvement over the JA implementation period. The JA was on track to achieve the activity’s expected results in 2018 or to reach their 2019 (the life-of-activity) results. In some cases the JA has already met the 2019 (the life-of-activity) targets. Annex I provides details on these indicators as well. 23 Exhibit 3: Aggregate results in processing corruption cases by all prosecutor offices in Bosnia and Herzegovina, 2014-2017 Source: HJPC administrative data and MEASURE-BIH GIS online presentation of the HJPC administrative data 24 Exhibit 4: Aggregate results in processing both corruption and economic crime cases by all POs in BiH in 2014 – 2017 Source: HJPC administrative data and MEASURE-BIH GIS online presentation of the HJPC administrative data 25 (EQ2) FINDINGS Outcome/Output 1.1: Strengthened organizational leadership, planning, and performance in prosecutor offices Finding 8: The evaluation team determined by reviewing the JA’s documents that the JA delivered assistance in strengthening organizational leadership, planning, allocation of resources, performance in prosecutor offices, and upholding public trust and integrity through prosecution of corruption and other serious crimes through tailored activities specified in PPPs/POAPs among individual POs and through other mechanisms (i.e., development of strategic plans for all entity POs and monitoring of annual reports and quantifiable indicators through the Collegium of Chief Prosecutors, including indicators related to PR as per methodologies set in the POs’ PR MEL plan). The JA developed POAPs after conducting the capacity assessment of each beneficiary PO, and tailored it to the individual needs of each. As previously stated, of 15 interviewed POs, 10 were able to recall that within PPP, the JA had conducted a capacity assessment of their institutions and assessed POs needs before implementing any intervention. Finding 9: The evaluation team found by reviewing the JA documentation that POAPs were structured around three major types of assistance: improvements in work on corruption, economic and organized crime cases; improvements in management practices and improvements in transparency of POs and public relations. In terms of activities related to the improvement of organizational leadership, planning, and performance in POs, the JA created a general list of possible TA to be offered to an individual PO, which included the following: assistance to POs in identifying evidence and structure of orders for expert witness testimony in financial investigations; providing economic and forensic accounting expertise as part of on-site training for prosecutors; assistance in organizing and allocating resources for processing corruption, and economic and organized crime cases; assistance to POs in setting priorities and establishing and leading joint investigative teams; assistance in establishing case weighting, developing methodologies to assess or estimate damages in corruption, and economic and organized crime cases; identifying best practices and disseminating them to other POs; and providing targeted specialized trainings as per identified needs. In relation to improvements in management practices of POs, the JA’s assistance was related mainly to assistance in structuring and developing POs’ budget proposals and their presentation to the executive and legislative branches of government; training in budget development related topics; assistance in monitoring POs’ performance in processing cases with application of relevant standards and statistical indicators; assistance in analyzing POs’ expenses related to expert witness testimonies; and assistance in determining adequate incentives to encourage prosecutors to work on the most complex cases. In relation to improvements in quality and transparency of POs’ PR activities, the JA’s offered technical assistance to develop POs’ PR strategic documents; develop standard indicators for tracking public relations of POs and corresponding registers; upgrades, content management and maintenance of POs’ web sites; introductory, advanced, and specialized trainings for PR officers in POs; strengthening the status of PR officers, including establishing an association of PR officers in POs; and improvements in public access to POs’ information and POs’ work performance. Based on the capacity assessment of a partner PO and expressed needs by a PO, the final version of POAP and TA that the JA delivered was tailored to the individual POs. 26 Finding 10: According to 6 interviewed POs, the JA’s interventions resulted in some positive changes in the work of POs. Some beneficiaries (4) believed that material assistance enabled the POs and ODC to work more efficiently. Some KIs mentioned that for corruption and economic crime cases, the JA’s interventions resulted in a new reporting system through which POs report their achievement on a quarterly basis. Finding 11: The provision of experts was the most frequently and positively mentioned type of assistance by the beneficiary POs (9); six POs stated that they found this kind of help to be the most useful. The JA, by providing intensive expert assistance to specific POs, helped POs realize and understand the importance of financial and economic expertise. However, only three interviewed POs confirmed that this assistance actually improved prosecutors’ abilities to conduct complex investigations. The majority of POs (9) expressed a need for this type of assistance, stating that all POs should have financial or economic experts on a permanent basis. This need was particularly emphasized by all large POs (1/3 of all POs). Some of the interviewed POs would like to see international organizations continue to provide financial support for technical assistance in the form of financial and economic experts. Finding 12: According to nine interviewed beneficiary POs, the number of convictions was low and sentencing policy is mild. One PO stated that, regardless of assistance from the JA or other projects, the number of indictments was low compared to the number of cases investigated. POs mentioned that these issues result from the police’s poor capacity to investigate crimes, especially CEC cases, the lack of experts and limited economic expertise within POs, and slow courts. Outcome/output 1.2: Prosecutors perform functions more efficiently through balanced allocation of resources Finding 13: The JA’s assistance related to strategic planning helped POs to substantially increase their technical capacity to manage CEC cases. Most of the beneficiary POs (11) that the evaluation team “We still have a small number of indictments in comparison with the total number of closed cases. One issue is with the lack of the capacity among police and we have to educate them. Another problem is with prosecutor offices where the cases are … and there are some indications that it will happen through the USAID project, to engage these experts, economic experts, so that they can work through the project in prosecutor offices, to help prosecutors, to guide investigations in the right directions. … Also, there are problems with our decisions not to commence with investigations, or to stop investigations, in comparison with the number of indictments” “If you do the analysis, in comparison with that number of initiated indictments there is a very small number of conviction verdicts. Even then, criminal code and punishment is very lenient on behalf of the prosecutor offices towards those conviction verdicts”. -PO “Well, when we consider the beginning of the work of this prosecutor office, comparing the time when the prosecutor had only himself, with today, when the office has an economic expert, when it has an expert associate for assistance, with the aid of dactylography, with its own computer station, is not comparable. All of that could not be done solely by this prosecutor office. We got assistance from USAID and from others that aided us and helped us.” -PO “The cooperation was on certain days more intense, depending on the case needs, because of the big cases. For example, for (names of some specific cases) etc. we needed expert help, economic experts, financial experts whose assistance was important to prosecutors in relation to expert witness segment and in many other segments as well. Usually situation would be presented to them, they never had any contact with real cases, trough hypothetical cases we would get answers from them.” “From the very start we got first economic expert, seasoned professional who worked in police agencies, so he knew right away what we needed, and where we were deficient considering what we do.” “Primarily it was about investigators and financial forensic. So, those were two persons that came with us on several occasions when we were doing investigations. They worked for some time and gave contribution in line with their expertise. They are from BiH Prosecutor Office, and it was very successful. It was really successful. It gave some clarity to prosecutors on how investigations are supposed to be conducted.” -POs 27 interviewed stated that implementation of their strategic plans was in progress and regularly monitored. An HJPC representative also confirmed these statements. POs monitored indicators related to their strategic plans; they analyzed the results and presented them quarterly in Collegiums of Prosecutors. According to beneficiary KIs, the JA’s interventions and assistance related to strategic planning were important and useful. For example, as a result of the JA, POs adopted strategic planning and annual work plans as standard processes. The JA’s assistance helped POs understand the importance of strategic planning and abandon day-to-day planning practices. Strategic planning was useful especially in reducing the backlog of old cases, which was also the focus of the Swiss/Norwegian project. Finding 14: According to four beneficiaries and two stakeholders the JA contributed to improvements in the operations of POs by providing them with tools to better organize their work and manage cases more effectively. The JA provided support to POs to strengthen institutional links between strategic plans and budgets, as well as improve operational performance and management by enhancing individual PO’s capacities in strategic management, organizational performance, and financial management. However, the POs that the evaluation team interviewed agreed that discrepancies remain between POs’ plans and actual implementation. According to four beneficiary KIs, POs remain understaffed, underfinanced, and ill￾equipped. POs lack support staff, professional PR officers, and economic experts. POs that have managed to get approval and a budget to hire economic experts on a full-time basis stated that the work of economic experts is extremely useful, saves a lot of time for a prosecutor, speeds up the process of document review and selection, and improves a prosecutor's understanding of economic issues. However, even the well-resourced POs stated they still need additional expert investigators to improve their efficiency. Also, seven POs emphasized that they still faced problems: high transportation costs and old vehicles, expensive payments for lawyers and high costs for criminal proceedings, and a lack of money to hire new staff. Finding 15: Several beneficiary POs (5) confirmed the usefulness of budgeting trainings provided by the JA. Although POs prepared their budgets and strategic plans in accordance with new guidelines learned at trainings on budget preparation and management, the Ministries of Finance ultimately allocated money to POs based on their own criteria. POs are improving their communications with Ministries of Justice about their needs for resources. However, Ministries of Finance continue to determine POs’ budgets based on their assessment of POs’ needs. Despite this, there are some positive examples of cooperation between POs and the executive branches of government regarding budget preparation, as some budgets are developed jointly through discussions of POs’ needs and available financial resources. Finding 16: Most interviewed beneficiary POs (10) disagreed that the JA improved their effectiveness in prosecuting CEC cases. Stakeholders mentioned some of the following reasons for the lack of improvement: poor financial capacities and human resources in POs; poor quality of work by law enforcement agencies; and lack of political will to process corruption cases. These reasons identified by POs are structural constraints, which are beyond the JA’s scope to remedy. Also, CEC cases are hard to prove, because citizens often are afraid to report them, and when they do, their reports are usually of poor quality. Moreover, the existing quota system does not motivate prosecutors to process CEC cases, and often financial resources for such cases are significantly lower than those at the defense’s disposal. “We work independently [on our PO’s budget], but with the approval of HJPC. I don’t know who or which department is in charge of budgets there, but they always go over it and give their suggestions and guidelines, then that draft of the budget is submitted to government. I remember when we started, our prosecutor office had some 50 prosecutors and personnel. Now we have more than 100 and we could not hire for some 10 years because of a shortage of resources. Now the budget process is starting again. We state and elaborate our needs, HJPC approves it, the Canton approves it, and that is that.” -PO 28 Outcome/output 1.3: Prosecutors uphold public trust and integrity through prosecution of corruption and other serious crime Finding 17: According to the JA 2016 Annual Report, the JA with the Central and Easter European Initiative (CEELI) Institute in Prague provided two trainings (study tours) to 19 BiH prosecutors and chief prosecutors on topics related to the investigation and prosecution of corruption. In addition, the JA organized a multi-year regional exchange of experiences with the Croatian State Prosecutor's Office for the Suppression of Organized Crime and Corruption (USKOK) on cooperation between prosecutors’ offices and enforcement agencies. The evaluation team found through its review of JA training agendas that representatives of USKOK were delivering training from 2015 to 2018. A Chief Prosecutor from a group of POs that process larger quantity of high-profile corruption cases particularly priced the training program provided by CEELI. Finding 18: More than a half of interviewed beneficiary POs (8) confirmed that they formed either permanent or ad-hoc joint teams with police.35 Some POs, such as Banja Luka PO, introduced joint investigation teams with the police in January 2012. This joint team has two inspectors, one lawyer, and one economist who work for one prosecutor. Zenica, Tuzla, and Sarajevo POs also implemented similar systems. In Sarajevo PO, police inspectors are not assigned to a specific case but rather to a specific prosecutor. In Sarajevo and Tuzla Cantons joint teams were created on the case-by￾case basis. Zenica-Doboj Canton established permanent joint teams formed from prosecutors and two inspectors from the FBiH Ministry of Interior and four inspectors from the Cantonal Ministry of Interior. However, there is a high turnover rate of inspectors, who often change work places within the Ministry of Interior. Police inspectors stay with POs for two years maximum, and after being trained and developing expertise in financial and economic cases, they are transferred to other positions. This turnover limits the effectiveness of these joint teams. In Tuzla Canton, there are no joint permanent teams. Instead, teams are formed on a case-by-case basis. Finding 19: According to 16 beneficiary and stakeholder KIs, enforcement (police) agencies have low capacity to investigate economic and organized crimes and corruption cases. Many interviewed beneficiary POs (8) stated that police reports about organized crime, corruption, and economic crime cases to POs are unsatisfactory and are not well substantiated with evidence. According to two beneficiary KIs, this is one of the reasons that POs discontinue many investigations. The evaluation team found that the situation has improved somewhat since the POs’ decision on obligatory cooperation between authorized official persons and prosecutors.36 According to some interviewed POs (2), police are highly politicized.37 One PO stated that prosecutors cannot control the quality of police work. There are some exceptions; one PO stated that it managed to impose standards and work ethics on enforcement agencies by treating police as an equal partner and providing trainings to their staff. 35 “A key aspect of PPP assistance will be to build effective prosecutor/police joint investigation teams” Contract No.: AID-168- TO-14-0001; September 22, 2014. p10 (Activity 1.3.). “The contract will provide intensive on-site training and mentorship to frontline prosecutors in all participating offices, directly helping them to work with police on investigating and prosecuting corruption and economic crime.”; Contract No.: AID-168-TO-14-0001; September 22, 2014. p18 (Activity 2.1.). 36 Prosecutors’ Office of the Federation of Bosnia and Herzegovina: Obligatory instruction on acting and cooperation of police officers and prosecutors in evidence collection and conducting investigations, No. A- 445/09 date September 30, 2009. 37 See “Context in which the JA operates” section of this report. “We have problems not only regarding teams, but regarding economic crimes, reshuffling of inspectors. This is due to the promotions and internal work organization. As it is now, we are without an entire economy department in Ministry [MUP]. One or two inspectors stayed, five or six changed their posts.” -PO “Then we said that prosecutors lack forensics know how, know how on how the police work, and inspectors lack know how of the Criminal Procedure Code and when they work together they complement each other, and exchange know how (forming joint teams).” -PO 29 Outcome/output 1.4: Prosecutors provide appropriate and accurate information to citizens to strengthen transparency and responsiveness Finding 20: Some POs (5) confirmed that PR trainings improved their PR capacities. This was especially true among those POs that do not have a PR officer, The JA supported the establishment of the Association of Spokespersons, and worked together with the Swiss/Norwegian project on MEL plan related to PR interventions. JA developed the POs’ PR MEL plan, which was adopted by the HJPC in September 2017. The plan includes regular monitoring and reporting of PR indicators by POs to the Collegium of the Chief Prosecutors. The JA created templates to help POs design information for dissemination to the public and media. These are particularly useful for PR officers in preparing quick and accurate information for dissemination via official PO websites or other communication channels.” In addition, indicators related to PR of POs are a part of POs’ strategic plans and as such they became regular topic of the Collegiums of the Chief Prosecutors. Finding 21: Some interviewed beneficiary POs (5) stated that the transparency of their operations increased due to the JA’s assistance (specialized trainings, strategic action plans, and support to monitoring and reporting systems). POs valued the JA’s assistance as practical and well targeted. However, nongovernmental organizations (2) that the evaluation team interviewed did not agree with the assessment that POs’ transparency and responsiveness had improved. A total of 11 interviewed beneficiaries and stakeholders did not think that the public’s perception of the judiciary has improved as a result of the JA, despite the fact that the JA provided POs with tools to improve their transparency and responsiveness. Finding 22: More than half of interviewed beneficiary POs (9) recognize that the public’s perception of the justice sector is poor and it will be difficult to improve its image. Over the last 20 years, the justice sector has made little progress in its fight against corruption and the public has grown frustrated. According to three KIs, (one PO, two CSOs), such perceptions have arisen because of sensationalized media reports on arrests of high-profile public officials, who are either released soon after arrest, or are not indicted, tried, or convicted. Further, the duration of court proceedings, mild sentencing policies, and public conflicts between judicial authorities and disciplinary investigations of these officials also contribute to negative public perception. Audit Offices also have published audit reports that highlight the negative work of public institutions. These reports create a perception that public institutions operate irregularly and inconsistently and misuse funds, which the justice sector has failed to prosecute. However, many reasons for the public’s poor perception about the justice sector are beyond the POs’ control. Although POs believed that they improved the effectiveness of their communication with media and the public in general through professionalization of PR practices, they felt that citizens’ perceptions are formed by the country’s general situation, which complicates progress toward changing such views. Also, a strong political influence and control over the media does not leave space to improve the public’s perception of the justice sector. Finding 23: Based on data from three waves of the NSCP survey, less than one-fifth of citizens agreed that BiH’s judiciary fights corruption effectively. Exhibit 5 shows no substantial change from 2015 through 2017 in the percentage of citizens that agreed that BiH’s judiciary fought corruption effectively. However, “Come on, it is incredible that all of our big corruption cases are goners, we have spectacular arrests, and at the end we have 90 percent free of charges verdicts, as you already know. That must not happen.” -PO “It is hard to access information on POs’ work. They refuse to give even the information they must provide by the Law. When they provide information, they are vague, minimum.” “Citizens cannot access any data related to work of courts and POs, especially data rated to processing corruption cases. Even when court decides that a PO broke the Law on Free Access to Information, TI never gets the information.” -NGOs 30 almost half of prosecutors and judges believed that the judiciary was combating corruption effectively; although, this percentage declined from 55 percent in 2016 to 48 percent in 2017. Exhibit 5: The judiciary is effective in combating corruption, percentage of respondents agreeing with the statement Source: NSCP surveys (conducted from 2015 through 2017) and the Survey of Judges and Prosecutors in BiH (conducted from 2015 through 2017) Finding 24: As of 2017, only 16 percent of citizens believed that BiH’s judiciary was able to identify and punish public officials who violated the law. Exhibit 6 shows that as of 2017, only a third of judges and prosecutors claimed that public officials who violated the law were generally identified and punished. Exhibit 6: Public officials who violate the law are generally identified and punished, percentage of respondents agreeing with the statement Source: NSCP surveys (conducted from 2015 through 2017) and the Survey of Judges and Prosecutors in BiH (conducted from 2015 through 2017) Finding 25: A lack of professional PR officers is one of several factors that can limit POs’ transparency and responsiveness. According to the HJPC web-portal,38 eight out of 20 POs39 have a professional PR officer. POs that are unable to employ professional PR officers use secretaries to carry out PR functions. 38 Source: www.pravosudje.ba. According the portal, seven POs do not have PR officer and for three POs there is no information. 31 However, these secretaries have many other responsibilities and are not well trained for PR roles. This situation limits POs’ communication with the public through web portals and written public statements. Outcome/output 1.5: Prosecutors’ status improved through performance appraisal, merit￾based career advancement, or incentives to prosecute cases Finding 26: Respondents in 14 beneficiary and 3 stakeholder KIIs stated that the JA played an important role in bringing about the changes introduced in the prosecutor quota system. Interventions to improve prosecutors’ performance and provide incentives to prosecute cases largely focused on the measurement of prosecutors’ work quotas using the Rulebook on Orientation Measurement of Prosecutors’ Work and Quotas. Respondents had differing opinions about the effectiveness of the changes made to the Rulebook. While a few KIs (3) believed that the revised 2016 Rulebook improved the performance of prosecutors, other interviewed beneficiary POs (10) thought the changes could become an obstacle to improving the performance of prosecutors, as the adopted changes favor quantitative over qualitative indicators on prosecutors’ performances. Respondents also had differing opinions about whether the quota systems should exist at all. One PO thought that resolving the lack of accountability alone could address performance issues without the need for a quota system. Only one PO believed that the new Rulebook stimulates prosecutors to enhance their performance. Other PO respondents thought that the quota system does not adequately value prosecutors’ work and that these changes would be insufficient to motivate prosecutors to engage in processing the most complex CEC cases. One PO stated that expecting prosecutors to complete more cases is unrealistic. Prosecutors who deal with the most complex cases, which often require investigations of over a year, fall behind in fulfilling their required quota, and, therefore, opt for simpler cases. According to a few POs (3), the quantity and quality of work of prosecutors is always in conflict, and the quota system that require prosecutors to complete more cases in one year negatively affects the quality of prosecutors’ work. A beneficiary KI also stated that CEC crimes should not have special status and, given the lack of resources, placing a higher weight on CEC cases may affect the backlog of other cases. Finding 27: The JA is working with the HJPC legal department to incorporate their recommendations related to the prosecutor appraisal system and their recommendations related to prosecutors’ appointment and carrier advancement into the new HJPC Law. In its previous work, the JA performed a comparative analysis of appointment and carrier advancement in select EU countries and presented recommendations to the HJPC’s Standing Committee. As most of the recommendations require changes to the HJPC Law, the JA redirected its assistance related to appointment and carrier advancement towards legal departments and the development of corresponding amendments to the HJPC Law. “[W]hen I talked about necessity to reduce quota, quota for prosecutors, because it is something that really can negatively impact work if not properly established. At that time (name) had to increase quota to all prosecutors in prosecutor offices in Bosnia and Herzegovina. And what did happen? Prosecutor’s quota was increased for organized crime, economic crime, corruption and terrorism. Therefore, now they have quota as cantonal and municipal prosecutor offices. In that segment changes must be made because it is not easy to work on the most complex cases under these criteria that we have now. It is difficult to achieve quota in that department.” “Indictments for one corruption case, in an instance that prosecutor with that indictments gets free verdict, if that verdict is free verdict, the quota for that prosecutor is negative. When you work with corruption you have, let’s say, 22 witnesses. Out of that three are key witnesses. Those three witnesses slightly change their testimonies during the trial in comparison to testimonies during the investigation, court rules for free verdict. A sincerely devoted hard work of the prosecutor can lead to negative quota at the end. Therefore, if prosecutor had five indictments on corruption and three of them were free verdicts, and prosecutor worked hard and diligently, at the end prosecutor will be scored using some parameters worse than somebody working on theft. Therefore, my argument is that verdicts for corruption cases should not be negatively scored for prosecutors that really worked hard on that case.” -POs 32 Outcome/output 2.1: Prosecutors investigate and prosecute high-profile corruption and economic crime cases free from political or improper influence In accordance with the JA Annual and Quarterly Reports, as well as the JA MEL plan, the data reflecting outcome 2.1 resulted from activities described within outcomes 1.1, 1.2, 1.3, 2.3, 2.4, 2.5 and 2.6, and do not specify any individual activities within this outcome. Thus, the evaluation team presented findings related to the individual specified outcomes at corresponding places in this report. Finding 28: Based on the 2017 NSCP survey, only 28 percent of citizens believe that judges are able to make decisions without direct or indirect interference from governments, politicians, the international community or other interest groups and individuals (see Exhibit 7). There is a large discrepancy between citizens and professionals (judges and prosecutors) in their perceptions of judges’ ability to make decisions without outside interference (28 percent among citizens vs. 82 percent among professionals). Exhibit 7: Judges are able to make decisions without direct or indirect interference by governments, politicians, the international community or other interest groups and individuals, percentage of respondents agreeing with the statement Source: NSCP surveys (conducted from 2015 through 2017) and the Survey of Judges and Prosecutors in BiH (conducted from 2015 through 2017) Outcome/output 2.2: Increased cooperation among state, entity, and local justice and other sector actors Finding 29: Based on seven beneficiary KIIs, POs see cooperation with state/entity government and justice sector institutions, initiated with JA assistance, as positive but limited, mainly due to the absence of support of state/entity government institutions for the POs’ work.40 As a result of the JA’s assistance, POs and some institutions, such as audit offices, organized meetings for the first time and began communicating about potential cooperation. However, respondents saw this cooperation as primarily donor driven and thought that it would decline with the JA’s withdrawal. Finding 30: The JA’s assistance provided a mechanism to facilitate coordination between POs and the Agency for the Prevention of Corruption and Coordination of the Fight against Corruption (APIK), by establishing a protocol for APIK to report filed corruption to POs (including appointing contact persons in both APIK and POs). Among interviewed POs, six thought that APIK is an ineffective institution (two 40 See “Context in which the JA operates” section of this report. 33 disagreed) and POs did not note any substantive results from APIK in detecting corruption. Some POs (2) emphasized that APIK’s jurisdiction is not well defined, and few corruption cases come from APIK. Only three of the beneficiaries and stakeholders that the evaluation team interviewed considered APIK’s inputs for processing corruption important. Beneficiaries and stakeholders viewed APIK’s role primarily as collecting and disseminating information. Finding 31: More than a half of interviewed beneficiary POs (9) that the evaluation team interviewed had positive views of JA’s work in establishing contacts and communication between POs and audit institutions. Cooperation with the Audit Offices is just beginning. However, seven of the POs that the team interviewed were dissatisfied with their cooperation with these institutions. There were substantial differences and disagreements between POs and Audit Offices regarding the role of the Audit Office in investigation processes. While POs (2) believed that cooperation with the Audit Offices should be formalized, the Audit Office representatives preferred an informal level of communication. In RS, as specified by the law, audit reports go to the police for a forensic assessment. In FBiH, this is not the case. According to one interviewed PO, the Audit Office in FBiH is not willing to report when they believe criminal activities may be taking place. All BiH citizens are obligated to report such cases as well. Almost half of POs (7) stated they had no use for these audit reports because prosecutors do not understand them. According to one interviewed PO, when the Audit Office publishes audit reports with negative opinions, the public believes POs are not doing their jobs. The public does not understand that the negative audit report does not necessarily mean that there is a criminal offense, while a positive audit report does not mean that there is no criminal offense. Finding 32: Beneficiaries and stakeholders were rarely aware of the JA’s collaboration with Agencies for Management of Forfeited Assets. Most of those who were aware (3 KIs; 2 POs and one CSO) positively assessed their collaboration. Through cooperation with the Agency, it was clear that CMS does not capture information about unlawfully acquired assets. Because of that, the JA provides assistance to the HJPC to further develop the CMS/TCMS to capture information on the unlawfully acquired property. The JA’s assistance in relation to CMS/TCMS changes on forfeited assets is ongoing. Finding 33: The development of a database of adjudications of the FBiH/RS Supreme Court has begun and is welcomed by some beneficiaries and stakeholders as an important source of information for streamlining judicial practice. Most beneficiaries and stakeholders were not aware that the JA introduced any changes related to information exchange between judicial institutions and/or law enforcement agencies, although four stakeholders mentioned that JA influenced some changes to CMS/TCMS. However, some beneficiaries and stakeholders (7) disagreed that statistics and data exchange has improved. In particular, POs still cannot access databases used by police, or the Financial-intelligence Agency and Intermediary Agency for IT and Financial Services databases. Also, the FBiH PO does not have direct access to data on cases processed by Cantonal POs. To access these data, FBiH PO must acquire permission from the HJPC. Finding 34: One important limitation to investigating corruption, economic and organized crime cases is the absence of one centralized database on individuals and legal entities from which POs can extract data. Presently, POs have to contact multiple sources and go through numerous steps to obtain the necessary data. SIPA’s Financial Intelligence Department has this type of database, but because of legal restrictions, cooperation with them is limited only to potential money laundering cases. “Auditors are not officially authorized persons like police agencies, Tax Authority, etc. They cannot conduct work as stipulated by Criminal Procedure Code. Honestly, this idea was there before USAID activities where they should come closer to us. We have not achieved anything. What we achieved was that they submitted seven audit reports with negative judgement, despite [] our agreement, and their opinions were released to [the] public and we didn’t know about it. After that we had again meeting with Audit General and we failed to agree on anything. For that reason it is good that this activity is present.” -POs 34 Outcome/output 2.3: ODC is properly resourced to manage complaints procedures, autonomously reviews the conduct of judges and prosecutors, and recommends appropriate sanctions. Finding 35: The ODC valued the JA for its implementation approaches and is using the JA developed guidelines to handle complaints and propose sanctions related to judicial and prosecutorial performance. In addition, the JA provided technical assistance to ODC in developing strategic plans, MEL plan and annual reports, and trainings designed on the previously performed training needs assessment. The JA also performed a functional review of ODC and conducted analysis and provided recommendations to upgrade the ODC Case Management System (DCMS). The Annual ODC 2016 Report recognized these activities as positive.41 Finding 36: According to the JA’s representatives interviewed by the evaluation team, the EC peer review missions incorporated, or at least similarly identified the results of the JA’s analytical work, studies and recommendations, into EC peer review recommendations. Similarities between the JA’s Functional Review of ODC and the EC peer assessment of disciplinary procedures are provided in Exhibit 8 and Annex VII. Exhibit 8: The JA input for EC Peer Assessment USAID JA Functional Review of Office of Disciplinary Council (June 2015) EC Peer Assessment of Disciplinary Procedures in BiH Judiciary (November 2016) Define ODC as an independent and autonomous body in the Law on the HJPC. ODC must be an independent body, not a body of the HJPC, to ensure both fair investigations and the confidentiality of information (its own budget, separate headquarters, separate phone line, and their drivers). Extend the term of office of the Chief Disciplinary Counsel to six years, without possibility of renewal. The Head of ODC, appointed by HJPC, should come from the members of ODC; must have managerial skills; his/her mandate should last longer than the current 4 years, but without possibility of renewal. Develop a repository of knowledge on disciplinary practice to ensure proportionate and consistent decisions on disciplinary sanctions and capacities to monitor and analyze the sanctioning policy and practice. All disciplinary decisions should be included in a compilation of jurisprudence -Praxis Manual, and an analysis of sanctioning praxis shall also be conducted for each of the types of offenses. This manual should be made available to the public, preferably through an online version. Analyze the disciplinary policy and practice in detail, and identify problems in qualification of disciplinary offenses. Revise the list of disciplinary offenses in accordance with the findings of the disciplinary policy and practice analysis, taking into account a wider legal framework, codes of conduct of judges and prosecutors and recommendations of the Venice Commission. Disciplinary offenses should be defined more precisely. Establish a mechanism to control dismissed complaints. A system should be set up to oversee ODC decisions not to pursue or to close an investigation. Source: JA documents Also, according to the JA’s representatives, some of the JA’s interventions, such as the guidelines in disciplinary proceedings, Integrity Plans, and guidelines on prevention of conflict of interest, began before 41 https://www.pravosudje.ba/vstv/faces/docservlet?p_id_doc=40366 35 the EC peer review missions issued its recommendations and created the necessary foundations to address these recommendations by BiH justice sector institutions.42 Finding 37: The JA, was not tasked with direct dissemination of information about its work with the ODC and on disciplinary procedures with courts and POs. However, to provide context, the majority of POs (8) were not familiar with ODC’s work (see also Finding 45). Some POs (4) were familiar with changes to disciplinary procedures (e.g., ODC has recently been granted independent access to Case Management System/Prosecutors Case Management System (CMS/TCMS) records). Overall, beneficiaries that were familiar with disciplinary procedures and ODC’s work (3) were dissatisfied with disciplinary procedures and disciplinary decisions. One PO (1) believed that the disciplinary office is performing well and that ODC has managed to maintain its autonomy and independence when initiating disciplinary proceedings. ODC recently was granted independent access to CMS/TCMS records, which will provide ODC with better overview of judges’ and prosecutors’ work and improve ODC’s investigative capacities. Finding 38: The ODC uses the HJPC “Guidelines For Determining Disciplinary Measures” that were developed with JA assistance (see Finding 40). In particular, ODC found those Guidelines useful in the process of plea bargaining. Moreover, upon ODC’s proposal, these guidelines are quoted in decisions of the HJPC disciplinary panels (see Finding 7). However, as presented in the previous finding (see Finding 37), KIs in POs were unfamiliar with these changes. Outcome/output 2.4: Disciplinary proceedings are processed expeditiously and fairly and disciplinary decisions are subject to independent and impartial review Finding 39: Based on the survey of judges and prosecutors, in 2017, fewer than 50 percent of surveyed professionals believed that disciplinary procedures against judges and prosecutors were consistent, fair, and objective (see Exhibit 9). Only 36 percent of respondents thought that disciplinary procedures were appropriate. The evaluation team’s data analysis across years showed no significant change in respondents’ opinions. Exhibit 9: Do you agree that disciplinary procedures against judges/prosecutors are…? Source: The Survey of Judges and Prosecutors in BiH, conducted from 2015 through 2017 42 HJPC Letter to EC Delegation, Directorate General for Neighborhood and Enlargement Negotiations (DG NEAR), Number: 01-14-1-177-7/2017; June 16, 2017. 36 Finding 40: In order to ensure that the HJPC’s disciplinary sanctioning policy is consistent, the JA provided TA to ODC and HJPC members of the disciplinary committees in developing the HJPC Guidelines for Determining Disciplinary Measures. These guidelines were presented to the HJPC Standing Legislative Committee (SLC). The HJPC subsequently adopted the guidelines on July 8, 2016.43 The HJPC disciplinary commissions use and refer to the adopted guidelines in disciplinary cases against judges and prosecutors.44 In a highly regulated sector such as the judicial system, the HJPC adoption and its disciplinary committees’ implementation of the JA’s assistance evidences the highest possible level of vetting. The JA is working on developing a draft Judicial Discipline Benchbook in partnership with the HJPC. This Benchbook incorporates EU peer review recommendations related to efficiency, fairness, and transparency of disciplinary proceedings. The Judicial Benchbook will also incorporate a template for disciplinary decisions, as suggested by the EU peer review recommendations. The JA is actively cooperating with the HJPC WG on Integrity to address HJPC’s comments on the first draft and is preparing the second draft. The evaluation team finds that this activity is on track. Outcome/output 2.5: Public trust and respect for justice sector institutions/actors are increased with respect to demonstrated ability to act independently and impartially and to be held accountable. Finding 41: Perception data from surveys of citizens did not show any substantial changes in opinions regarding the justice sector’s impartiality or independence. Only 25 percent of citizens believed that judges are impartial, and 26 percent of citizens believed that prosecutors are impartial (see Exhibit 10). Exhibit 10: Judges can be trusted to conduct court procedures and adjudicate cases impartially and in accordance with the law / Prosecutors can be trusted to perform their duties impartially and in accordance with the law, percentage of respondents agreeing with the statements Source: NSCP surveys (conducted from 2015 through 2017) and the Survey of Judges and Prosecutors in BiH (conducted from 2015 through 2017) 43 https://usaidjp.ba/assets/files/publication/1470216400-high-judicial-and-prosecutorial-councils-guidelines-for-determining￾disciplinary-measures.pdf, HJPC/ODC 2016 annual report quotes at https://www.pravosudje.ba/vstv/faces/docservlet?p_id_doc=40366 , p.24 44 Guidelines applied in the Decision of the HJPC first instance disciplinary commission for judges, no 04-07-6-2340-4/2018, date September 5, 2018. Available at https://vstv.pravosudje.ba/vstv/faces/docservlet?p_id_doc=48237 37 Finding 42: In further examining public perceptions of the judiciary’s vulnerability to corruption, the evaluation team analyzed proxy indicators from the NSCP. These data show that a large portion of BiH’s citizens believed that corruption is deeply rooted in the judiciary. Exhibits 11 and 12 present citizens’ perceptions of bribery among judges and prosecutors in BiH. More than half (57 percent in 2017) believed judges take bribes; the same proportion believed prosecutors take bribes. However, these numbers are slightly lower than in 2015 and 2016 and there is little difference by entity. Exhibit 11: Bribery among Judges: How much do you agree or disagree with statement that judges do not take bribes? Source: NSCP 2017 Exhibit 12: Bribery among Prosecutors: How much do you agree or disagree that prosecutors do not take bribes? Source: NSCP 2017 38 Finding 43: The JA provided technical assistance to HJPC to develop both Integrity Plans and Guidelines for Preventing Conflict of Interest to promote accountability in the justice sector. In addition, the JA sponsored annual anti-corruption conferences held under the auspices of HJPC to further promote accountability in the justice sector. Technical assistance for Integrity Plans included developing integrity plan models and assistance in drafting guidelines for developing Integrity Plans (adopted by HJPC). This specific intervention was implemented on the basis of a tripartite Memorandum of Understanding (MoU) between APIK, HJPC, and USAID. The guidelines for the development of Integrity Plans required all courts and POs to develop and adopt their own Integrity Plans. Furthermore, the JA provided support to HJPC in reviewing individual Integrity Plans developed by courts and prosecutor offices. In addition, the JA committed itself to support the implementation of Integrity Plans in two locations (one court and one PO). The JA also provided assistance in drafting the guidelines on conflict of interest, including procedures for submission, verification, and management of asset declarations of judges and prosecutors. The JA invested significant effort in working with the HJPC on reforming the asset declaration rules by drafting the BoR, which will enable the use of asset declarations to combat corruption within the framework of the current Law on the HJPC. POs are still expected to demonstrate their ability to act independently and impartially, through successful implementation of the adopted documents. Finding 44: The majority of beneficiary POs (10) that the evaluation team interviewed considered the development of Integrity Plans to be very useful; this effort should be considered a success. The JA supports individual courts and POs in developing the integrity plan framework so that these plans comply with the Integrity Plan Guidelines. The JA provided input on the Integrity Plans of over 100 courts and POs in BiH. Integrity plans identified numerous risk processes, risk factors, and measures to be implemented in the next four years. POs previously developed Integrity Plans through other international projects (e.g., a UK-financed project), and some of them developed these plans as part of local government efforts (e.g., in RS). While the development of these plans was successful, a large number of interviewed POs (9), including the HJPC, believed that due to the broadness and comprehensiveness of Integrity Plans, implementation will be a challenge. Some interviewed POs (4) were also skeptical about successful implementation and noted that ultimate success remains to be seen over the next three to four years. According to some KIs, the main challenges to implementation of the Integrity Plans include: (i) informing and educating all judges and prosecutors (and other staff) about the content and substance of these plans; and (ii) tracking and monitoring implementation of the Integrity Plans by one person (one person designated by POs to monitor and coordinate implementation). However, BiH PO, for example, has 58 prosecutors and a total of 200 staff. It might be too great a challenge for one person to monitor how all prosecutors and staff comply with the integrity plan. “[Y]ou know what happened. Integrity plan is being done in Council [HJPC], and it is also individually done by government, and then you have two different documents, but in essence same documents. I immediately warned them not to, but alas, everybody is pushing its own. I have no expectations of those integrity plans, it is to me like strategy for war crimes is. We have it, if something is good, or if something is not good, who cares. We will make new one.” “[I]t is very important document, on the other hand very all encompassing and massive. Its practical implementation will require truly remarkable effort on behalf of whoever will be in charge [of] its implementation.” -PO 39 Finding 45: Although not directly related to the JA contract obligations and consequently delivered TA, it is relevant for USAID/BiH to know that a majority of beneficiary (8) and stakeholder (8) KIs noted that problems with the integrity of the justice sector and its performance start with an inadequate selection process for prosecutors and judges. Appointed prosecutors and judges are often incompetent and inexperienced. Although integrity plans, guidelines, and codes of conduct are important, they will have little effect if the system is not staffed with qualified personnel. Furthermore, beneficiary KIs stated that HJPC members rarely visit individual POs and that information on POs’ needs are not conveyed to HJPC. The evaluation team also noted that information on ongoing HJPC activities is not shared with POs. Finding 46: The JA developed a draft Ethics Handbook in partnership with HJPC. In developing the Ethics Handbook, the JA engaged with the HJPC WG on Integrity. Currently, the JA is addressing comments on the first draft and is preparing the second draft. The evaluation team finds that this activity, as outlined in the JA MEL plan, is on track. According to four beneficiary KIs, the main issue with enforcement of the Code of Ethics for Prosecutors and the Code of Ethics for Judges is that these codes are not obligatory, and breaches of these ethic codes are not directly treated as disciplinary matters. Therefore, judges and prosecutors do not consider ethic codes to be mandatory. According to these KIs, conduct prescribed in the ethics codes should be enforced by laws that punish those who violate ethical actions or behavior. This activity (which is ongoing) was conducted with a limited number of HJPC members and staff. As presented in finding 45, KIIs revealed that beneficiaries and stakeholders were rarely aware of the JA’s activities related to changes to ethics codes (only five interviewed beneficiaries were aware of these efforts). Finding 47: According to the JA’s representatives, additional JA interventions related to guidelines in disciplinary proceedings, integrity plans, and guidelines on prevention of conflict of interest will further contribute to the implementation of EC peer review recommendations. The JA’s work on amendments to the Law on the HJPC was large in scope and relied on other JA work as well, such as the Diagnostic Analysis of the Integrity of Judicial Sector and Potential Corruption Risks and the Functional Analysis of the ODC. Furthermore, the BoR on the submission, verification, and management of asset declarations for judges and prosecutors (developed by the JA in partnership with HJPC) will enable further compliance of the HJPC’s regulatory framework with the EU peer review recommendations. The JA began before the EC peer review missions provided recommendations. In many ways, the JA created the necessary foundations to address these recommendations among BiH justice sector institutions.45 Because the EC peer review recommendations will serve as guiding principles for further reforms in BiH justice sector and must be implemented through the development of laws, bylaws, and other regulation, the JA’s technical assistance provides direct operational support to implement these recommendations. 45 HJPC Letter to EC Delegation, Directorate General for Neighborhood and Enlargement Negotiations (DG NEAR), Number: 01-14-1-177-7/2017; June 16, 2017. “Judicial integrity within the judicial system is what it is. Are we happy with it, no we are not. Starting with a people selection method for justice sector, on this method we have no influence. It is incomprehensible for me that he is somewhere selected for Chief Prosecutor and nobody asks does he know anything, he came for some institution, nobody asks do you really need somebody like him…” “[A]s far as successful fight against corruption is concerned, I believe that HJPC is a key because HJPC is responsible for appointing judges and prosecutors. HJPC is instance that appoints Chief Prosecutors, and Chief Prosecutors in essence determine how seriously prosecutor office will fight corruption.” -PO “We had Ethics Codex introduced since 2003 after the justice reform. In the beginning judges and prosecutors predominantly adhered to those codes. I am for a long time, for 25 years in prosecution office, from the very beginning. I came from prosecutor at municipal level to now Acting Chief Prosecutor position. However, with time things changed, and prosecutors begun to break those codes. Perhaps they [Ethic Codex] do not contain all that they should.” -PO 40 Outcome/output 2.6: Prosecutors and judges are trained in identifying elements of corrupt activities and are able to investigate and prosecute corrupt practices and enforce the law against offenders Finding 48: The majority of beneficiary POs (9) that the evaluation team interviewed confirmed that specialized trainings equipped prosecutors with new skills. Interviewed POs considered specialized trainings for prosecutors to be an important and useful type of assistance. Moreover, this was mentioned as one of the most useful types of TA the JA delivered (see Finding 7). In addition, they noted that they were able to immediately put into practice the knowledge they gained from these trainings. Training topics related to corruption, organized crime, and cybercrime, and were well targeted; the training methods relied on a coordinated effort between the JA and POs. The JA provided trainings to between 30 and 40 prosecutors from the major POs in BiH; interviewed POs expressed their interest in future trainings. The JA designed and implemented this intervention together with the Swiss/Norwegian project. The JA drafted the Universal Benchbook on how to prosecute and adjudicate corruption, organized crime, and economic crime. The document incorporates the discussions, recommendations, and lessons learned from specialized trainings and provides a valuable instrument for securing the sustainability of this educational effort. This activity is on track and is expected to be completed on schedule in Y5. Finding 49: Despite specialized training and direct technical assistance from financial and economic experts (see Finding 48), all POs agreed that they still lacked economic expertise and the capacity to investigate economic crime cases. They continued to feel that prosecutors are in general ill equipped to prosecute economic crime cases. According to one KII, 11 prosecutors working in one PO’s economic department did not understand audit reports. Moreover, the majority of POs lack economic advisors. Through a KII the evaluation team also learned that INL will launch a training in forensic accounting for local government employees starting in fall 2018. Finding 50: KIIs with donors revealed that there is a lack of coordinated donor efforts. However, it should be noted that this is not within JA’s mandate. However, a few POs (3) said they would benefit from better coordination between different projects (USAID, Swiss / Norwegian project, United Nations Development Programme [UNDP], EU Twinning Light Projects, and Center for Education of Judges and Prosecutors) given that there are a lot of parallel activities. Furthermore, POs felt that the amount of time spent participating in these projects was problematic, especially for small POs. Some prosecutors stated that they spent more than 30 days in a six-month period on different trainings, and, therefore, POs were forced to choose which trainings to attend. Often, “Coordinate all aid activities, including that from Swiss and USAID and UNDP and Center for Education of judges and prosecutors. Often we have loads of those parallel activities helping us with our work, and they should be coordinated. We have some obligations towards the Center for Education, and sometimes we don’t have it towards you, a legal obligation. Therefore, we prefer to engage with things that we are legally obliged to do, rather than with those where we are figuring out on our own should we enage or not.“ “[P]oorly uncoordinated activities of this project [the JA] and HJPC and the Center’s project related to education, seminars, with the result that for many days per year, people are exposed to obligations on various projects.” “The Council [HJPC], through you, organized specialized trainings in three areas: corruption, cybercrime. Suddenly, three more states, spending the European Union’s money, appeared. Austria, Italy and one more connected to the cybercrime… we pointed out to them to connect with the Council [HJPC], look, people cannot manage it all. Then, the Ministry of Security appears with its own project, confusing prosecutors.” -POs 41 they selected events at the Center for Education because of legal obligations to participate in training programs organized by the Center.46 Finding 51: (the former Component 2): JA Component: Improving efficiency in the enforcement of judgments was predicted under the original JA contract. This Component operated in Year 1 of the JA’s implementation. By consequent modification of the JA’s initial contract in Year 2, this Component was abolished as requested by USAID/BIH. USAID/BiH requested that the JA refocus on its’ work with prosecutors and justice sector institutions to better combat corruption. By reviewing available JA documentation, the evaluation team found that in Year I this Component focused on completing the diagnostic assessment in the area of enforcement of civil judgments. The JA formed the Diagnostic Assessment Expert Team, consisting of prominent international and local experts who assisted JA in analysis and drafting of the Diagnostic Assessment Document on the Enforcement Regime of Civil Claims in Bosnia and Herzegovina (DAD). Similarly, as in case of DA within Component 1 (as elaborated previously), JA intended to create supplementary activities related to the implementation of findings and recommendations of DAD. DAD’s findings and recommendations were presented jointly with HJPC to the local professional community, consisting of HJPC and MOJs representatives, court presidents, enforcement judges, court bailiffs, bar and notary associations, representatives of banks and investors, and other legal professionals involved in enforcement procedures. This event also marked the end of the JA’s activity in this Component and the transition of achieved results to HJPC to continue work on implementing DAD’s recommendations. (EQ2) CONCLUSION The JA tailored and delivered its principal assistance directly to individual POs to address the identified needs of POs, as such specified in PPPs and delivered through POAPs. The main activities included strengthening organizational leadership and planning, balancing resource allocation, improving performance in prosecutors’ offices, and upholding public trust and integrity through prosecution of corruption and other serious crimes. The JA’s interventions equipped POs with tools to improve their tracking of administrative data to more effectively process CEC cases, increase their transparency, and improve communication with the media and public. The JA provided POs with tools to better organize their work and manage cases more effectively. In collaboration with the Swiss/Norwegian project, the JA contributed to the following achievements in this area: initiation of the Collegium of Chief Prosecutors; implementation of structured discussion on the achievements of the POs Strategic Plans and Annual Plans; and defining the quantitative indicators that guide the annual and strategic plans, including PR indicators, as per methodologies set in the POs’ PR MEL plan. The JA’s interventions and assistance in strategic planning have been important and useful to POs. The JA’s assistance to POs in improving the quality and transparency of POs’ public relations, including developing POs’ PR strategic documents and monitoring, evaluation and learning (MEL) plans, monitoring of corresponding performance indicators, improvements in content management and upgrades of POs’ websites, were especially useful to the POs that did not have a PR officer. 46 The HJPC determines a minimum level of trainings each prosecutor and judge must attend annually. According to the HJPC’s decision, the minimum number of training days attended by each prosecutor and judge is four. Source: http://www.rs.cest.gov.ba/index.php/o-nama 42 Overall, beneficiaries and stakeholders viewed the interventions implemented by the JA as positive and important to improve work in the judicial sector. Among the five types of assistance that respondents rated as the most useful, external experts, certified training, material assistance, Integrity Plans, and the Guidelines for Preventing Conflict of Interest, two compensated for resources currently missing in POs, namely experts and material assistance. The JA’s assistance in these two areas, although valuable and important, was not sufficient to cover the large needs of POs. POs remain underfinanced, understaffed, and underequipped. The JA made substantial efforts to develop a framework to provide incentives to prosecutors to prosecute CEC cases. However, despite the changes introduced in the prosecutor quota system, prosecutors still are not motivated to work on the most complex CEC cases. Adequately valuing prosecutors’ work on the most complex CEC cases remains an issue to be resolved by further JA interventions. Cooperation among state/entity and justice sector institutions initiated with the JA’s assistance remains primarily donor driven, and the cooperation may decline with the JA’s withdrawal. The JA managed to initiate cooperation between POs, and different government agencies and institutions that could contribute to prosecuting CEC cases. This cooperation included initial meetings and, in some cases, negotiations between POs and these agencies and institutions led to the establishment of procedures for future cooperation. However, POs’ opinions about the usefulness of this cooperation was unfavorable, as government agencies and institutions did not provide substantial support for the POs’ work. The JA’s activities directed toward the ODC (i.e. development of the ODC’s strategic plans, MEL plan and annual reports, trainings based on the training needs assessment, a functional review of ODC and recommendations for upgrading the ODC Case Management System (DCMS)) have been very useful to ODC staff. The evaluation team concludes that the HJPC Guidelines for Determining Disciplinary Measures developed with JA assistance are used by both the ODC and the HJPC disciplinary panels, and are formally applied in disciplinary proceedings. ODC was recently granted independent access to Case Management System/Prosecutors Case Management System (CMS/TCMS) records, which creates the opportunity for ODC to conduct data-driven ex-officio47 investigations and improve the ODC’s investigative work. However, current ODC staff lack the technical and analytical skills required to conduct data-driven investigations. The JA provided technical assistance to HJPC in developing Integrity Plans, Guidelines for Preventing Conflict of Interest, including development of Asset Declaration Forms, reviewing Codes of Ethics and developing the Ethics Handbook, the Judicial Discipline Benchbook and the Book of Rules on the submission, verification and management of asset declarations of judges and prosecutors that promote accountability in the justice sector. In addition, the JA sponsored annual anti-corruption conferences held under the auspices of HJPC. Technical assistance for Integrity Plans included developing integrity plan models and drafting the guidelines for developing Integrity Plans (adopted by HJPC). In developing and implementing Integrity Plans, the JA exceeded the expectations stipulated in the contract and MEL Plan, and KIs considered this to be one of the most successful JA activities. However, this remains an ongoing effort, and, thus, successfully completing this task will require further attention and resources. The JA’s Diagnostic Analysis of the Integrity of the Judiciary Sector in BiH and the Possible Risks of Corruption or Unethical Conduct in the Judiciary System (DA) and consequent studies (i.e., Analysis of System of Use of Expert Witnesses in Cases of Corruption, Organized and Economic Crimes, Analysis of Court Verdicts, Analysis of Asset Forfeiture) are generally perceived by beneficiaries as providing high quality assessments of justice sector issues. The JA’s analytical work and diagnostic studies began before the EC peer review missions. The results of the JA’s analytical work and diagnostic study were incorporated 47 Investigation conducted, and the case perused by authority and right of the ODC Office. 43 into, or at least supported by, the European Commission (EC) peer review. In addition, some JA interventions, such as developing diagnostic studies; guidelines on integrity plans, conflict of interest, and disciplinary sanctioning; functional analysis of the ODC; and asset declaration reform activities created the necessary foundations for the JA amendments to the new Law on the HJPC. As the recommendations of the EC peer review missions will guide continued reforms in the BiH judiciary, JA’s work in this area will become even more important. The JA provides support for implementing these recommendations by developing laws, bylaws, and other regulations. While major diagnostic studies and consequent analyses were of high quality, the evaluation team found that these studies were not translated into English, preventing non-local-language speakers from understanding the documents. For the JA targets set through the JA MEL plan, the evaluation team identified two major groups of indicators. The first group is connected with direct TA of the JA, while the second group includes indicators sourced from the administrative data on processing CEC cases and indicators sourced from perception data from the public, and judges, and prosecutors. The first group of indicators include the number of beneficiaries trained, the number of judges and prosecutors who earned a certificate of specialized (two-year long) training in the investigation, prosecution and adjudication of corruption and financial crimes, the number of courts and POs that adopted Integrity Plans, and number of codes of conduct revised for improved standards in judicial conduct, among others. These measures revealed consistent improvement throughout the activity and that JA has reached the MEL Plan results and potentially met the 2019 targets. The second group of indicators, sourced from administrative data, tracks all major steps in processing CEC cases, including processing criminal reports, conducting investigations, filing indictments, and obtaining convictions. From 2014 to 2017, prosecution data on corruption or economic crimes showed no noticeable advancements in processing these cases. While the number of criminal reports filed, which was the responsibility of police and individuals filing and out of the hands of POs, showed variation during this period, other key variables, including the number of investigations, indictments, and convictions, hardly experienced any changes. When such changes occurred, the direction of change varied from year to year: some JA MEL indicators, for example number of convictions and convictions rate, experienced a dramatic negative change in 2017 compared to 2016. Similarly, indicators sourced from the public and professional surveys showed volatility or inconsistent trends and patterns. As such, the evaluation team cannot estimate if the end-of-activity JA targets will be achieved. Further, because both the JA MEL data and HJPC administrative data measure the overall progress of the BiH judiciary to prosecute corruption and economic crimes, these indicators may be related to the effect of the JA, other donor-funded interventions, and the efforts made by domestic stakeholders. The complexity and interconnectedness of these efforts made it impossible for the evaluation team to disentangle the contributions of individual activities to the aggregate changes observed in the data. Despite this difficulty, the evaluation team concluded that introducing these indicators into the JA MEL plan and monitoring them were positive developments. USAID can use the resultant data to assess the judiciary’s progress in processing corruption and economic crime cases. As such, these indicators are a valuable source of information for decision makers regarding further programming. Unfortunately, the evaluation team concludes that there is tangible dissatisfaction among the public with the judiciary because there have been few observable results in the fight against corruption. The public perceives the justice sector to be corrupt, ineffective, unprofessional, and under the influence of politicians. 44 The majority of citizens and professionals (prosecutors and judges) believe that the judiciary is unable to identify and punish public officials who violate the law. Despite extensive onsite and offsite trainings provided by the JA, which beneficiaries welcomed and appreciated, the evaluation team’s analysis of KIIs showed that prosecutors, judges, and government agencies still lack the necessary expertise to prosecute economic crime and corruption, as well as cases related to cybercrime. Prosecutors, judges, and police need further training in CEC to systematically address the issues that are unique to these cases. Given that few domestic organizations possess the capacity to provide CEC training, continued donor engagement in this area may be necessary. To build upon the JA delivered training, upcoming training from INL in forensic accounting represents an opportunity for staff from POs to further improve their knowledge and skills in this area. Using semi-structured interview protocols, the evaluation team conducted KIIs that asked about JA activities as described in the JA’s contract and work plans. Nevertheless, during KIIs, participants provided relevant information about the BiH justice sector that was not directly related to the JA's tasks and performance. We discuss this information here, given its relevance for USAID/BiH's understanding of the sector and to provide context on the environment in which JA is operating. Namely, the evaluation team identified a noticeable lack of communication among stakeholders in the judicial sector. Judges and prosecutors are unaware of ODC’s activities and practice, and POs are unaware of individual practices of some POs related to improvements in processing CEC cases within the existing regulatory framework. In addition, HJPC members rarely visit individual POs to disseminate information on ongoing HJPC activities and receive information on PO needs. KIs pointed out a number of issues that negatively influence the work of justice sector institutions, including disciplinary proceedings, appointments, performance appraisals, current legislative and regulatory initiatives, and knowledge of issues and needs among judicial institutions, judges, and prosecutors. Further, KIs pointed out that these issues created serious impediments for POs and courts in effectively processing cases. Finally, the contract modification terminated activities that were in the early stage of development and implementation related to improving efficiency in the enforcement of judgments. Although the SoW for this evaluation did not include the evaluation of activities performed prior to this termination, through a review of available documents, the evaluation team recognized that similar approaches and implementation methodologies were planned, as in the case of DA, which included collaborative process with local stakeholders and HJPC as principal partner. The evaluation team concluded that initial activities within the former Component 2 were performed correctly. 45 Evaluation Question 3 (EQ3) Within the current task order (TO) scope, what opportunities can be identified to additionally complement the ongoing JA interventions? What are the lessons learned from implementation of the current TO? The evaluation team describes below some of the main opportunities and lessons learned that it identified in evaluating the JA and developing the Brief Assessment of the Justice sector in BiH. To avoid potential duplication, the more extensive analysis and further recommendations for future interventions in the justice sector in BiH are provided in the Brief Assessment of Bosnia and Herzegovina Justice Sector produced by MEASURE-BiH. Opportunities To build upon past work, USAID should continue activities within the current JA framework that help institutionalize the EC peer review recommendations and continue work on ongoing changes to the HJPC law. Under the JA, these interventions are primarily associated with Component 2. Namely, implementation of integrity plans, guidelines for preventing conflicts of interest, including implementation of the asset declaration forms, work with ODC, disciplinary proceedings, and asset forfeiture activities. The evaluation teams’ findings and conclusions presented through Evaluation Question 2, indicate that JA interventions related to processing CEC frequently did not achieve their targets on relevant MEL indicators. Namely, processing of criminal reports, investigations opened, indictments filed, and convictions for CEC were below the targets set. Additionally, there was little change in the number of CEC cases processed from 2014 through 2017. In addition, the public’s poor perception of the effectiveness of the judiciary in combating corruption and prosecution of public officials who violate the law indicated the BiH’s judiciary’s lack of advancement in processing high-profile corruption cases. The evaluation team found that changes to performance standards for prosecutors did not adequately motivate prosecutors to work on the most complex corruption cases. In addition, the evaluation team found that courts neither prioritized adjudication of corruption cases nor motivated judges to work on the most complex corruption cases. Consequently, because trials in corruption cases take multiple years, there is limited deterrence of potential new offenders. Given that the JA is in its final year of implementation, there is sufficient time for only a slight change in effort to address these issues. Nevertheless, the evaluation team suggests that USAID explore the following opportunities: Opportunity No.1: In addition to providing TA to POs in PPP group 4, resources should be also directed toward, to the extent possible, providing more technical assistance in prosecuting corruption cases to those POs that are more likely to prosecute high-profile cases, including the Republic of Srpska Special Prosecutors Office for Organized Crime and Corruption (RSSPO) and cantonal POs in Sarajevo, Zenica, Tuzla, and Bihac. Opportunity No. 2: Within the HJPC, there should be renewed initiatives to review prosecutors’ performance criteria and adopt amended Book of Rules regulating this topic. The JA should propose changes and amendments to authorize prosecutors who are working on the most complex corruption cases to be freed up from work on other cases. Additionally, the JA could provide technical assistance in establishing a procedure for determining which cases qualify as the most complex corruption cases. 46 Opportunity No. 3: The evaluation teams’ findings and conclusions presented through Evaluation Question 2, indicate that POs are rarely aware of technical assistance that the JA delivers to HJPC and other HJPC activities that should lead to improvements in POs’ work. During KIIs, interviewees discussed a number of issues that affect POs’ work, including that HJPC members do not visit individual POs or share information. USAID can help facilitate a better exchange of information between HJPC and POs, particularly regarding information related to JA assistance to the HJPC by facilitating visits by HJPC members to individual POs (at least an attempt to be made within PPP-4 group of POs) and enabling discussions between HJPC members and prosecutors on issues of mutual concern. Opportunity No.4: USAID should continue delivering training in budget preparation and include representatives of POs and the Ministry of Finance. This would facilitate an exchange of information about the Ministry’s expectations regarding budget proposals and PO needs. USAID should continue delivering specialized training to prosecutors, as it was found that POs are still in need of this type of training. In addition, as part of building upon the JA delivered specialized training, USAID should look for possibilities to involve representatives of POs in INL’s upcoming certified training program for forensic accounting. Opportunity No.5: All USG agencies operating in BiH include the fight against corruption as a priority, and USG agencies undertake related activities within their respective mandates. For example, the evaluation team identified a complementary INL training activity. USAID should identify opportunities for synergies and to maximize return on investment by reinforcing of projects and activities across USG activities and project. USAID should attempt to foster more effective coordination between JA and INL-financed OSCE ARC work related to monitoring corruption trials and analysis of verdicts in order to maximize the effects of USG-financed interventions. At a minimum, JA and the INL-financed OSCE ARC project should engage in an active exchange of information about their ongoing activities in this area, and coordinate in planning their future activities and organizing joint public presentations of JA and INL-financed OSCE ARC work. Opportunity No. 6: To the extent possible, USAID should provide technical assistance to ODC in using Business Intelligence software and analytical skills to identify potentially corrupt behaviors in courts and POs in BiH. For example, data-mining and analysis of CMS/TCMS data could identify irregularities in processing cases. Attempts to avoid the random case assignment in courts and POs create an opportunity for corruption. This can in turn influence inappropriate outcomes of courts’ and POs’ proceedings, as identified in the JA diagnostic study.48 In June 2018, ODC received full access to CMS/TCMS records, which provides an opportunity to identify how often or how severely these attempts are made. In order to examine possible risks of corruption, it will be necessary to test and analyze CMS/TCMS database records and identify cases which have unusual patterns in their processing. ODC should then review these cases individually on case-by-case basis. If ODC identifies criminal acts, ODC should report them to a competent PO. However, current ODC staff lack the skills needed to perform data-mining and related analytical work. This creates an opportunity for JA to deliver further technical assistance. Opportunity No. 7: Finally, the evaluation team found that the JA studies were very valuable sources of information. However, their dissemination is limited because they have not been translated into English. Additionally, a limited number are available on the JA web site. USAID should translate major diagnostic studies prepared by the JA into English and make them available to USAID/BIH at a minimum. 48 JA Diagnostic study: The integrity of the judicial sector in BiH and potential risks of corruption or unethical conduct in judiciary, p 21. 47 Lessons Learned Lessons Learned No.1: It is necessary to thoroughly map other donor activities prior to designing an Activity to avoid overlapping interventions and use of resources, and to ensure effective coordination. USAID/BiH should map donor assistance continuously and thoroughly and update donor activities. This should not be limited to before and during the design phase of USAID/BiH’s specific activities. As recommended in the BiH Justice Sector Brief Assessment,49 USAID/BiH might consider offering technical assistance to HJPC in coordinating donor projects. This could include the creation of an interactive platform available to all donors. An interactive platform would enable donors to view how their own activities interact and overlap with the efforts of others and would be beneficial not only to USAID/BIH, but to the BiH justice sector and other donors more broadly. This type of platform could direct limited resources to areas where they are most needed without overlapping or duplicating efforts. Furthermore, although joint work with other donors is often desired, when working on the same activities, USAID’s efforts should be effectively identified and branded to distinguish them from the interventions of other donors. Lessons Learned No.2: Activity contracts should balance flexibility with quantifiable monitoring. When USAID expects Activities to meet exact outputs or outcomes (i.e., number of indictments filed, or convictions for corruption cases), these should be clearly established in the IP’s contract. However, as evidenced in the JA, a lack of contractual targets often facilitates flexibility in implementation of Activity’s interventions (i.e., allowing for ‘demand driven approach’). A prioritization of flexibility, quantification, or an appropriate balance between the two, should be clear in the Activity contract. Lessons Learned No.3: Focusing an intervention only on POs (one component of the justice sector) can be viable if it was ensured that courts or law enforcement agencies will follow on progress achieved by assisted POs. Differences in capacities and a lack of coordination among all key stakeholders in processing criminal cases limits achievements of interventions directed towards one component of the justice sector only. If limited resources are available for an Activity, a smaller number of institutions should be selected, but those selected institutions should cover the whole chain of processing criminal cases. Successes achieved on a limited scale by these institutions, should be replicated in other (non-treated) institutions by disseminating good practices. Lessons Learned No.4: The highly regulated environment in which judges and prosecutors work in BiH requires that new interventions’ priorities are adequately reflected in changes in corresponding regulations or policy documents. During the JA’s implementation the Book of Rules, which defines the metrics by which judges’ and prosecutors’ performance is measured against, did not adequately prioritize work on the most complex corruption cases. These cases were also not prioritized within Strategic plans and consequently, judges and prosecutors were given the option to work on simpler cases and more easily meet performance expectations. Given absence of these priorities, technical assistance to support processing the most complex corruption cases was ineffective. Lessons Learned No.5: The appointment processes for judges and prosecutors need careful attention when designing new activities in the justice sector. The appointments of judges and prosecutors are a key point for establishing professionalism and integrity within the justice sector, as mentioned by a majority of KIIs (16). The design of new justice sector activities should include a review of the appropriateness of current appointment procedures and criteria, and if found inadequate, technical 49 MEASURE-BiH: The BiH justice sector brief assessment available to USAID/BiH 48 assistance should address identified shortcomings. When unqualified or politically connected judges and prosecutors are appointed, technical assistance aimed at reducing corruption may be ineffective. Lessons Learned No.6: The absence of observable results in the fight against corruption has resulted in dissatisfaction among citizens with the work of the entire justice sector. The evaluation team found that the public perceives the justice sector as ineffective, unprofessional, under the influence of politicians, and corrupt. The majority of citizens and professionals (prosecutors and judges) believe that the judiciary is not able to identify and punish public officials who violate the law. Public perception of work of the justice sector will not improve until the justice sector demonstrates its ability to prosecute high-profile CEC cases. 49 ANNEXES 50 ANNEX I: DETAILED PRESENTATION OF THE JA MEL INDICATORS (ACTUALS AND TARGETS) SOURCED IN HJPC ADMINISTRATIVE DATA This annex contains an overview of JA MEL plan indicators. The team presents the indicators in consecutive order, as provided in the JA MEL plan. For Indicators related to outputs/outcomes 1.1 – 1.3, the team presents each indicator by name with a corresponding exhibit that provides the following details: the full name of the indicator, its definition, targets and actuals. Each exhibit also contains a graphical presentation for actuals/targets in 2014 to 2019, as stated in the JA MEL plan (graph in the left bottom corner), and actuals in 2014 to 2017, in accordance with the corresponding HJPC administrative data where the actual for 2017 was calculated by the evaluation team (graph in the right bottom corner). For all indicators, the team provides a narrative description of results achieved so far and status of current actual versus the end-of-activity target. 1.1.1 Number of convictions for corruption related crime in POs that implement PPPs Exhibit 2 in the main report provides tabular and graphical outputs for this indicator. The evaluation team found a small increase in the number of convictions for corruption related cases in 2017 over the 2014 baseline: 13 more convictions (or 6.1%). Although there was a positive trend in the increase of number of convictions for corruption cases for the period 2014 to 2016, there was a negative annual change in the number of convictions in 2017 as compared to 2016 (10 convictions less than in 2016). The target set for this indicator in 2019 is 261 convictions, which is 15 percent above its actual in 2017. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 1.1.2 Number of indictments for corruption related crime in POs that implement PPPs Exhibit 1 in the main report provides tabular and graphical outputs for this indicator. The evaluation team found that the number of indictments for corruption related cases had a small increase in 2017 over the 2014 baseline, more precisely an increase of 10 indictments (or 3.7%). The trend in the number of indictments for corruption cases was highly volatile. While there was a dramatic positive change in 2015, with an increase of 61 indictments, there was a negative change in 2016, when 72 fewer indictments were filed than in the year before. The target set for this indicator in 2019 is 298 indictments, which is 5.3 percent above its actual in 2017. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 51 1.1.3 Conviction rate for corruption related crime in POs that implement PPPs Exhibit 13: 1.1.3 Conviction rate for corruption related crime in POs that implement PPPs Source: The JA MEL Plan (Modification 450) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the conviction rate of verdicts for corruption offenses increased in 2017 over the 2014 baseline of 77.4 percent conviction rate by 5.31 percentage points (pp). Although there was a positive trend in the increase of the conviction rate of verdicts for corruption offenses in 2014 to 2016, there was a negative annual change in the conviction rate in 2017 compared to 2016 (3 pp lower than in 2016) and the 2017 annual target was not achieved. The target set for this indicator in 2019 is 88 percent, which is 5.2 pp above its actual in 2017. Exhibit 13 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 50 Table is copied directly from JA MEL plan (modification no. 4). It should be noted that summary numbers in the last row of the table are incorrect. The evaluation team checked the numbers presented in the last row, against raw datasets provided by HJPC, and found that consistently in all 10 JA’s MEL indicators (covering outputs 1.1 through 1.3) sourced from HJPC administrative data, summary data present aggregate values for all POs in BiH (including BiH PO which is not beneficiary of the JA Activity). The evaluation team thought that this inconsistency in the JA MEL Plan is misspelled or that whole the JA MEL Plan and the last rows in these indicators need revision. Graphs of the data (as stated in the upper table) in JA MEL plan are provided on the left-hand side, while the right￾hand side graph contain additional data on actuals for 2017 sourced from HJPC administrative data, which are calculated by the evaluation team. The notes made here equally apply for presentation of all indicators in JA Outcomes 1.1 – 1.3. 52 1.2.1 Number of unresolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs Exhibit 14: 1.2.1 Number of unresolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 451) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the number of unresolved criminal reports for corruption and economic crime was lower in 2017 (2414 unresolved criminal reports) compared to the 2014 baseline figure of 2,459 unresolved criminal reports. The current level of decrease in 2017 compared to 2014 was 45 criminal reports (or -1.8%). Between 2014 and 2016, the number of unresolved criminal reports increased, thus the JA MEL targets were not achieved. In 2017, the first decrease of unresolved criminal reports in the observed period occurred. In total, there were 68 fewer unresolved criminal reports in 2017 compared to 2016. Still, the 2017 target was not achieved. The target set for this indicator in 2019 is 2,213 unresolved criminal reports, which is 9 percent below its actual in 2017. Exhibit 14 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 51 As provided in footnote no. 50. 53 1.2.2 Percentage of resolved criminal reports for corruption and economic crime versus total number of cases in prosecuting offices that implement PPPs Exhibit 15: 1.2.2 Percentage of resolved criminal reports for corruption and economic crime versus total number of cases in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 452) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the percentage of resolved criminal reports for corruption and economic crime versus the total number of cases in POs improved in 2017 (56.86%) over the 2014 baseline of 54.06 percent, an increase of 2.8 percentage points. Although this was a positive change, the actual deteriorated in 2017 compared to 2016 by 3.3 pp. The evaluation team also noticed that in Year 3 the end-of-activity target was reached (60.18%), but it was not maintained at that level in 2017. The target set for this indicator in 2019 is 60 percent, which is 3.1 pp above the current actual in 2017. Exhibit 15 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 52 As provided in footnote no. 50. 54 1.2.3 Number of resolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs Exhibit 16: 1.2.3 Number of resolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 453) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the number of resolved criminal reports for corruption and economic crime was higher in 2017, at 3,192 compared to the 2014 baseline of 2,897 resolved criminal reports representing an improvement of 8 percent. While there was a positive trend in resolving these criminal reports in 2014 to 2016, a dramatic decrease was experienced in 2017, when the number of resolved criminal reports was 462 less than in 2016, thus not achieving the 2017 target. The target set for this indicator in 2019 is 3,865 resolved criminal reports, which is 17 percent above its actual in 2017. Exhibit 16 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 53 As provided in footnote no. 50. 55 1.3.1 Percentage of investigations for corruption and economic crime completed within the one calendar year in prosecuting offices that implement PPPs Exhibit 17: 1.3.1 Percentage of investigations for corruption and economic crime completed within the one calendar year in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 454) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the percentage of investigations of corruption and economic crime cases completed versus the total number of investigations opened in a calendar year was lower in 2017 as compared to the 2014 baseline of 67.58 percent. The evaluation team noticed that the end-of-activity target for this indicator was reached in 2015, but it did not remain at that level in the following years. This indicator showed a downward trend from 2015 to 2017. While there was a positive change in 2015, negative changes occurred in both subsequent consecutive years, making the 2017 actual 1.3 pp below the 2014 baseline. The target set for this indicator in 2019 is 70 percent and the current actual is 3.7 pp below this target. Exhibit 17 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 54 As provided in footnote no. 50. 56 1.3.2 Number of investigations for corruption and economic crime completed in one calendar year in prosecuting offices that implement PPPs Exhibit 18: 1.3.2 Number of investigations for corruption and economic crime completed in one calendar year in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 455) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the number of investigations for corruption and economic crime completed in one calendar year was lower in 2017, at 1,237, compared to the 2014 baseline of 1,432 investigations completed (or -13.6%). While there was a positive change in number of investigations completed from 2014 to 2016, 343 fewer investigations were completed in 2017 than in 2016 (or -21.7%). The target set for this indicator in 2019 is 1,641 competed investigations, which is 24.6 percent above its actual in 2017. Exhibit 18 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 55 As provided in footnote no. 50. 57 1.3.3 Number of ordered investigations for corruption and economic crime in one calendar year in prosecuting offices that implement PPPs Exhibit 19: 1.3.3 Number of ordered investigations for corruption and economic crime in one calendar year in prosecuting offices that implement PPPs Source: The JA MEL Plan (Modification 456) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the number of ordered investigations for corruption and economic crime in one calendar year was lower in 2017, at 1,174, as compared to the 2014 baseline of 1,289 ordered investigations (or -9%). While there was a positive change in the number of ordered investigations from 2014 to 2016, in 2017 there was a dramatic decrease, at 493 ordered investigations compared to 2016. The target set for this indicator in 2019 is 1,726, which is 47 percent above the actual value on this indicator in 2017. Exhibit 19 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will be reached. 56 As provided in footnote no. 50. 58 1.3.4 Number of unresolved investigations for corruption and economic crime in one calendar year in prosecutor offices that implement PPPs Exhibit 20: 1.3.4 Number of unresolved investigations for corruption and economic crime in one calendar year in prosecutor offices that implement PPPs Source: The JA MEL Plan (Modification 457) and HJPC administrative data Note: In accordance with the JA MEL plan, actuals in the left-hand graph are presented in blue (2014-2016) and targets are presented in red (2017-2019). Actuals in accordance with HJPC data are presented in the right-hand graph where actual for 2017 was calculated by the evaluation team. The evaluation team found that the number of unresolved investigations for corruption and economic crime in one calendar year was lower in 2017 at 630, compared to the 2014 baseline of 688 unresolved investigations (-8.4%). The trend in the number of unresolved investigations for corruption and economic crime is highly volatile. While there was a positive change in 2015, when the number of unresolved investigations decreased to 81 unresolved investigations, a negative change was present in 2016, as reflected by the increase of 91 unresolved investigations. In 2017, there was a decrease of 68 unresolved investigations compared to 2016. The target set for this indicator in 2019 is 633 and the 2017 actual achieved both this target and the end-of-activity target. Exhibit 20 provides tabular and graphical presentation of targets and actuals for this indicator. Due to the volatility of actuals in 2014 to 2017, the evaluation team cannot estimate if the end-of-activity targets will continue to be maintained. 57 As provided in footnote no. 50. 59 1.4 Prosecutors provide appropriate and accurate information to citizens to strengthen transparency and responsiveness In accordance with the JA MEL Plan, outcome 1.4 - Prosecutors provide appropriate and accurate information to citizens to strengthen transparency and responsiveness - is tracked by three indicators: 1.4.1. Number of POs that regularly disseminate information to the public through their websites, including appropriate information on corruption and economic cases in POs that implement PPPs; 1.4.2 Number of press releases (or other types of communication channels) issued to the public including information on investigation and prosecution for corruption, economic crime cases in POs that implement PPPs; 1.4.3 Percentage of public that agree that BiH POs adequately inform public about their work (net high – strongly agree, agree, and somewhat agree). The evaluation team examined the JA MEL indicators that tracked this outcome. 1.4.1 Number of POs that regularly disseminate information to the public through their websites, including appropriate information on corruption and economic cases in POs that implement PPPs In accordance with the JA MEL Plan, as of 2017, 10 POs regularly disseminated information to the public. This met both 2017 and the end-of-activity targets of 10. 1.4.2 Number of press releases (or other types of communication channels) issued to the public including information on investigation and prosecution for corruption, economic crime cases in POs that implement PPPs In accordance with the JA MEL Plan, a 2019 target for indicator 1.4.2 (number of press releases) was not established for all POs in BiH (or all POs which implement PPPs). As noted in the JA 2017 Annual Report, the cumulative actual for PPP Group I was 1,124 and the cumulative actual for PPP Group 2 was 1,794. In accordance with the JA Annual 2017 Report both PPP groups exceeded their cumulative targets for 2016 (400 for PPP Group I and 1,400 for PPP Group II). 1.4.3 Percentage of public that agree that BiH POs adequately inform public about their work (net high – strongly agree, agree and somewhat agree) JA MEL Plan indicates that calculation of this indicator is based on MEASURE-BIH NSCP data. According to the evaluation team’s analysis of NSCP data, the 2017 actual for indicator 1.4.3. (Percentage of public that agree that BiH POs adequately inform public about their work) was 28.76 percent. This was lower than the 2016 baseline of 28.96 percent and 1.2 percentage points below the 2017 target. The 2019 target for indicator 1.4.3. is 5 percentage points over the 28.96 percent actual established for 2016. 1.5. Prosecutor status improved through performance appraisal, merit-based career advancement, or incentives to prosecute cases In accordance with the JA MEL plan, the outcome 1.5. - Prosecutor status improved through performance appraisal, merit-based career advancement, or incentives to prosecute cases - is tracked by one indicator: 1.5.1. Number of PPP Prosecutors’ Offices with a score of 4 or 5 on JA’s Prosecutors’ Office Capacity Matrix (POCM) for appraisal process dimension. 60 1.5.1. Number of PPP Prosecutors’ Offices with a score of 4 or 5 on JA’s Prosecutors’ Office Capacity Matrix (POCM) for appraisal process dimension In accordance with the JA MEL plan, the 2019 target for indicator 1.5.1 is 12 POs scoring 4 or 5 on the POCM appraisal process dimension. According to the JA 2017 Annual Report, the 2016 actual was 0, while the target for that year was 3. The target was not reached. 2.1. Prosecutors investigate and prosecute high profile corruption and economic crime cases free from political or improper influence In accordance with the JA MEL plan, outcome 2.1 is tracked through performance indicators 1.1.1 to 1.3.4, which were presented by the evaluation team through findings related to outcomes 1.1 through 1.3. 2.2. Increased cooperation among state, entity, and local justice and other sector actors In accordance with the JA MEL plan, outcome 2.2. - Increased cooperation among state, entity, and local justice and other sector actors - is tracked by one indicator: 2.2.1. Number of collaborative mechanisms established and/or improved among justice sector institutions and other parties involved in anti-corruption efforts. 2.2.1. Number of collaborative mechanisms established and/or improved among justice sector institutions and other parties involved in anti-corruption efforts In accordance with the JA MEL plan, the 2019 target for indicator 2.2.1 is 4 collaborative mechanisms established or improved. According to the 2017 JA Annual Report, 9 were established and the target for 2017 was 3. Therefore, the target for this indicator was reached for both 2017 and the end-of-activity. 2.3. ODC is properly resourced to manage complaints procedures, autonomously review the conduct of judges and prosecutors, and recommend appropriate sanctions In accordance with the JA MEL plan, the outcome 2.3. - ODC is properly resourced to manage complaints procedures, autonomously review the conduct of judges and prosecutors, and recommend appropriate sanctions - is tracked by one indicator: 2.3.1. Number of improvements implemented in ODC performance in disciplinary matters. In accordance with the JA MEL plan, the 2019 target for indicator 2.3.1 is 4 improvements in ODC performance implemented. According to the 2017 JA Annual Report, the actual number of improvements implemented was 4, which matched the 2017 and the end-of-activity target. This indicates that the target for this indicator was reached for both 2017 and the end-of-activity. 2.4. Disciplinary proceedings are processed expeditiously and fairly, and disciplinary decisions are subject to independent and impartial review In accordance with the JA MEL plan, outcome 2.4. - Disciplinary proceedings are processed expeditiously and fairly, and disciplinary decisions are subject to independent and impartial review - is tracked by two indicators: 2.4.1. Judicial Discipline Bench book and Ethics Handbook that will enable ODC staff and disciplinary committees to conduct proceedings more expeditiously and with greater consistency (fairness) developed; 2.4.2 Percentage of judges and prosecutors who strongly agree with the statement that the disciplinary process is fair and impartial. The evaluation team examined JA MEL indicators that tracked this outcome through the JA MEL plan. 61 2.4.1. Judicial Discipline Bench book and Ethics Handbook that will enable ODC staff and disciplinary committees to conduct proceedings more expeditiously and with greater consistency (fairness) developed In accordance with the JA MEL plan, the 2019 target for indicator 2.4.1 is to develop the Judicial Discipline Bench book and Ethics Handbook developed. According to the JA 2017 Annual Report, these documents had not yet been developed. However, the evaluation team did not expect this task to be completed by 2017. As noted in JA’s 2017 Annual Report, the Judicial Bench Book and Handbook have been drafted and presented to the HJPC’s Standing Committee, which provided JA with their comments on the submitted documents. The JA currently is working to address these comments and deliver revised documents; thus, the JA is on track to achieve its end-of-activity targets. 2.4.2 Percentage of judges and prosecutors who strongly agree with the statement that the disciplinary process is fair and impartial According to the JA 2017 Annual Report, the 2017 actual for indicator 2.4.2 was 33.94 percent. The target for 2017 was 43.51 percent, representing an increase of 5 percentage points above the 2015 baseline. This implies that the JA activity did not reach the 2017 target and was 12.57 percentage points below the 2019 target. The 2019 target for indicator 2.4.2 is 8 percentage points above the 38.51 percent baseline established in 2015. 2.5 Public trust and respect for justice sector institutions/ actors are increased with respect to demonstrated ability to act independently and impartially and to be held accountable In accordance with the JA MEL Plan, outcome 2.5 - Public trust and respect for justice sector institutions/ actors are increased with respect to demonstrated ability to act independently and impartially and to be held accountable - is tracked by three indicators: 2.5.1 Percentage of public who agree with the statement that the judiciary is not vulnerable to corruption; 2.5.2 Number of codes of conduct revised for improved standards in judicial conduct; 2.5.3 Number of Courts and POs that adopted Integrity Plans. The evaluation team examined JA MEL indicators that tracked this outcome through the JA MEL plan. 2.5.1 Percentage of public who agree with the statement that the judiciary is not vulnerable to corruption According to the JA 2017 Annual Report, the 2017 actual for indicator 2.5.1 was 22.7 percent. The target for this indicator in 2017 was 14.6 percent (representing 5 percentage points above the 2015 baseline of 9.6%). Therefore, the JA activity reached the 2017 target. The 2019 target for indicator 2.5.1 is 17.6 percent. The JA already has reached the desired level of target for this indicator for the end-of-activity and should maintain it at that level (or above) to surpass the end-of-activity target. 2.5.2 Number of codes of conduct revised for improved standards in judicial conduct By 2019, the target for indicator 2.5.2 (codes of conduct revised for improved standards in judicial conduct) is two revised codes. While the target for 2017 was to have one code revised, according to the JA 2017 Annual Report, two revisions were complete. The target for this indicator has already been achieved for both 2017 and the end-of-activity. 62 2.5.3 Number of Courts and POs that adopted Integrity Plans The 2019 target for indicator 2.5.3 is 20 Integrity Plans adopted by POs and courts. According to the JA 2017 Annual Report, by 2017, no Integrity Plans had been adopted. However, it was not expected that any plans would be adopted by 2017 (the 2017 target was 0). Through KIIs, the evaluation team learned that a substantial number of courts and POs have already adopted the Integrity Plans, although this was not yet reflected in the JA MEL Plan version that the evaluation team possessed. The evaluation team concludes that the JA already reached its end-of-activity target for this indicator. 2.6 Number of judges and prosecutors earned the certificate of specialized training in the investigation, prosecution and adjudication of corruption and financial crime cases According to the JA 2017 Annual Report, 24 judges and prosecutors earned the certificate of specialized training in the investigation, prosecution, and adjudication of corruption and financial crime cases. According to the JA MEL Plan, the targeted number of trained and certified judges and prosecutors in 2017 was 0, thus the JA was on track to reach its 2018 target, and as per information gained through KIIs, the JA was also on track to reach its end-of-life activity which is 50. USAID Standard Foreign Assistance Indicators JA reports on the number of beneficiaries trained through USAID standard foreign assistance indicators. DR 1.3-1 Number of judges and judicial personnel trained with USG assistance, and DR 2.4-1 Number of government officials receiving USG-supported anti-corruption trainings. Indicator DR 1.3-1 and DR 2.4-1. According to JA’s MEL Plan, the cumulative the life of Activity (2019) target for DR 1.3-1 is 600. According to the JA 2017 Annual Report, the target for this indicator in 2017 was 100. The reported actual in 2017 was 1,032, exceeding the target for 2017. According to the JA MEL Plan, the cumulative the life of Activity (2019) target for DR 2.4-1 is 420. According to the JA 2017 Annual Report, the target for this indicator in 2017 was 100 and the reported actual in 2017 was 426, exceeding the 2017 target. The JA exceeded the targeted number of beneficiaries trained through their interventions. 63 ANNEX II: DETAILED PRESENTATION OF HJPC ADMINISTRATIVE DATA ON PROCESSING CORRUPTION CASES IN 2014 – 2017 IN BIH AND SELECTED NUMBER OF POs In addition to reviewing the JA MEL indicators and KII findings, the evaluation team analyzed spatial data on the processing of corruption cases by HJPC from 2014 through 2017. These data summarize the JA MEL indicators (see Annex I) and provide additional information (not covered by the JA MEL indicators) on the processing of corruption cases from initiation in POs (a criminal report being filed) through investigations, indictments filed, and convictions in a calendar year. This analysis allowed the evaluation team to review both aggregate results for all POs in BIH and results for each individual PO. Overall, HJPC administrative data show that there has not been a substantial change in processing corruption cases between 2014 and 2017. The number of criminal reports filed, which is the responsibility of police and individual filings rather than POs, varied from 2014 through 2017. Other key variables, such as the number of convictions and indictments for corruption cases (see Exhibit 21), showed little change over this period. When changes have occurred, the direction of change varied from year to year. For example, the number of investigations opened in 2015 and 2016 was higher than the baseline value in 2014. On the contrary, the value for 2017 was less than values from all three previous years. The JA implemented PPPs/POAPs with groups of POs. The first group consisted of PO Doboj, PO Zenica, PO Tuzla, and PO East Sarajevo (PPP-1), the second group consisted of PO Siroki Brijeg, PO Mostar, PO Trebinje, PO Sarajevo, and PO FBiH (PPP-2), the third group consisted of PO Banja Luka, PO Orasje, PO Bijeljina, PO Brcko and PO RS (PPP-3), and the fourth group consisted of PO Bihac, PO Prijedor, PO Gorazde, PO Livno and PO Travnik (PPP-4). At the time the evaluation team conducted this performance evaluation, the JA had completed implementation in the first three PPP groups, while it was in progress in group four. 64 Exhibit 21: Aggregate results in processing corruption cases by all prosecutor offices in Bosnia and Herzegovina, 2014-2017 Source: HJPC administrative data and MEASURE-BIH GIS online presentation of the HJPC administrative data 65 To examine individual performances of POs in processing corruption cases and estimate if any PO had a different pattern and trend in processing corruption cases than the general trends for the whole of BIH, the evaluation team conducted several additional analyses, which are presented in this Annex. The analysis targeted to examine performance of POs over the period of time in which JA PPPs/POAPs were implemented or were in the process of being implemented (group PPP-4). The evaluation team conducted these analyses by using MEASURE-BIH’s GIS online presentation of processing corruption cases, which used HJPC administrative data in accordance with the HJPC Corruption Cases List. Due to a large number of Exhibits and limitation in available space for this report, the team presents in this Annex only processing of corruption cases. Details on the complete analysis of variables not covered here, as well on data on economic crimes or joint data for corruption and economic crime cases, are available in the full online GIS application that was prepared for this evaluation. Access rights to the application can be obtained through MEASURE-BiH COR. The evaluation team made the analysis by comparing performances of 17 cantonal and district POs in the following stages of the criminal procedure: criminal reports, investigations, indictments, and convictions. The team compared these four stages for each year from 2014 to 2017. The team compared 10 cantonal POs in FBiH, 6 district POs in RS, and the PO of the Brcko District. The full list of PO names and corresponding abbreviations used in mapping is provided in Exhibit 22. Exhibit 22: Full Prosecutors’ Office names and corresponding abbreviations used in mapping Abbreviation PO Full Name Administrative Seat JA PPP Group BI Prosecutors’ Office of the Una-Sana Canton Bihac PPP-4 OR Prosecutors’ Office of the Posavina Canton Orasje PPP-3 TZ Prosecutors’ Office of the Tuzla Canton Tuzla PPP-1 ZD Prosecutors’ Office of the Zenica-Doboj Canton Zenica PPP-1 GO Prosecutors’ Office of the Bosnia Podrinje Canton Gorazde PPP-4 TR Prosecutors’ Office of the Central Bosnia Canton Travnik PPP-4 MO Prosecutors’ Office of the Herzegovina-Neretva Canton Mostar PPP-2 SB Prosecutors’ Office of the West Herzegovina Canton Siroki Brijeg PPP-2 SA Prosecutors’ Office of the Sarajevo Canton Sarajevo PPP-2 LI Prosecutors’ Office of the Canton 10 Livno PPP-4 BL District Prosecutors’ Office in Banja Luka Banja Luka PPP-3 BN District Prosecutors’ Office in Bijeljina Bijeljina PPP-3 DO District Prosecutors’ Office in Doboj Doboj PPP-1 TB District Prosecutors’ Office in Trebinje Trebinje PPP-2 IS District Prosecutors’ Office in East Sarajevo East Sarajevo PPP-1 BD Prosecutor's Office of Brcko District BiH Brcko PPP-3 PR District Prosecutors’ Office in Prijedor Prijedor PPP-4 Analysis of data in criminal reports phase The team’s comparison analysis of case load in processing corruption criminal reports identified five POs with big numbers of criminal reports compared to all others. Those five were POs located in Sarajevo, Banja Luka, Tuzla, Zenica, and Bihac. As found in the presentation of the data on processing corruption criminal cases for the whole of BiH, a big number of rejected criminal reports was equally present in each individual PO. The comparative graphical presentation of processing corruption criminal reports is provided in Exhibit 23. 66 Exhibit 23: Processing corruption criminal reports in 2014 – 2017 by cantonal/district POs 67 Analysis of data in investigation phase Comparison analysis of case load in investigation phase (see Exhibit 26) identified the same five POs as ones with the biggest numbers of cases in the criminal report stage (see Exhibit 23). Those five were again POs located in Sarajevo, Banja Luka, Tuzla, Zenica, and Bihac. In addition to the group of five POs which processed major number of criminal reports, the evaluation team identified an additional group which processed a medium number of criminal reports. That group consisted of the following six POs located in: Mostar, Travnik, Siroki Brijeg, Bijeljina, East Sarajevo, and Doboj (see Exhibit 24). When the team compared figures among all 11 POs (both big and medium POs), the team concluded that there is not such a big difference in the number of cases in investigative phase, as was the case in processing criminal reports among them (as provided in Exhibit 25). This might indicate a noticeable number of criminal reports being rejected by the five POs identified as big ones in processing criminal reports. To probe on this, the evaluation team examined number of resolved versus rejected criminal reports among these five POs as Exhibit 26 shows. Exhibit 24: Number of solved and rejected criminal reports in 2017 for 11 POs with big and medium caseload Although, the team found similar patterns in all 11 POs, which is that large number of filed criminal reports were rejected, five big POs rejected more criminal reports relative to their number of resolved criminal reports, and that brought the group of big and medium caseload POs closer in the stage of investigations. Moreover, this resulted in PO Zenica and PO Bihac having the highest number of newly opened investigations in 2017, while PO Sarajevo and PO Banja Luka (POs with the greatest number of prosecutors) had fewer newly opened investigation in 2017 than PO Zenica and PO Bihac, as Exhibit 25 shows. Exhibit 26 provides data for number of solved, rejected criminal reports, and number of investigations opened in 2017, while the 2014 – 2016 figures have a similar pattern. Exhibit 25: Number of newly opened investigations in 2017 for 11 POs with big and medium caseload 68 Exhibit 26: Processing of investigations for corruption cases in 2014 – 2017 by cantonal/district POs 69 Analysis of data in indictment phase The evaluation team analyzed the number of indictments filed and confirmed from 2014 to 2017, which distinguished two groups of POs. The first filed a relatively large number of indictments, while the second filed a medium number of indictments. The team noted that the POs that filed the largest numbers of indictments annually changed from year to year. For example, PO Tuzla and PO Zenica led over this period of time, while the number of indictments filed by PO Sarajevo and Banja Luka was smaller in the same period. Furthermore, PO Bihac was getting to figures quite similar to those of PO Sarajevo and PO Banja Luka over the same period of time. The comparative graphical presentation of number of indictments filed and confirmed is provided in Exhibit 27. Analysis of data in judgment phase In terms of judgments for corruption related cases, PO Banja Luka led throughout the whole period of 2014 to 2017. Although PO Banja Luka is not the permanent lead in number of indictments filed, the result obtained in number of judgments could be related to the fact that corresponding courts have a backlog of these cases, and that judgements are rendered at a higher pace than number of indictments filed in the most recent one or two years. However, the evaluation team did not have the necessary data to confirm this assumption. Other POs maintained a stable ratio of indictments filed to convictions obtained, so in general, the two groups with big and medium caseloads largely remained the same in the adjudication phase as well. In general, there was a high level of convictions relative to acquittals for all POs in these two groups except PO Siroki Brijeg, which had a relatively high number of acquittals compared to number of convictions. 70 Exhibit 27: Indictments (filed and confirmed) for corruption cases in 2014–2017 by POs Individual performances of POs with big and medium caseload in 2014 – 2017 To provide a fully rounded examination of the individual performances of 11 POs with big and medium caseloads from 2014 to 2017, the evaluation team performed a time series analysis for each PO. Exhibits 28 to 38 provide individual key variables, as well as all variables in four stages of criminal procedure for corruption cases for POs with big and medium caseloads from 2014 to 2017. Observations on key variables are provided for each PO. 71 Group of five POs with big caseloads: Exhibit 28 provides data for Prosecutors’ Office of the Sarajevo Canton in four stages of criminal procedure of corruption cases. The only observable change in key variables is that the number of criminal reports was quite volatile and annual change went in different directions. The patterns of newly received criminal reports and number of rejected criminal reports remained unchanged from 2014 to 2017. The number of newly opened investigations, as well as indictments filed in 2017 was smaller than in 2015 and 2016. There was no observable change in the number of filed indictments and number of convictions from 2014 to 2017. Exhibit 28: Graphical output of processing corruption cases by PO Sarajevo 72 Exhibit 29 provides data for District Prosecutors’ Office in Banja Luka in four stages of criminal procedure of corruption cases. While there was no significant change in the pattern of the key variables, the team observed a negative change across all of them. The team attributed some changes to the establishment of the RSSPO and the possible transfer of cases, but that could explain just a portion of trends from 2016 onward. In all other stages, the numbers of newly opened investigations, indictments filed, and number of convictions declined from 2014 to 2017. Exhibit 29: Graphical output of processing corruption cases by PO Banja Luka 73 Exhibit 30 provides data for Prosecutors’ Office of the Tuzla Canton in four stages of criminal procedure of corruption cases. The evaluation team observed changes in key variables, and that the number of criminal reports was quite volatile and the annual change went in different directions. Prosecutors’ Office of the Tuzla Canton had an exceptionally good year in 2015 compared to other years in the observed period. With the exception of 2015 values, number of newly opened investigations, indictments filed, and number of convictions remained similar from 2014 to 2017. Exhibit 30: Graphical output of processing corruption cases by PO Tuzla 74 Exhibit 31 provides data for Prosecutors’ Office of the Zenica-Doboj Canton in four stages of criminal procedure of corruption cases. The team observed constant improvements in almost all variables that tracked processing corruption cases in all four stages of criminal procedure. Moreover, 2017 was the year with the best performance figures since 2014. In accordance with the administrative data for Prosecutors’ Office of the Zenica-Doboj Canton, the results achieved were different from the general pattern for the whole BiH; the team noted an individual improvement. Exhibit 31: Graphical output of processing corruption cases by PO Zenica 75 Exhibit 32 provides data for Prosecutors’ Office of the Una-Sana Canton in four stages of criminal procedure of corruption cases. The only observable change in key variables was that there was an outlier in the number of newly filed criminal reports in 2016. Prosecutors’ Office of the Una-Sana Canton had the best results in 2016 in terms of all key variables. Results in 2017 were very similar to their values in 2015. The general pattern in key variables in Prosecutors’ Office of the Una-Sana Canton did not differ from the general pattern for the whole BiH. Exhibit 32: Graphical output of processing corruption cases by PO Bihac 76 POs with a medium size case load Exhibit 33 provides data for Prosecutors’ Office of the Herzegovina-Neretva Canton in four stages of criminal procedure of corruption cases. The only observable change in key variables was that there was an outlier in the number of newly filed criminal reports and number of investigations opened in 2016. Prosecutors’ Office of the Herzegovina-Neretva Canton had mixed and volatile results from 2014 to 2017. The general pattern in key variables in Prosecutors’ Office of the Herzegovina-Neretva Canton did not differ from the general pattern for the whole BiH. Exhibit 33: Graphical output of processing corruption cases by PO Mostar 77 Exhibit 34 provides data for Prosecutors’ Office of the Central Bosnia Canton in four stages of criminal procedure of corruption cases. The team observed volatility in key variables for Prosecutors’ Office of the Central Bosnia Canton. The team observed changes in both positive and negative directions, and on an annual basis. Prosecutors’ Office of the Central Bosnia Canton’s general pattern of key variables to some extent followed the general pattern for all POs in BiH. Exhibit 34: Graphical output of processing corruption cases by PO Travnik 78 Exhibit 35 provides data for Prosecutors’ Office of the West Herzegovina Canton in four stages of criminal procedure of corruption cases. The team observed that there was an outlier in number of newly received criminal reports in 2015 and 2016, while that number returned back to its 2014 value in 2017. In terms of the number of newly opened investigations and indictments filed, 2017 values were the lowest in years 2014 to 2017. In comparison to the general BiH pattern of key variables, it could be noted that if investigations opened, in general they resulted in indictments filed at a higher rate in comparison to the general pattern for the whole of BiH. Exhibit 35: Graphical output of processing corruption cases by PO Siroki Brijeg 79 Exhibit 36 provides data for District Prosecutors’ Office in Bijeljina in four stages of criminal procedure of corruption cases. The team observed that there was an outlier in the number of newly received criminal reports in 2015. District Prosecutors’ Office in Bijeljina also had the highest values in almost all key variables in 2015. Except in conviction judgments, District Prosecutors’ Office in Bijeljina opened the lower numbers of investigations and filed fewer indictments than it in 2014. In comparison to the general BiH pattern of key variables, the number of convictions was higher than the number of indictments filed and similar to number of newly opened investigations, which indicated that the corresponding courts had a backlog of cases and the increased number of convictions resulted from the courts resolving those cases. Exhibit 36: Graphical output of processing corruption cases by PO Bijeljina 80 Exhibit 37 provides data for District Prosecutors’ Office in East Sarajevo in four stages of criminal procedure of corruption cases. The team observed that there was an outlier in number of newly received criminal reports in 2015. District Prosecutors’ Office in East Sarajevo also had the highest values in almost all key variables in 2015. District Prosecutors’ Office in East Sarajevo had a lower number of indictments filed and convictions from 2015 to 2017 than in 2014. The general pattern of key variables to some extent followed that of BiH, although the team observed a higher level of volatility and annual changes. Exhibit 37: Graphical output of processing corruption cases by PO East Sarajevo 81 Exhibit 38 provides data for District Prosecutors’ Office in Doboj in four stages of criminal procedure of corruption cases. District Prosecutors’ Office in Doboj experienced high volatility and changes of direction in key variables. While the PO opened the most investigations in 2015, the Prosecutors’ Office in Doboj filed the biggest number of indictments in 2017. In general, District Prosecutors’ Office in Doboj had the biggest mismatch relative to other POs presented in this Annex in the pattern of key variables from year to year from 2014 to 2017. Exhibit 38: Graphical output of processing corruption cases by PO Doboj 82 The evaluation team found after comparatively analyzing data among all cantonal and district POs, that individual cantonal and district POs follow a general pattern for the whole BiH as presented in this Annex. The only PO which steadily improved from 2014 to 2017, based on HJPC administrative data in all key variables, was PO Zenica. PO Tuzla had an exceptionally productive year in 2015, while PO Bihac had notably the most successful year in 2016 within the 2014 to 2017 period. In prevailing part of key variables, 2015 was the most successful year for PO Bijeljina and PO East Sarajevo in the 2014 to 2017 period. When mirroring these individual performances with the JA intervention, the evaluation team found that the JA PPP/POAP was implemented in 2015 in PO Zenica and that PO Zenica’s performances experienced continuation of good results in the following years. In the same year, the JA PPP/POAP was implemented in PO Tuzla, which experienced then its best year in terms of processing corruption cases, but PO Tuzla did not continue to improve its performances in the following years, as did PO Zenica. Similarly, PO East Sarajevo received the JA’s assistance and experienced its best year in 2015. On the contrary, PO Bihac and PO Bijeljina achieved their best results when they had not yet received the JA’s technical assistance. In making these conclusions, the evaluation team relied only on the HJPC administrative data. Because the BiH judiciary’s administrative data do not account for case weight or case complexity, the evaluation team similarly could not account for that information. Both the JA and the evaluation team use HJPC administrative data extracted in accordance with the HJPC Corruption Case List for compatibility and comparability of actuals. In parallel to the extraction model in accordance with the HJPC Corruption Case List, the HJPC uses an additional model. The additional model extracts corruption cases, which are registered in the KTK registry book. In order to examine if there were differences in observed trends and patterns presented in this Annex, the evaluation team collected and analyzed reported data extracted by the second model (the KTK registry book). The data extracted by this model are presented in Annex III. The evaluation team found that by using the second extraction model, the actual values for processing corruption cases were smaller which was explained by the discretional power of POs which cases from the HJPC Corruption Case List (in accordance with particular circumstances of each case) will register in the KTK registry book. On the other hand, the historical trends and patterns presented in this Annex were unchanged when the team analyzed data extracted by the HJPC second extraction model. 83 ANNEX III: HJPC DATA IN PROCESSING CORRUPTION RELATED CASES IN 2015-2017 BY KTK REGISTRY BOOK KTK 2015 2016 2017 Criminal Reports Received Criminal Reports 1133 1219 1061 Criminal Reports Resolved 1040 1204 1122 Criminal Reports Rejected (Investigations not opened) 520 661 614 Criminal Report disposed in other way 61 107 135 Unresolved CR on December 31st 812 825 738 Investigations Investigations opened 459 436 373 Investigations Resolved 453 409 393 Investigations terminated 198 198 141 Investigations disposed in other way 10 11 21 Unresolved Investigations on December 31st 183 210 189 Indictments Indictments filed (no of cases) 244 195 231 Indictments filed (no of persons) 351 282 353 Indictments filed up to 5 yrs. imprisonment sanction prescribed (persons) 240 179 237 Indictments filed up to 10 yrs. imprisonment sanction prescribed (persons) 90 64 75 Indictments filed over 10 yrs. imprisonment sanction prescribed (persons) 21 39 42 Convictions Convictions (no of cases) 112 204 216 Convictions (no of persons) 135 234 284 84 ANNEX IV: EVALUATION PURPOSE, DESIGN, AND LIMITATIONS IMPAQ International (IMPAQ) has been commissioned by USAID/BiH within the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct the performance evaluation of the USAID/BIH’s Justice Activity in Bosnia and Herzegovina (the JA). The main purpose of the performance evaluation was to provide USAID/BiH with an evidence-based and independent review of USAID/BiH’s the JA, and to produce a report that provides a qualitative as well as quantitative analysis of the Activity’s performance. MEASURE-BIH conducted a performance evaluation of the JA, which has been implemented by the Millennium DPI Partners, LLC under the Task Order (TO) No.: AID-168-TO-14-000001. The Activity started on September 23, 2014 and will continue to September 22, 2019. To fulfil its mandate MEASURE-BIH used a rigorous methodological approach in addressing the evaluation and assessment questions. The work conducted provides insights into progress towards the JA’s expected results. Based on this information, MEASURE-BiH provides recommendations for USAID/BiH, and the implementing partner of the JA activity for the remaining period of the Contract. The primary audience for this evaluation is USAID/BiH, who will use the findings, conclusions and recommendations for assessing the achievements of the JA and for informing further the JA interventions. The JA performance evaluation asked the following questions: 1. To what extent has the Activity design been appropriate in terms of chosen activities and beneficiaries needs? Are chosen activities vetted by beneficiaries and stakeholders? How was the Activity implemented in terms of sequencing, timing, resources allocation, flexibility and cooperation with beneficiaries and stakeholders? How was the work of the JA perceived and valued by beneficiaries and stakeholders? 2. To what extent the JA achieved expected results under both components as outlined by TO, and what are the prospects of meeting life of activity targets specified by the monitoring and evaluation plan? More specifically, was the individually tailored assistance to each PO appropriate and what results were achieved in the sense of increasing the overall quality of the work of prosecutors in BiH? Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 3. Within the current TO scope, what opportunities can be identified to additionally complement ongoing the JA interventions? What are the lessons learned from implementation of the current TO? The evaluation team used a mixed-method approach and relied on triangulation methods for data analysis process, combining different sources of information. Whenever possible, the evaluation team used data from the following sources: the JA implementation documentation, data and records; donors, State and HJPC / POs official documents; HJPC administrative data on cases being processed by courts and prosecutor offices, including data on processing corruption and economic crime cases in 2014 - 2017,; Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) from 2015 - 2017; Selected historical data from the MEASURE-BiH Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary in 2015 – 2017 and Semi-structured key informant interviews (KIIs). 85 Desk research of the JA implementation documentation included among other: Relevant parts of the JA’s contract (original and modifications); the JA reports (annual and quarterly); the JA annual work plans; the JA MEL plan including data and records (databases); PPPs and POAPs signed with POs; PPP and POAP progress reports; the JA MoU(s) signed during the life of the Activity; Other documents provided by the JA (i.e. drafts of Bench book, Handbooks, Guidelines, Manuals, Integrity Plans, Training Curricula and Agendas, Assessments, Studies, etc.). The MEASURE-BiH team also reviewed international and local documents relevant for assessment of BiH justice sector and issues relevant for the JA activities including: EU progress reports for BiH 2014 – 2017 (relevant sections); Transparency International National Integrity Systems in the Western Balkans and Turkey: Priorities for Reform (October 2016); European Parliament Anti-Corruption Efforts in the Western Balkans Briefing (April 2017), U.S. Department of State 2017 Investment Climate Statement (August 2017); OSCE Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases Project Report (February 2018) and the relevant Council of Europe Anti-corruption Body - The Group of States against Corruption (GRECO) reports issued in February 2016 – May 2018. The extensive list of documentation reviewed is contained in Annex III. Key informant interviews (KIIs) were conducted with: USAID/BIH, INL, OPDAT, and other donors (i.e. EC/BiH, EUSR, Swiss, Norway, Sweden, UK and Italy) MoJs, HJPC members and HJPC Secretariat representatives, ODC, Courts, POs, other government agencies (i.e. APIK, SAI, Agency for forfeiture of assets FBiH), Associations of Judges, Associations of Prosecutors and NGOs. A total of 55 KIIs were conducted with 107 informants participating in those interviews. Extensive list of all interviews held and participants present is provided in Annex X. Interviews were structured according to defined interview protocols in order to ensure objectivity, focus, consistency and comparability of responses. KII protocols are provided in Annex IX. All KIIs were conducted with a minimum of two team members present. The team used already available historical HJPC administrative data collected for the calculation of the JA MEL indicators and the USAID project level indicators. This data (covering processing of criminal reports, investigations, indictments, judgments and sanctions) helped identify trends and changes in processing corruption-related and economic crime cases by POs and courts in BiH in 2014-2017, a period which coincides with the JA implementation. To simplify presentation and to visualize the data, MEASURE-BIH used GIS technology and expertise on our disposal. Detailed presentation of data is in Annex II. In addition, the team used available historical HJPC administrative data collected for calculation of the Judicial Effectiveness Index of Bosnia and Herzegovina, which cover processing of major case types (including criminal, civil, commercial, administrative and enforcement cases) in basic and appellate courts and (criminal case types belonging to) prosecutor offices. These data helped identify general trends and bottle￾necks in processing cases in the BiH judiciary in 2014-2017. The team used two sources of survey data: The National Survey of Citizens’ Perceptions (NSCP) which is conducted in 2015-2017 by MEASURE-BIH, and The Survey of Judges and Prosecutors on Effectiveness of BiH Judiciary (SJP) conducted in 2015-2018. Both surveys cover topics and issues relevant for this evaluation and the assessment. The survey data were triangulated with KII and HJPC administrative data in order to provide a holistic picture and thorough answer to the evaluation and assessment questions. In order to address: Evaluation Question 1 – the team conducted a desk review of the JA’s planning and implementation documentation and databases (including MEL documentation), a review of secondary documentation relevant to the JA, such as documentation from other donors involved in the justice sector, as well as relevant documentation from BiH government/public institutions (e.g. HJPC, Ministries). This also included documentation related to interaction with beneficiaries i.e. conference/meeting agendas, conclusions, MoUs, etc. The team conducted KIIs with USAID/BiH staff involved in the JA management and the JA’s implementing partner. Furthermore, KIIs were conducted with other donors/international organizations 86 relevant for the justice sector. The data collected from these different sources were triangulated on the same topics corresponding to this question and related sub-questions. Evaluation Question 2 – in estimating achieved results of the JA as outlined in the contract, the team used several sources of data. In the first step, the team identified relevant questions from the National Survey of Citizen Perceptions’ and the Survey of judges and prosecutor questions and isolated related historical data from these surveys. In the second step, the team identified and extracted administrative data used in the JA MEL plan and administrative data used for calculating USAID/BIH project level indicators. Third, the team processed data obtained through KIIs and desk review findings. Finally, whenever possible data from these four sources were triangulated to answer the evaluation question and related sub-questions. Evaluation Question 3 – the same approach and four sources used to answer Evaluation Question 2 was also used to answer Evaluation Question 3. The main limitations of the evaluation include the possibility of social desirability bias, data contamination, lack of complete information, and recall bias. Social desirability bias: This type of response bias exists when key informants answer questions in a way that they feel will be viewed positively and may make the Activity look more favorable to the evaluation team. Therefore, the KIs that had intensive cooperation with the Activity may overstate the positive effects of the interventions and understate its negative effects. Data contamination: As other donor interventions and international organizations have been supporting the justice sector in similar areas of work (for example, the Swiss/Norwegian project), it may be challenging for respondents to isolate the JA’s contribution. In addition, the administrative data on processing corruption and economic crime cases show the collective results of interventions by all donors, and the efforts, or lack thereof, of domestic stakeholders; thus, it is not possible to isolate the JA’s singular contribution. Lack of complete information: Any error in the report that is a result of KIs inability or unwillingness to provide accurate answers. In many instances during the interview session, some of the respondents were unwilling to provide an answer or had no detailed knowledge about the Activity due to their tasks and roles (e.g. Deputy Chief Prosecutors, Assistants, Advisors) and therefore unable to provide an accurate answer. Recall bias: The JA’s implementation began in 2014 and respondents may not clearly remember its early activities. To mitigate these biases, we selected a broad range of stakeholders, both direct JA beneficiaries and other stakeholders in and around the BIH judiciary who were not exposed to JA interventions, to interview. We triangulated the respondents’ answers with historical data and Implementing Partner’s (IP) records. 87 ANNEX V: EVALUATION AND BRIEF ASSESSMENT STATEMENT OF WORK PERFORMANCE EVALUATION OF USAID’S JUSTICE ACTIVITY IN BOSNIA AND HERZEGOVINA (BIH) AND BRIEF ASSESSMENT OF BIH JUSTICE SECTOR Scope of Work (SoW) May 29, 2018 88 INTRODUCTION Name of Activity to be Evaluated: USAID’s Justice Activity in Bosnia and Herzegovina (the JA) Implementer: Millennium DPI Partners, LLC Award Number: IQC No.: AID-OAA-I-13-00029 Task Order No.: AID-168-TO-14-000001 (TO) Task Order Value: $ 9,500,000 Life of Activity: September 23, 2014 – September 22, 2019 Period to be Evaluated: September 23, 2014 – present PURPOSE The main purpose of a performance evaluation under this SoW is to provide USAID/ Bosnia and Herzegovina (BiH) with evidence-based and independent review of USAID/BiH’s JA, and to produce a report that provides a qualitative as well as quantitative analysis of the Activity’s performance. MEASURE-BIH will conduct a performance evaluation of the JA, which is implemented by the Millennium DPI Partners, LLC under the Task Order (TO) No.: AID-168-TO-14-000001. The Activity started on September 23, 2014 and will continue to September 22, 2019. Furthermore, under this SoW, the secondary purpose is to conduct a brief assessment to identify current issues and needs of BiH justice sector that can most effectively be met by USAID assistance and to make specific recommendations for potential future programming. The primary audience for this evaluation and brief assessment is USAID/BiH, who will use the findings, conclusions and recommendations for assessing the achievements of the JA and for informing further programming in the justice sector of BiH. THE JA BACKGROUND USAID/BiH’s the JA is designed to support USAID’s broader democracy and governance goal of achieving more functional and accountable institutions that meet BiH citizen needs (Development Objective [DO] 1) as expressed in USAID’s Country Development Cooperation Strategy (CDCS). The JA’s goal is to contribute to the USAID/BiH Intermediate Result (IR) and Project 1.1. - More effective judicial, executive and legislative branches of government and further USAID’s IR of making government more responsive to citizens by strengthening the capabilities of justice sector actors (IR1.1.1). The TO specifies the following key objectives and outcomes of the JA: 89  Preserving the independence of justice sector institutions through greater self-accountability (and external accountability mechanisms such as civil society organizations [CSOs])  Moving key justice sector institutions from the planning and standards-development stage to actual improved performance based upon results analysis  Strengthening the management and decision-making practices of the High Judicial and Prosecutorial Council of Bosnia and Herzegovina (HJPC) and Prosecutor Offices (POs) so that resources are allocated strategically to fight corruption and other serious crime  Supporting on-site knowledge and skills application by front-line prosecutors and police handling corruption cases so that they work as a team and network across jurisdictions  Building consensus for key reforms within the justice sector and standardizing cross-jurisdiction cooperation between state, entity, and local actors  Using diagnostic studies to inform major policy solutions that improve system-wide performance and lead to strategies that will effectively fight corruption  Increasing public confidence in the justice sector through professional regulation, accountability, and transparency THE JA DESIGN The Activity’s initial design was aligned with provisions of the TO, dated September 21, 2014, which includes three Activity components. Modification No.1 of the TO refocused the JA resources on the work with prosecutors and justice sector institutions to better combat corruption (Components I and 3 of the JA) and eliminate work related to the improvement of efficiency in the enforcement of judgments (Component 2). The original Component 3 was re-named as Component 2 after the Modification No.1 of the TO. Thus, the Activity now has two components: Component 1: Strengthening the professional status and performance of prosecutors (C1); and Component 2: Strengthening justice sector institutions to uphold public integrity/combat corruption (C2). Component 1 (C1) TO requirements for C1 In accordance with the TO, under C1 the Activity should have designed interventions to achieve the following five outcomes: 1.1 Strengthened organizational leadership, planning, and performance in prosecutor offices 1.2 Prosecutors perform functions more efficiently through balanced allocation of resources 1.3 Prosecutors uphold public trust and integrity through prosecution of corruption or other serious crime 1.4 Prosecutors provide appropriate and accurate information to citizens in order to strengthen transparency and responsiveness 90 1.5 Prosecutor status improved through performance appraisal, merit-based career advancement or incentives to prosecute cases In accordance with the contract, the JA implements the Prosecutor Partnership Program (PPP) directly in all 19 prosecutor offices to improve management and help achieve strategic objectives (Activity 1.1). In addition, the JA provides intensive, on-site, practice-based training and mentorship to senior managers and to front-line prosecutors, involving the direct application of skills to actual management issues and criminal cases (Activity 1.2). Through the PPP, the JA introduces office guidelines on case weighting and performance awards to incentivize prosecutors to prosecute major corruption and serious crime cases. The JA also helps prosecutor offices more efficiently process criminal cases by improving staffing and efficient work-flow procedures (Activity 1.3). Through Activity 1.4 the JA focuses on increased transparency of the work of prosecutor offices, public access to information, and public outreach (roundtables) to build trust in local communities. Additionally, through Activity 1.5 the JA involves both the HJPC and individual prosecutor offices in improving performance appraisals for prosecutors, with the aim of advancing prosecutors who are able to lead investigations and obtain convictions. Finally, the JA works with the HJPC to emphasize merit over ethnic quotas for purposes of appointment and career advancement of prosecutors (and judges). The JA reported status of activities in C1 To meet the TO’s objectives in C1, as described in the JA’s annual reports, the JA established the Prosecutor Partnership Program (PPP) with 4 groups of POs. The first was created in 2015 (4 POs), the second in 2016 (5 POs), the third in 2017 (5 POs) and the forth in 2018 (5 POs). In 2019, all POs in BiH (19) except PO BiH have signed the PPP with the JA. The JA develops Prosecutor Office Assistance Plans (POAPs) as PPPs are signed. POAPs are developed for all 4 groups of PPP POs. Following development of POAPs, the JA engages in POAPs implementation and coordinates activities with the HJPC Standing Committee (SC) in relation to POAP implementation, donor support, management policy, building the organizational and casework capacity of POs, and building counterpart consensus and understanding. In addition, the JA has conducted several studies/analyses and worked to implement their recommendations. The major studies/analysis are: an analysis of the system of engaging expert witnesses in cases of corruption and organized and economic crime, the audit report analysis, court verdicts in cases of corruption and the asset forfeiture analysis. Upon development and presentation, the JA works on implementation of Analysis with relevant stakeholders. The JA also completed the diagnostic study to determine the sources of corruption in the justice chain (DA) which revealed many shortcomings in the prosecution of corruption in BiH. The JA focuses on providing more specialized training aimed at building capacity for more efficient investigation and prosecution of cases of corruption and economic crime in accordance with DA findings. These activities should lead to strengthening organizational leadership, planning, and performance in prosecutor offices. The JA also provides substantial support to POs in strengthening institutional links between strategic plans and budgets, and operational planning and management. Through POAP implementation, the JA strengthens individual PO capacities in the areas of strategic management, organizational performance and, to some extent, financial management. This support should enable prosecutors to perform functions more efficiently through balanced allocation of resources. 91 The JA continuously deliver on-the-job training, workshops, and consultations to facilitate the exchange of best practices among partner POs. This technical assistance is provided by utilizing prosecutor-guided investigation (PGI) experts. The JA provides technical assistance in organizing and delivering trainings for three groups of prosecutors: 1) the group on corruption, 2) the group on economic and organized crime, and 3) the group on cybercrime. The JA supports POs in strengthening their public outreach to citizens. The JA and the Association of Spokespersons in POs in BiH organized an advanced training on the application of the Freedom of Information Act and worked with the Association on developing its own strategic framework and action plan for the coming years. The JA provided support for chief prosecutors to analyze the level of information available to the public about the POs’ achievement of their mandate and the results of their work. In addition, the JA provides support in the process of drafting the strategic and annual plans and standardizing access to case-related information on websites, using best practices in some partner POs. The JA worked actively with public information officers from all POs on developing PR MEL Plan, which was adopted by the HJPC Standing Committee on the Efficiency of POs in BiH in September 2017. It is anticipated that this tool will be used by the HJPC in the process of annual performance monitoring of all POs. The JA also conducted an analysis of the appointment and career advancement systems of several countries in Europe in order to assist the HJPC Standing Committee for Legislation (SCL) in identifying the best system for BiH. The analysis covered topics such as initial selection, trainings, advancement system, lateral and parallel transfers within different judicial offices, and the selection of candidates for managerial positions. The analysis was used by the HJPC SCL to develop a more informed position on changing the law of the HJPC. Component 2 (C2) TO requirements for C2 In accordance with the TO, under C2 the Activity should have designed activities to achieve six outcomes: 2.1 Prosecutors investigate and prosecute high profile corruption and economic crime cases free from political or improper influence 2.2 Increased cooperation among state, entity, and local justice and other sector actors 2.3 ODC is properly resourced to manage complaints procedures, autonomously reviews conduct of judges and prosecutors, and recommends appropriate sanctions 2.4 Disciplinary proceedings are processed expeditiously and fairly, and decisions are subject to independent and impartial review 2.5 Public trust in and respect for justice sector institutions are increased due to greater ability to act independently and impartially and be held accountable 2.6 Prosecutors and judges are trained in identifying elements of corrupt activities and investigate and prosecute corrupt practices Programming under Activity 2.1 to strengthen prosecutorial capacity to investigate and prosecute high profile cases of corruption and serious crime is largely executed through the PPP. In particular, the PPP 92 strengthens the capacity of the Republika Srpska Special Prosecutors Office for Organized Crime and Corruption (RSSPO) and the JA disseminates RSSPO best practices to other prosecutor offices. The JA works closely with the SC to build further consensus for strategies to fight corruption across jurisdictions. Activity 2.2 features a comprehensive diagnostic to analyze sources of corruption in the justice sector and other government sectors. It is followed by regional diagnostics that further the consensus process and cooperation on local levels. Activities 2.3 – 2.5 reinforce integrity in the judiciary through increased performance by the Office of Disciplinary Counsel (ODC) staff, more transparency in the disciplinary process, and effectively applying codes of conduct to all personnel in courts and prosecutor offices through the ODC and with linkages to BiH’s anticorruption agency and other government agencies. Activity 2.6 reinforces anti-corruption efforts under Activity 2.1, bringing both judges and prosecutors together to develop legal specialization in corruption and other complex criminal matters (organized crime, economic crime, money laundering, etc.), including developing practical manuals and sentencing guidelines applicable to the processing of these types of cases. The JA reported status of activities in C1 To meet the TO’s objectives in C2, as provided in the JA annual reports, the Activity has worked across both C1 and C2 to create preconditions for enabling prosecutors to investigate and prosecute high profile corruption and economic crime cases free from political or improper influence. the JA focuses its efforts on the promotion of the recommendations of the Diagnostic Analysis of the Integrity of the Judicial Sector in BiH and Potential Risks of Corruption or Unethical Conduct in Judiciary (DA) and on conducting a study of specific aspects of criminal proceedings (including asset forfeiture and the use of expert witnesses carried out by C1) towards increasing cooperation among state, entity, and local justice and other sector actors. The JA provided recommendations for improvements to the disciplinary case management system (DCMS) to ensure better usage, management, sharing, and protection of information, as well as training to ODC staff on internal and external communications, strategic planning, and on the application of the Disciplinary Sanctioning Guidelines (DSG). These activities should help ensure that ODC is properly resourced to manage complaints procedures, autonomously review conduct of judges and prosecutors, and recommend appropriate sanctions. In addition, the JA provides training to the HJPC disciplinary committees, ODC, and the HJPC Secretariat on the implementation and application of the Disciplinary Sanctioning Guidelines (DSG). Finally, to this end, the JA supports a wide range of activities related to the ODC communications strategy, including but not limited to digests of disciplinary decisions in-house, improving the annual reporting process of the ODC, proposing a new ODC logo to improve the ODC’s public image, and submitting written recommendations for improvement of the ODC website yet to be implemented by the HJPC and others. The JA helps ensure that disciplinary proceedings are processed expeditiously and fairly, and decisions are subject to independent and impartial review. Advocacy efforts of the JA, aimed at securing local ownership and the sustainability of the Judicial Discipline Bench book, resulted in the establishment of the HJPC Working Group on the Integrity and Accountability of Judges and Prosecutors (WGIA), which is tasked with developing the Bench book. In addition, the JA supports HJPC surveys of judges and prosecutors on their perception of the fairness and impartiality of disciplinary actions and on ethics training needs. The survey questions relate to the independence of the ODC and the fairness, impartiality, and transparency of the disciplinary process. The survey also addresses ethics training needs. The JA has instituted efforts to build public trust in and respect for justice sector institutions by increasing their ability to act independently and impartially and be held accountable. Namely, the JA assisted the 93 HJPC and entity Judicial and Prosecutorial Training Centers (JPTCs) in an ethics training needs assessment. Based on the assessment, the JA developed a draft ethics training program and draft chapters of the ethics handbook for judges and prosecutors on personal integrity, conflict of interest, and diligence. The JA also supported the HJPC Working Group on the Integrity and Accountability of Judges and Prosecutors (WGIA) in the development of an improved asset declaration form for judges and prosecutors. The JA, in partnership with the Association of Expert Associates and Advisors (AEA), developed of a model code of ethical standards for expert associates and advisers in the courts and POs of BiH. The Activity also supported the HJPC, working in partnership with the Agency for the Prevention of Corruption and Coordination of the Fight against Corruption (APIK), to organize the Justice Sector Annual Conference on Anti-corruption and Integrity. Besides members of the HJPC, its Secretariat, and the chief disciplinary prosecutor, the conferences were regularly attended by chief prosecutors, court presidents, judges, prosecutors, representatives of professional associations, lawyers, ministries of justice, the non-governmental organization sector, academia, and the media. The JA also supports the implementation of the APIK Book of Rules (BoR) on the processing of individual reports of suspected corruption and provides technical assistance to the HJPC in developing the framework for creating Integrity Plans in individual courts and POs in compliance with the Integrity Plan Guidelines. To this end, the JA provided an opinion on the Integrity Plans of over 100 courts and POs in BiH. Finally, the JA provided a specialized two-year training program, which was designed to help judges and prosecutors adjudicate corruption and economic crime more efficiently and effectively. The training focused on addressing investigation planning, the drafting of indictments and judgments, international legal assistance, prosecution challenges, and the adjudication of high-profile cases. Exhibit 1 shows the JA LogFrame with Level of Results/Results, Outcomes/Outputs, Inputs and Indicators across both the JA Components. 94 Exhibit 39: The JA Activity LogFrame Level of Result Outcome/Output Input Indicators Activity Goal Develop more effective judicial, executive, and legislative branches of government • Legislative Capacity Index (LCI) Score aggregate • World Justice Rule of Law Index • Nations In Transit (NIT) Local Democratic Governance Score Activity Purpose Create more effective, independent, and accountable justice sector actors • Judicial Effectiveness Index (JEI) Score • Percentage change in public confidence in the rule of law • Number of judges and judicial personnel trained with USG assistance • Number of government officials receiving USG-supported anti-corruption training Activity Sub-Purpose Strengthen the status and performance of POs/prosecutors 1.1 Score on POCM in POs that implement PPPs Strengthen the status and performance of POs/prosecutors Outcome/Output 1.1: Strengthened organizational leadership, planning, and performance in prosecutor offices Input 1.1: Prosecutor Offices Assessment Report; Development of the PPP and POAP; Implementation of PPP and POAP; Outreach 1.1.1 Number of convictions for corruption related crime in POs that implement PPPs 1.1.2 Number of indictments for corruption related crime in POs that implement PPPs 1.1.3 Conviction rate for corruption related crime in POs that implement PPPs Outcome/Output 1.2 Prosecutors perform functions more efficiently through balanced allocation of resources Input 1.2 Workflow and Business Process Assessment, Strengthening strategic organization performance of POs 1.2.1 Number of unresolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs 1.2.2 Percentage of resolved criminal reports for corruption and economic crime versus total number of cases in prosecuting offices that implement PPPs 1.2.3 Number of resolved criminal reports for corruption and economic crime in prosecuting offices that implement PPPs Outcome/Output 1.3 Prosecutors uphold public trust and integrity through prosecution of corruption or other serious crime Input 1.3 STTA and LTTA review of Prosecutors’ Guided Investigations (PGI), Formation of Joint Investigative Teams for corruption cases 1.3.1 Percentage of investigations for corruption and economic crime completed within the one calendar year in prosecuting offices that implement PPPs 1.3.2 Number of investigations for corruption and economic crime completed in one calendar year in prosecuting offices that implement PPPs 1.3.3 Number of ordered investigations for corruption and economic crime in one calendar year in prosecuting offices that implement PPPs 95 1.3.4 Number of unresolved investigations for corruption and economic crime in one calendar year in prosecuting offices that implement PPPs Outcome/Output 1.4 Prosecutors provide appropriate and accurate information to citizens in order to strengthen transparency and responsiveness Input 1.4 Assessment of POs public information procedures and practices, Training of POs staff on collecting and disseminating information on corruption cases, monitoring and evaluation, Development of public information guidelines for POs 1.4.1 Number of POs that regularly disseminate information to the public through their websites, including appropriate information on corruption and economic cases in POs that implement PPPs 1.4.2 Number of press releases (or other types of communication channels) issued to the public including information on investigation and prosecution for corruption and economic crime cases in POs that implement PPPs 1.4.3 Percentage of public that agree that BiH POs adequately inform public about their work Outcome/Output 1.5 Prosecutor status improved through performance appraisal, merit-based career advancement or incentives to prosecute cases Input 1.5 Assessment of performance evaluation in POs, Strengthening the HJPC appointment and career advancement system, Roundtable Session 1.5.1 Number of PPP POs with score 4 or 5 on the JA Prosecutors Office Capacity Matrix (POCM) for appraisal process dimension Justice Sector Institutions strengthened to combat corruption Justice Sector Institutions strengthened to combat corruption • Score in Judicial Efficiency Index (JEI) – dimension of justice sector institutions’ capacity to fight corruption • Percentage change in public perception of justice sector institutions’ capabilities and performance to fight corruption Outcome/Output 2.1 Prosecutors investigate and prosecute high profile corruption and economic crime cases free from political or improper influence • See performance indicators 1.1 to 1.3.4 Outcome/Output 2.2 Increased cooperation and coordination among state, entity, and cantonal law enforcement and justice sector institutions leads to successful investigation and prosecution of corrupt cases Input 2.2 Improved judicial statistics on corruption cases and their exchange with law enforcement agencies, Established mechanisms of cooperation between APIK and prosecutor offices, The judicial annual conference on anticorruption 2.2.1 Number of collaborative mechanisms established and/or improved among justice sector institutions and other parties involved in anti￾corruption efforts Outcome/Output 2.3 The ODC is properly resourced to manage complaint procedures and Input 2.3 ODC assessment of operations, Guidelines for disciplinary sanctions development, Training 2.3.1 Number of procedural improvements implemented in ODC performance in disciplinary matters 96 autonomously review the conduct of judges and prosecutors and recommends appropriate sanctions for unethical conduct and corruption Outcome/Output 2.4 Disciplinary proceedings are processed expeditiously and fairly, and disciplinary decisions are subject to independent and impartial review Input 2.4 Survey of judges and prosecutor, Study of disciplinary case precedent 2.4.1 Judicial Discipline Bench-book and Ethics Handbook that will enable ODC staff and disciplinary committees, to conduct proceedings more expeditiously and with greater consistency (fairness) developed 2.4.2 Percentage of judges and prosecutors who strongly agree with statement that disciplinary process is fair and impartial Outcome/Output 2.5 Public trust and respect for justice sector institutions/actors are increased with respect to their demonstrated ability to act independently and impartially and to be held accountable Input 2.5 Review of ethics standards for judges, prosecutors and other stakeholders, Improved public perception of the judiciary through strategic communication of the ODC 2.5.1 Percentage of public that strongly agree and mostly agree with statement that judiciary is not vulnerable to corruption 2.5.2 Number of codes of conduct revised for improved standards in judicial conduct 2.5.3 Number of Courts and POs that adopted Integrity Plans Outcome/Output 2.6 Prosecutors and courts are trained in identifying elements of corrupt activities and are able to investigate and prosecute corrupt practices and prosecute corrupt practices and enforce the law against offenders Input 2.6 Training needs assessment, Long-Term Training 2.6.1 Number of judges and prosecutors who earned the certificate of specialized training in the investigation prosecution and adjudication of corruption and financial crime cases USAID STANDARD FOREIGN ASSISTANCE INDICATORS DR 1.3-1 Number of judges and judicial personnel trained with USG Assistance DR 2.4-1 Number of government officials receiving USG-supported anti￾corruption training 97 EVALUATION AND BRIEF ASSESSMENT QUESTIONS Illustrative evaluation questions addressed by this performance evaluation are listed below: 1. To what extent has the Activity design been appropriate in terms of chosen activities and beneficiaries needs? Are chosen activities vetted by beneficiaries and stakeholders? How was the Activity implemented in terms of sequencing, timing, resources allocation, flexibility and cooperation with beneficiaries and stakeholders? How was the work of the JA perceived and valued by beneficiaries and stakeholders? 2. To what extent the JA achieved expected results under both components as outlined by TO, and what are the prospects of meeting life of activity targets specified by the monitoring and evaluation plan? More specifically, was the individually tailored assistance to each PO appropriate and what results were achieved in the sense of increasing the overall quality of the work of prosecutors in BiH? Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 3. Within the current TO scope, what opportunities can be identified to additionally complement ongoing the JA interventions? What are the lessons learned from implementation of the current TO? The brief assessment will address the following questions: 1. What is the current context under which justice sector operates? How is the current context different from the one identified at the commencement of the JA? 2. What are the current most pressing issues in justice sector in BiH? How can they be addressed? To what extent are these issues addressed by current projects/interventions implemented by international and / or local organizations? 3. What are identified gaps and windows of opportunity in terms of needs for further technical assistance? What are recommendations to the Mission in terms of further programming in justice sector? 4. Is there a need or potential benefit of continuing direct assistance to the HJPC in some of the significant areas of HJPC’s mandate or were these sufficiently improved over the past decade? 98 DESIGN AND METHODOLOGY The Evaluation Team’s proposed methodology will ensure systematic and efficient collection of data and triangulation of information across different sources and methods. Evaluation Question Data Source Methodology 1. To what extent has the Activity design been appropriate in terms of chosen activities and beneficiaries needs? Are chosen activities vetted by beneficiaries and stakeholders? How was the Activity implemented in terms of sequencing, timing, resources allocation, flexibility and cooperation with beneficiaries and stakeholders? How was the work of the JA perceived and valued by beneficiaries and stakeholders? - the JA implementation documentation, data and records - Semi-structured key informant interviews (KIIs) with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, POs, law enforcement agencies, Associations of Prosecutors and NGOs Mixed Method Triangulation 2. To what extent the JA achieved expected results under both components as outlined by TO, and what are the prospects of meeting life of activity targets specified by the monitoring and evaluation plan? - the JA implementation documentation, data and records - Donor, State and HJPC / POs official documents relevant for the JA activities - HJPC administrative data relevant for the JA activities - Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) - Selected historical data from the Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary Semi-structured KIIs with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, POs, Law enforcement agencies, Associations of Prosecutors and NGOs Mixed Method Triangulation 3. Within the current TO scope, what opportunities can be identified to additionally complement ongoing the JA interventions? What are the lessons learned from implementation of the current TO? - the JA implementation documentation, data and records - Donor, State and HJPC / POs official documents relevant for the JA activities - HJPC administrative data relevant for the JA activities Mixed Method Triangulation 99 - Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) - Selected historical data from the Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary Semi-structured KIIs with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, POs, Law enforcement agencies, Associations of Prosecutors and NGOs Assessment Question Data Source Methodology 1. What is the current context under which justice sector operates? How is the current context different from the one identified at commencement of the JA? - HJPC, MoJs, International Organizations and NGO documents - HJPC administrative data relevant for work of courts and prosecutor offices - Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) - Selected historical data from the Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary - Semi-structured key informant interviews (KIIs) with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, Courts, POs, law enforcement and other government agencies, Associations of Judges, Associations of Prosecutors and NGOs Mixed Method Triangulation 2. What are the current most pressing issues in justice sector in BiH? How can they be addressed? To what extent are these issues addressed by current projects/interventions implemented by international and / or local organizations? - HJPC, MoJs, International Organizations and NGO documents (secondary data) - HJPC administrative data relevant for work of courts and prosecutor offices Mixed Method Triangulation 100 - Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) - Selected historical data from the Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary - Semi-structured key informant interviews (KIIs) with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, Courts, POs, law enforcement and other government agencies, Associations of Judges, Associations of Prosecutors and NGOs 3. What are identified gaps in terms of the needs for further technical assistance? What are recommendations to the Mission in terms of further programming in justice sector? - HJPC, MoJs, International Organizations and NGO documents (secondary data) - HJPC administrative data relevant for work of courts and prosecutor offices - Selected historical survey data from MEASURE-BiH National Survey of Citizen Perceptions (NSCP-BiH) - Selected historical data from the Survey of Judges and Prosecutors in BiH on effectiveness of BiH judiciary - Semi-structured key informant interviews (KIIs) with: USAID/BIH (possibly other US agencies) and other donors, MoJs, HJPC, Courts, POs, law enforcement and other government agencies, Associations of Judges, Associations of Prosecutors and NGOs Mixed Method Triangulation To support the team’s initial desk study, USAID/BiH and/or the JA will provide electronic copies to MEASURE-BIH of all documents to be reviewed. These include: Activity’s relevant parts of the contract (original and modifications); Activity’s reports (annual and quarterly) and annual work plans; PPPs and POAPs signed with POs, PPP and POAP progress reports, the JA MoU(s) signed during the life of the Activity, Activity’s MEL plan, other relevant documents from donor and official local government institutions, and any other piece of documentation relevant for this evaluation. During the data collection, the evaluation team will conduct semi-structured KIIs with the beneficiaries, and important local and international stakeholders in the BiH justice sector relevant for the JA. It is expected that KIIs will be performed with the following stakeholders and beneficiaries: - USAID/BiH, USAID/BiH Implementing Partner(s) and other US Agencies: (USAID/BiH; the JA; Anti￾Corruption Civic Organizations' Unified Network (ACCOUNT): Strengthening Governing 101 Institutions and Processes (SGIP) with regards to anti-corruption assistance provided to the judicial sector; Independent Media Empowerment Program (IMEP); and as per further USAID instruction: US Embassy and/or other US Agencies operating in BiH (i.e. Department of Justice Office of Overseas Prosecutorial Development Assistance and Training (OPDAT), The International Criminal Investigative Training Assistance Program (ICITAP)). - USAID the JA direct beneficiaries and stakeholders: High Judicial and Prosecutorial Council of BiH (Council/Presidency, ODC, HJPC Secretariat, HJPC Standing Committees and Work Groups representatives); All POs in BiH (represented by the Chief Prosecutor, a Prosecutor involved in the JA activities, PR and MEL Officer), the JA partner courts, Agency for Prevention of Corruption in BiH (APIK), FBiH Agency for Forfeiture of Illegally Obtained Assets, Law enforcement agencies participating in joint investigative teams and relevant Ministries of Interior and Ministries of Justice; Supreme Audit Institutions; Judicial Training Centers; Associations of Prosecutors BiH, Association of Expert Associates and Advisors and Association of Spokespersons in POs in BiH; FBiH and RS Supreme Courts; Association of Women Judges in BiH. - International organizations in BiH involved in judicial sector reforms: EU/EC Mission in BiH; OSCE; Swiss Embassy / Swiss Development Agency; UNDP, Regional Anti-corruption Initiative (RAI) - Local NGOs/CSOs, media, and/or justice/corruption experts involved or operating in the justice sector (e.g., Analitika, Tender, Center for Investigative Reporting, MediaCenter, Whistle Blower Association), including the relevant issue-based coalitions supported though the USAID/BiH Civil Society Sustainability Project (CSSP) e.g. lead members of the Justice and, Anti-corruption networks. To provide answers to assessment questions within this SoW, MEASURE-BIH will organize additional interviews with 1st / 2nd / Supreme instance courts (court presidents, judges and support staff) in Sarajevo, Banja Luka, Tuzla, Zenica and Mostar, Court and PO of BiH, HJPC bodies in charge of court efficiency, extended interviews with HJPC and MOJs, donor projects operating through HJPC in justice sector (Norway, Sweden, The Netherlands) and the Ministry of Security BIH. The evaluation team will conduct approximately 60 KIIs with representatives from the above-mentioned stakeholder groups. The evaluation work plan will contain a precise list of KIIs and sites to be visited. The evaluation team will use already available historical HJPC administrative data collected for the calculation of the JA MEL indicators and the USAID project level indicators, as well as other historical administrative data on work performance of courts and prosecutor offices. The evaluation team will use already available historical survey data on perception of citizens and perception of judges and prosecutors on effectiveness of BiH judiciary. DELIVERABLES AND SCHEDULE The deliverables will include: 1. Detailed evaluation work plan and data collection instrument(s) The evaluation work plan will include: (1) a detailed evaluation design matrix (including the key questions, methods, and data sources used to address each question and the data analysis plan for each question); (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. The work plan will include: (1) the anticipated 102 schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. 2. Presentation of preliminary findings Presentation of preliminary findings to USAID/BiH shall discuss the summary of preliminary findings and recommendations to USAID/BiH. 3. Draft evaluation report Draft evaluation report will be consistent with the USAID Evaluation Report Requirements provided in ADS REFERENCE 201MAH ( https://www.usaid.gov/ads/policy/200/201mah ) and take into account criteria to ensure the quality of the evaluation report specified in ADS REFERENCE 201MAA ( https://www.usaid.gov/ads/policy/200/201maa ). Once the initial draft evaluation report is submitted, USAID/BiH will have 10 calendar days in which to review and comment on the initial draft, and submit the consolidated comments to the evaluation team. The evaluation team will address the consolidated comments and submit a revised final draft report in 10 days hence. 4. Final evaluation report The evaluation team will be asked to take no more than 10 calendar days to respond/incorporate the final comments from USAID/BiH. The evaluation team leader will then submit the final report. The tentative schedule is as follows: Timeframe (2018) Task May 31 SoW approved and forwarded to MEASURE-BiH May 29 – June 7 Desk research; Development of data collection instrument; Identification of available administrative data; Identification of available survey data; Identification of STTA team members, if needed Drafting evaluation plan June 8 Submitting evaluation plan for USAID/BIH approval (modifications of the plan until June 18, if needed) June 11 USAID/BiH approval of the evaluation plan granted June 11 – June 14 Scheduling of meetings, engagement of transcribers and other logistics preparations June 18, 2018 In-briefing with USAID/BiH June 18/19, 2018 Meeting with the JA implementing partner June 18 – July 19 Data collection (Option: the Evaluation team should extend invitation to USAID/BiH to have a mid-term briefing, which will be subject of further USAID/BIH acceptance) July 20 Exit briefing with USAID/BiH July 23 – August 9 Finalization of transcribing Finalization of data analysis Preparation of presentation with preliminary findings Defining outline of the evaluation report August 10 Presentation of preliminary findings to USAID/BiH 103 August 13 – August 24 Drafting evaluation report August 27 – August 29 HO review of draft evaluation report August 30 – September 3 USAID/BIH review of draft evaluation report September 3 – September 10 Addressing USAID/BiH comments and final report editing September 11 Delivering final evaluation report to the Mission TENTATIVE TEAM COMPOSITION AND KEY QUALIFICATIONS Tentative team composition and team members’ key qualifications are estimated as follows: Position Key Qualifications Team Lead (TL) Team and project management skills; Adequate level of seniority to conduct interviews with highly ranked judicial and government officials and subject matter expertise in rule of law (RoL) or expertise in evaluation methodologies Legal Expert (LE) Subject matter expertise in RoL and acquaintance with ongoing activities and reforms in BiH justice sector Senior Research Analyst (SRA) Subject matter expertise in program evaluation, qualitative and quantitative research methods Research Analyst (RA) Ability to ensure smooth process of data collection and processing at a junior level position HO/FO support HO/FO support should ensure limited amount of additional expertise needed for completion of the evaluation which individual team members lack The team composition and level of effort will be finalized in the Evaluation Work Plan. 104 ANNEX VI: DOCUMENTS REVIEWED # Title 1 Advanced specialized training for prosecutors - Sample agenda (Financial and forensic investigation and confiscation of property gains obtained through criminal offense - current challenges and good practice) 2 Agenda - The Collegium of Chief Prosecutors of FBiH (Konjic, December 14 - 15, 2017) 3 Analysis of statistical data for processing acts of corruption and economic crime - Cantonal prosecutor's office of Tuzla Canton (Tuzla) 4 Analysis of the functionality of the prosecutor case management system TCMS 5 Analysis of the Use of Audit Findings in Prosecutors' Offices for the purpose of detection and prosecution of criminal offenses (main findings and recommendations) 6 List of international standards (Annex 6) 7 Annual conference of prosecutors - Sample agenda (IX Conference of Prosecutors in BiH) 8 Assessment of Case management system (CMS) of the Disciplinary Prosecutor's Office 9 Assessment of Performance Evaluation in Prosecutor Offices 10 Assessment of the training needs to improve the investigation, prosecution and adjudication of corruption and economic crime cases in BiH 11 BiH Anti-Corruption Strategy for 2015 - 2019 and Action Plan for the implementation of BiH Anti-Corruption Strategy for 2015 – 2019 12 Chapter VIIIa - Disclosure statements on finances and interests 13 Conclusions from a seminar organized for chief prosecutors and public relation officers of the prosecutors' offices in BiH: Informing the Public about Complex Cases - The Challenges and Good Practices 14 Conclusions from the conference „The development path of judicial reform - the scope and challenges“ 15 Consultative meeting on preliminary findings and recommendations of the Analysis of the Use of Audit Findings in Prosecutors' Offices for the purpose of detection and prosecution of criminal offenses, Agenda 16 Corruption prevention in respect of members of parliament, judges and prosecutors, Compliance Report for BiH 17 Corruption prevention in respect of members of parliament, judges and prosecutors, Evaluation Report for BiH 18 Decision on the establishment of Working group for identifying risk factors for integrity violation in BiH judicial institutions 19 Diagnostic Analysis of the Integrity of the Judicial Sector in BiH and Potential Risks of Corruption and Unethical Conduct in Judiciary 20 Diagnostic Assessment of the Enforcement Regime of Final Civil Claims in BiH 21 Draft Agreement on mutual consent to establish disciplinary liability 22 Draft conclusions from the Round table on "Executive procedure - Options for improvement", Banja Luka, May 26, 2015 23 Draft Ethics Codebook 24 Draft Form/Checklist for the first instance commission disciplinary decision 25 Draft Judicial Desk Benchbook 105 26 Draft Rulebook on the submission, verification and processing of financial statements of judges and prosecutors 27 EC Bosnia and Herzegovina 2014, 2015, 2016 and 2018 Progress Reports 28 E-mail: Cooperation with Standing Committee for POs Efficiency BiH 29 European Commission for Democracy through Law (VENICE COMMISSION) Opinion on legal certainty and the independence of the judiciary in Bosnia and Herzegovina 30 European Commission Staff Working Document: BiH 2015 Report, Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, EU Enlargement Strategy 31 European Commission Staff Working Document: BiH 2016 Report, Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, 2016 Communication on EU Enlargement Policy 32 European Commission's recommendations based on expert assessments of the disciplinary procedures in BiH judiciary 33 European Commission's recommendations based on expert assessments of the financial declarations of judges and prosecutors in BiH judiciary 34 European Parliament Anti-Corruption Efforts in the Western Balkans Briefing (April 2017) 35 European Union, Delegation to BiH Letter to HJPC on integrity 36 Expert report - Support to the PO in Mostar - Budgetary planning 37 Expert scope of work for developing the impact analysis of existing structure of prosecution system in BiH on the efficiency of prosecutors' offices and making recommendations for its improvement 38 Final conclusions of EU Round-Table on Fight against Corruption and Organized Crime from May 2018 including many the JA proposed recommendations 39 Financial statement of judges and prosecutors form 40 Functional Review of ODC 41 Functional Review of ODC - Presentation for the HJPC (November 2015) 42 GRECO Fourth Evaluation Round Corruption prevention in respect of members of parliament, judges and prosecutors COMPLIANCE REPORT BOSNIA AND HERZEGOVINA (February 2016) 43 GRECO Fourth Evaluation Round Corruption prevention in respect of members of parliament, judges and prosecutors COMPLIANCE REPORT BOSNIA AND HERZEGOVINA (May 2018) 44 GRECO Third Evaluation Round Fourth Interim Compliance Report on Bosnia and Herzegovina (June 2017) 45 HJPC Action Plan 12/2017 - 6/2018 46 HJPC BiH Annual Report 2014, 2015, 2016, 2017 47 HJPC Guidelines for determining disciplinary measures 48 HJPC Guidelines for drafting & implementing Integrity Plans in the judicial institutions of BiH 49 HJPC Guidelines for the prevention of conflict of interest in the judiciary 50 HJPC Letter of Support and Coordination with the JA 106 51 HJPC Rules on Applying BOR on Orientation Criteria for Measuring Work of Prosecutors 52 Identification and exchange of good management practices for processing cases in partnering prosecutors' offices 53 II Advice from a minor offence area - Comparative analysis of misdemeanor legislation in BiH and the region, Proceedings 54 Information on the undertaken activities within USAID Justice Activity in BiH 55 Informing the Public about Complex Cases - The Challenges and Good Practices, Agenda 56 Integrity plan for Municipal Court in Sarajevo 57 Invitation and Agenda of EU Roundtable on Criminal Procedure and Peer review on corruption 31 May-1 June 18 58 Invitation to a consultative meeting on draft Analysis of the Use of Audit Findings in Prosecutors' Offices for the purpose of detection and prosecution of criminal offenses 59 Invitation to a consultative meeting on preliminary findings and recommendations of the Analysis of the Use of Audit Findings in Prosecutors' Offices for the purpose of detection and prosecution of criminal offenses 60 Invitation to attend the HJPC BiH session 61 Joint Letter to the JPTC RS by Association of Prosecutors of RS and the JA on Round Table on new Criminal Law RS 2018 62 Justice Sector Reform Strategy for BiH 2014-2018 63 Law on HJPC of BiH - Amendments - USAID the JA Proposal 64 Letter from EU to DEI referring to the JA work (June 2018) and FINAL RECOMM from the PR Roundtable in May 65 Letter from PO Trebinje requesting support from the JA 66 Letter on Activities conducted within USAID Justice Activity in BiH (February 13, 2015) 67 Letter on Follow up on SDJ Conclusion - re: peer review reporting 68 Letter on Proposal for cooperation on conducting expert discussion on assessing the work of judges and prosecutors by insight in BiH and regional practice 69 Letter on Realization of the undertaken activities related to the improvement of the disciplinary procedure and the proposal for continuation of support in this area 70 Letter on Report on USAID Justice Activity in BiH activities in the period from October 2014 to March 2016 71 Letter on Request for expanding cooperation between USAID Justice Activity in BiH and HJPC BiH 72 Letter on the Request for joint organization of third joint EUSR and HJPC BiH Conference 73 Letter on Visit of representatives of the HJPC of BiH and USAID to justice sector institutions in RS 74 List of contacts of the USAID the JA 75 Meeting minutes from the Extended Collegium of Chief Prosecutors of FBiH 76 Meeting minutes from the Extended Collegium of Chief Prosecutors of FBiH (September 14 and 15, 2017) 77 Meeting Notes - Cooperation with the HJPC Standing Committee for Efficiency of POs (February 9. 2017) 78 Meeting Notes with EUSR 79 Meeting Notes with Swiss/Norwegian project (January 14. 2015) 107 80 Meeting with FBiH Minister of Justice 81 Memorandum of Understanding and Cooperation between HJPC of BiH, Agency for Prevention of Corruption and Coordination of the Fight against Corruption and USAID through its the JA in BiH 82 Monitoring and evaluation - main aspects, Cantonal prosecutor's office of Canton Tuzla (Tuzla, September 23, 2015) 83 Monitoring and Evaluation Plan of prosecutors' offices' public relations in BiH 84 OSCE Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases Project Report (February 2018) 85 Pilot Court Program & Standards 86 Planning and preparing budgets (first workshop), Agenda 87 Presentation of the Diagnostic Assessment on System of Enforcement of Final Claims in BiH 88 Presentation of the Diagnostic Assessment on System of Enforcement of Final Claims in BiH, Agenda 89 Professional Capacity of Judges, Legal Associates, and Court Bailiffs Training, Mentoring, and Resource Material 90 Prosecutor Partnership Program between Cantonal Prosecutor's Office of Una-Sana Canton and USAID Justice Activity in BiH 91 Prosecutor Partnership Program between Cantonal Prosecutor's Office of Tuzla Canton and USAID Justice Activity in BiH 92 Prosecutor Partnership Program between District Prosecutor's Office Banja Luka and USAID Justice Activity in BiH 93 Prosecutor Partnership Program between Federal Prosecutor's Office of the FBiH and USAID Justice Activity in BiH 94 Prosecutor's Office Assistance Plan for the Cantonal Prosecutor's Office of Herzegovina￾Neretva Canton 95 Prosecutor's Office Assistance Plan for the Cantonal Prosecutor's Office of Tuzla Canton 96 Prosecutor's Office Assistance Plan for the District Prosecutor's Office in Banja Luka 97 Prosecutor's Office Assistance Plan for the District Prosecutor's Office in Bijeljina 98 Prosecutor's Office Assistance Plan for the District Prosecutor's Office in Doboj 99 Prosecutor's Office Assistance Plan for the District Prosecutor's Office in East Sarajevo 100 Prosecutor's Office Assistance Plan for the Special Department for Organized Crime, Economic Crime and Corruption at the Republic Prosecutor's Office of RS 101 Report by Prosecutor Forensic Investigation (Financial) Expert for District PO in Doboj 102 Review and recommendations for improvement of judicial statistics on corruption cases and their exchange with law enforcement agencies 103 Review of the situation in the partner prosecutors' offices conducted for the purpose of collecting the information necessary for the preparation of Prosecutor's Office Assistance Plans - Cantonal Prosecutor's Office of Tuzla Canton - March 2015 104 Round table "Executive procedure - Options for improvement" – Agenda 105 Sample letter to JPTCs 106 Score on Prosecutor’s Office Capacity Matrix (POCM) in prosecutors’ offices that implement Prosecutor Partnership Programs (PPPs) 107 Selection of prosecutors' offices for participation in the Prosecutor Partnership Program in first year of Activity implementation 108 108 Seminar for judges and prosecutors: Ethical standards focusing on prevention of conflict of interest in BiH judiciary, Agenda 109 Short overview of discussion, conclusions and recommendations from the consultative meeting on preliminary findings and recommendations of the Analysis of the Use of Audit Findings in Prosecutors' Offices for the purpose of detection and prosecution of criminal offenses 110 Short overview of the Cantonal prosecutor's office of Tuzla Canton website (Tuzla) 111 Strategic Framework of prosecutors' offices in FBiH for the period 2018 – 2020 112 Strengthening judicial institutions to combat corruption and organized crime - Presentation for the HJPC's Standing Committee for Prosecutorial Efficiency (July 2, 2017) 113 Timeliness of disciplinary proceedings 114 Transparency International National Integrity Systems in the Western Balkans and Turkey: Priorities for Reform (October 2016) 115 U.S. Department of State 2017 Investment Climate Statement (August 2017) 116 USAID the JA Annual Reports for FY 2015, 2016 and 2017 117 USAID the JA BiH - All Events and Trainings as of June 5. 2018 118 USAID the JA MEL Plan Modification 4 119 USAID the JA Work Plan for FY 2015, 2016, 2017 and 2018 120 USAID Justice Activity in BiH - Presentation for the Standing Committee for Prosecutorial Efficiency (Sarajevo, June 24, 2015) 121 USAID MEASURE Judicial Effectiveness Index of Bosnia and Herzegovina 2015, 2016 and 2017 122 USAID Quarterly report for Q1 and Q2 FY 2018 123 Working document of Commission members - Report on BiH progress in 2014, Attachment to the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and Committee of the Regions, Enlargement Strategy and Main Challenges 2014-2015 (unofficial translation) 124 Workshop on Decisions in disciplinary procedures and disciplinary measures, Agenda 109 ANNEX VII: RECOMMENDATIONS RELATED TO CRIMINAL PROCEDURE (INCLUDING ASSET FORFEITURE) STEMMING FROM THE JA ANALYSES AND ASSESSMENTS WHICH WERE ALSO REFLECTED IN THE EU ADOPTED DOCUMENTS (PEER REVIEWS AND TAIEX RECOMMENDATIONS) USAID JA EU Diagnostic Analysis on the Integrity of the Judicial Sector in BiH and Potential Risk of Corruption or Unethical Conduct in Judiciary (DA) from October 2015: Chapter: Cooperation of Law Enforcement Agencies and Prosecutor’s Offices: Recommendation:  Analyze the implementation of the Instruction on conduct and cooperation of police officers and prosecutors and identify sections of this document which have not been implemented, provide continued training, and evaluate previous trainings with the view of improving the available and introducing of new trainings, as appropriate. EU Peer Review Expert’s Report on corruption, organized crime and money laundering/criminal procedure (Peer review mission on criminal procedure) from October 2017: It is recommended:  In order to improve synergy between police and prosecutors, that police will be more involved in the investigations. A much higher level of interaction between Prosecutors and Police is necessary. Joint conclusions of EU, OSCE and the JA from the TAIEX roundtable on enhancing the fight against corruption, organized crime, including money laundering and encompassing the entire rule of law chain ( Complemented with recommendations issued following the Peer review mission on criminal procedure) (TAIEX Conclusions) (June 2018): Discussion topic 2): 1) Improve regular cooperation between prosecutors and law enforcement agencies through joint investigation teams The Analysis of the System of Forfeiture of Proceeds of Crime in BiH (Asset Forfeiture Analysis) from July 2017:  Efforts should continue on allowing direct access for prosecutors and law enforcement in BiH to data bases of selected agencies, bodies and organizations in possession of information on property. It is recommended that cooperation agreements be concluded with certain bodies possessing information on property, as well as focal points appointed for the benefit of more expedient and efficient exchange of information during financial investigations.  Amendments to banking regulations should be considered, in order to provide for exceptions from the banking secret in case of prosecutorial requests for information. This would eliminate the need for a court order and thus expedite the process of data collection.  A practice of provisional prohibiting measures regarding transfer and disposal of unregistered real-estate should be promoted in BiH, through introduction of a type of urgent procedure of Peer review mission on criminal procedure from October 2017:  It is recommended to improve cooperation between Prosecution Offices, police, tax administration, customs, FIU and financial regulators. Electronically accessible data bases (company registers, real estate registers and bank accounts …) have to be established at all levels and mutual access for all law enforcement entities should be guaranteed. Urgent steps aiming at establishing an integrated platform through which competent enquiring bodies may be granted - in accordance with their legal mandate - direct access to a wide range of disconnected national systems are needed. TAIEX Conclusions from June 2018: Discussion topic 4): 110 registering in land registry the title and ban on disposal of property. c) Create adequate data bases on property and to develop and implement a project aiming at completing registration of all immovable property d) Prohibit the disposal of assets not entered into the land books]. (DA) from October 2015:  Initiate a discussion among professionals through the HJPC on the obligation of the prosecutor to engage proactively in detection of criminal offenders as of the moment when he/she acquires information on existence of grounds for suspicion that a criminal offence was committed aimed at proper understanding of the role of the prosecutor.  Efficient detection and proving of complex criminal offences requires that the prosecutor has a proactive approach to and plays a more active role in the identification of offenders of whom he/she obtained information from the public afterword and communications, when such information prove justified, both before establishment of grounds of suspicion and after issuance of the investigative order, and that the process is free of corruption risk to the greatest extent possible in terms of selective approach to cases. It is necessary to set up a system which allows prosecutors to play a more active role in the initiation of proceedings (cooperation with the police, enhanced capacity and qualification of the staff in the prosecutors’ offices), as well as to more actively fulfil their role as the leader of the team who directs the investigation.  Analyze possibilities for defining of objective and transparent criteria for selection of priority cases, and initiate a broader professional discussion on selection of cases with the participation of authorized officials and prosecutors who work on such cases, within a broader strategy to combat corruption and organized crime.  Work to expand the sources of information for the prosecutors’ offices which should enable their proactive engagement, through regular exchange of information with the police on so￾called operational monitoring of certain criminal groups, intelligence etc. Peer review mission on criminal procedure from October 2017: It is recommended to improve the performance of the Prosecutor’s offices when it comes to proactive launching of investigations related to high-level corruption, organized crime, money laundering etc., and in that regard:  to establish a detailed track record of ex officio investigations (launched upon audit reports, other external sources, etc.)  to focus more on cases of unexplained wealth of officials (based on money laundering investigation, analyzing asset declarations, etc. …). TAIEX Conclusions from June 2018: Discussion topic 2): 2) Improve proactive investigation in the law enforcement agencies and prosecutor’s offices, including through improving the exchange of criminal intelligence Asset Forfeiture Analysis from July 2017:  Hiring experts in forensic accounting in prosecutor’s offices would be particularly Peer review mission on criminal procedure from October 2017:  Further improvement on forensic expertise within a relevant timeframe, is 111 important for improving the efficiency of the work on cases with financial elements recommended. More initiatives to engage specialized experts are needed. A cooperation with the universities in order to come to a special degree as forensic scientist is strongly suggested. TAIEX Conclusions from June 2018: Discussion topic 3): 2 b): Increase number of qualified financial experts/employment of experts in forensic accounting Asset Forfeiture Analysis from July 2017:  Statutory provisions at all levels in BiH which regulate the scope of application of the special laws on asset forfeiture should be harmonized. This should be done in particular in relation to extended forfeiture, so as to ensure its application on the most serious types of crime, or criminal offences involving significant material gain, ensuring equal protection of the right to property.  It is recommended that key definitions used in special laws, foremost the definitions of property and property gain, be harmonized in line with international standards. This is a key precondition to ensure their equal understanding and application.  Contentious provisions of criminal codes and/or special laws in FBiH, Brčko District and BiH, should be specified, so as to make clear that extended forfeiture, unlike the forfeiture of direct material gain, does not require specification of the criminal offence through which gains were acquired. Accordingly, the link between the assets subject to the motion for extended forfeiture and the offence of which the individual is convicted does not have to be proven.  Chief prosecutors at all levels should issue instructions to acting prosecutors that financial investigation be conducted in all cases where significant material gains were acquired. Peer review mission on criminal procedure from October 2017:  It is recommended to harmonize laws in the four jurisdictions on financial investigations, which in addition have to be in line with international standards.  It is strongly recommended to harmonize laws inside BIH and align these with EU standards (based on the guiding principles of the Directive 2014/42/EU) aimed at revising the rules governing extended seizure and confiscation of assets. In particular, it should be allowed to confiscate all assets directly or indirectly at the disposal of persons convicted of specific offences -typically connected to illicit enrichment-, such as: corruption, organized crime and money laundering, when there is a huge disproportion between their assets and legal income, as declared for tax purposes, or their licit economic activity, when he\she is unable to prove the lawful origin of the assets.  to expand and include systematically financial investigations when conducting organized crime and high-level corruption investigations. TAIEX Conclusions from June 2018: Discussion topic 4): 1) Harmonize the existing legislation 112 [Amend substantive, procedural and organizational provisions in legislations covering asset forfeiture at all level of authorities to ensure efficient and effective asset seizure and harmonized prosecutorial procedures ] 3) Improve practices regarding extended confiscation and apply thoroughly the relevant legal provisions in place Asset Forfeiture Analysis from July 2017:  Due to the lack of relevant statutory provisions in FBiH and BD, there are no specialized law enforcement units that would deal exclusively with evidence gathering in the financial investigation, which is not the case at the level of BiH and RS. Therefore, it is recommended that multidisciplinary, ad hoc teams, be formed to support the work of a prosecutor on certain cases.  Action should be taken to establish a body in the Federation of BiH as a counterpart to the Special Unit of the Ministry of the Interior in the RS. A multidisciplinary team of investigators and experts of suitable profiles would work within such body to support prosecutors working on financial investigation, in all cantonal prosecutor’s offices.  The efficiency of financial investigation greatly depends on the specialization of all bodies participating in it. There is an evident need for trainings for the police about initial estimates of obvious disproportion between the assets and legal income, which inform prosecutor’s decision on whether a financial investigation is warranted. Peer review mission on criminal procedure from October 2017:  to make sure that at the State level, in FBiH and BD financial investigations are be conducted by specialized police officers (what already is the case in the RS)  It is recommended to increase the level of specialization and expertise in seizure and confiscation of assets, and set up specialized police teams, prosecutors and judges, responsible to processing complex cases of seizure and confiscation of assets. TAIEX Conclusions from June 2018: Discussion topic 4): 3) [a) Increase the level of specialization and expertise in seizure and confiscation of assets; Asset Forfeiture Analysis from July 2017:  It is important to have a body or institution established at all levels of the BiH judiciary that would deal exclusively with the management and use of assets seized and forfeited in criminal proceedings. Special effort in this regard is required at the state level, given that there is no system or body whatsoever in charge of managing forfeited assets, which is not the case at the other levels. Peer review mission on criminal procedure from October 2017:  to establish asset recovery and management offices at the State level; align the existing offices with European Standards and ensure their proper functioning in all four jurisdictions  to ensure proper management of the seized assets. TAIEX Conclusions from June 2018: Discussion topic 4): 2) Adopt functional legislative solutions for managing confiscated assets at BIH level after consultations with professional community 113 b) Ensure proper management of the seized assets and to sell seized assets in order to safeguard their value]. Asset Forfeiture Analysis from July 2017:  Increasing the frequency of seeking and ordering security measures on legally acquired assets of the suspect during the criminal proceedings would be a positive and logical step towards the successful asset forfeiture from convicted persons and complete upholding of the principle that no one may retain the proceeds of crime.  In order for security measures to achieve their purpose and yield results, there has to be a possibility to order such measures, before the suspect becomes aware of the proceedings against him, which means before the first action in the proceedings by which the suspect is effectively informed of them.  All jurisdictions should provide for the possibility of seeking provisional measures after the completion of the criminal proceedings, or keeping the measures in force pending the full compensation of the claim, along with the obligation of periodic review of justification of the measure, so as to avoid. Peer review mission on criminal procedure from October 2017:  It is recommended to use more the instrument of (temporary) seizure. TAIEX Conclusions from June 2018: Discussion topic 4): 4) Ensure effective confiscation of proceeds of crime following final and binding court decision by using the tool of temporary securing measures [a) Secure assets in the earliest stage of the criminal procedure (temporary seizure); Asset Forfeiture Analysis from July 2017:  Regular, proper and accurate recording of parameters of asset forfeiture must be ensured by the users of T/CMS, in order to capitalize on the efficiency of the new TCMS module and establish a valid system of monitoring and measuring results achieved in the field of asset forfeiture.  The Rulebook on Orientation Criteria for the Work of the Prosecutors should be upgraded, in order to ensure that, aside from the increased valuation of cases which is exceptionally initiated in complex cases, there is proper and objective regular evaluation of the work on financial investigations reflected through prosecutorial quotas.  Adequate valorization of work of prosecutors and judges should be considered, since they have to invest additional effort to understand frequently extensive body of documents, evaluate evidence and reach a decision in cases with financial investigations. Special attention should be paid to the work on investigating, documenting and deciding on facts relevant to extended asset forfeiture.  It is important to include in the performance evaluation criteria for managers of judicial Peer review mission on criminal procedure from October 2017:  It is recommended to establish a detailed follow-up of results on seizure and confiscation on a systematic basis. Collecting statistical data, but also analyzing them in order to detect deficiencies and determine how and where improvement can be made. TAIEX Conclusions from June 2018: Discussion topic 4): 4 b) Establish track records and a detailed follow up of results (incl. analyzing); 5) Set criteria for measurement of performance of prosecutors conducting financial investigations and asset confiscation [a) Provide regular, proper and accurate documenting of relevant parameters; and, b) Establish a system of regular performance evaluation]. 114 Source: the JA documentation Recommendations of the EC Peer Assessment on financial declarations and their verification in the judiciary (April 2017) that were addressed by USAID the JA through its support to the HJPC in drafting the Rulebook on the submission, verification and management of asset declarations and in drafting amendments to the HJPC Law related to asset declarations (May 2017- to date) Introduction In its Diagnostic Analysis of the Integrity of the BiH Judicial Sector and Potential Risks of its Corrupt of Unethical Conduct (October 2015), USAID the JA recommended that ‘a functional system for the submission and monitoring of assets declared by judges and prosecutors should be established’. In April 2017, EC issued a peer assessment recommending improvements to the asset declaration system, which resulted in the drafting of the HJPC Rulebook on the submission, verification and management of asset declarations (Rulebook) and in the drafting of amendments to the HJPC Law related to asset declarations with USAID the JA significant involvement and support. The following are some of the key EC Peer Assessment recommendations that were addressed through the Rulebook or amendments to the HJPC Law:  A new, more comprehensive disclosure form should be drafted: this new form should include the missing items in the current form, such as unpaid activities or gifts received and more detailed and comprehensive charts regarding assets. It should also lower to the threshold to disclose a piece of asset to 5 000 KM, which would be more appropriate considering the financial situation of judges and prosecutors in BIH. (EC Peer Assessment)  See below article 4, provision 6 of the Draft Rulebook. institutions the criteria of successful asset forfeiture at the level of the entire institution they manage. One option would be a higher mark for the work of the entire institution, if it achieves higher rates of asset forfeiture ordered in final verdicts and/or successfully enforced. Analysis of Verdicts in Corruption Cases Rendered by Courts in BiH in the Period 2013-2015 from December 2017:  It is necessary to strengthen the confirmation of the indictment as a critical stage of criminal proceedings at which the validity and quality of the indictment is assessed. This stage is currently neglected, resulting in significant omissions that reflect on the course of the entire trial. In order to prevent the (negative) consequences of discussing unsustainable or ungrounded criminal charges, it is necessary that pre-trial judges, during the confirmation stage, pay particular attention to whether the indictment contains all the facts and circumstances from which a criminal offense can be deduced, and that in the event of its shortcoming, the indictment is returned with clear instructions for corrections. TAIEX Conclusions from June 2018: Discussion topic 5): 1 e) Strengthen the confirmation of the indictment as a critical stage of criminal proceedings by ensuring the pre￾trial judges pay particular attention to the validity and quality of the facts and circumstances listed in the indictment assessed and ensure indictments with shortcomings are returned to the competent prosecutor with clear instructions for corrections]. 115 Article 4 (Content of a financial statement) (1) A financial statement shall contain the following data of a judge or prosecutor, their spouse and children living in the same household: a. Personal information; b. Information on income from performing of the judicial or prosecutorial office, and from the core activity for the spouse and children living in the same household; c. Information on income from other paid activities; d. Information on ownership of real estates, including the way and time of acquisition, and purchase value; e. Information on ownership of vehicle which is subject to registration within and outside of BiH, including the way and time of acquisition, and value at the time of acquisition; f. Information on ownership of other movables the individual value of which exceeds BAM 5,000 including the way and time of acquisition, and value at the time of acquisition; g. Information on deposits in banks or other financial institutions and cash the value of which exceeds BAM 5,000; h. Information on life insurance policy; i. Information on gifts and donations received related to office above a value of 25 BAM; j. Information on gifts and donations received as a private person if value above BAM 500 each or 1,000 BAM/per year in total; k. Information on investments (including shares and other securities); l. Data on other expenses the individual value of each exceeds BAM 5,000. (2) For the information referred to in paragraph (1) of this Article, the actual value of income and expenses should be specified (e.g. purchase price instead of market price of real property, vehicles, valuables). (3) The information referred to in paragraph (1) of this Article relates to finance, assets and interests in BiH and abroad. (4) For the purpose of prevention of conflict of interest, in the financial statement, a judge or a prosecutor shall also provide information about activities for which no income was earned. (5) A judge or a prosecutor shall also include information about relatives working in the judiciary in the financial statement. Relatives are those in the direct line, relatives in the collateral line up to the third degree of kinship, relatives by marriage up to the second degree of kinship and child of the spouse; marital and extramarital partners; adoptive parents and adopted children. (6) The financial statement form is attached to this Rulebook.  The HJPC should build an effective online solution to fill the disclosure forms: this solution should allow members of the Judiciary to fill their disclosure form online and the HJPC to crosscheck the data, both between several statements and, with specific agreements from competent authorities, with outside databases. It would also enable, if allowed by the legislation at some point, to publish the statements on the Internet. (EC Peer Assessment)  See below article 5 of the Draft Rulebook. Article 5 (Method of submission of financial statements)  A judge or a prosecutor shall submit the financial statement to the Council via the electronic system for the submission of financial statements. 116  The Law on HJPC should be amended to allow the HJPC to obtain any relevant information from other institutions: it should include a legal provision stating that no public or private institution can withhold any information or document, even covered by banking secrecy, requested by the HJPC in respect of its prerogative to monitor the financial disclosures of judges and prosecutors. In the meantime the HJPC should make contact with relevant institutions, notably the tax administration, in order to determine which information could be provided to help monitoring the financial disclosures of members of the judiciary. Memorandums of understanding could also be signed to ratify this kind of cooperation. (EC Peer Assessment)  See below amendments to the HJPC Law recommended to HJPC by USAID the JA. Article 86c (Data Collection and Cooperation with the HJPC) 1) The following are obliged to provide the HJPC with all data necessary for verifying statements on finances and interests: All bodies and institutions of all level of authority in Bosnia and Herzegovina; private natural and legal persons with whom the judge or prosecutor or their family members have concluded legal transactions or who hold data on declared legal transactions; and the family members referred to in Article 86(1) of this Law. 2) All banks and financial institutions and organizations shall confirm to the HJPC upon request whether account balance declared for a certain date in the statement on finances and interests corresponds to the real account balance, and whether there are any other accounts or other contracted financial services with the bank and other financial institutions and organizations that are not specified in the statement on finances and interests. 3) The subjects referred to in paragraphs (1) and (2) of this Article shall provide the requested data as soon as possible, no later than within 15 days of receiving the request. 4) For the purposes of verification of information provided in the statements on finances and interests, the HJPC shall also collect data held abroad in line with domestic and international regulations, as needed.  The financial disclosures monitoring should lead to appropriate measures depending on the result of the verification: If the monitoring reveals a potential incompatibility with the judicial function, the structure in charge of monitoring should refer the case to the HJPC Ethics committee, in order to rule on this matter. If the monitoring reveals that information might have been omitted or inaccurately disclosed, the financial disclosure should be passed over to the ODC for further investigation. Finally, if the monitoring reveals that a criminal offence might have been committed, for example a case of corruption, the file should be passed over to the relevant prosecutor. (EC Peer Assessment)  See below article 18 of the Draft Rulebook: Article 18 (Irregularities found during verification and HJPC’s practice) (1) During the verification of a financial statement, the HJPC may found the following irregularities: a. failure to declare data in the financial statement.; b. declaration of false data in the financial statement. 117 c. hiding of income or expenses; d. financial imbalance that a judge or prosecutor is unable to justify; e. hiding of an activity which is incompatible with the judicial or prosecutorial office; f. hiding of a conflict of interest of a judge or prosecutor. (2) If the verification detects any of the irregularities referred to in paragraph (1) of this Article, the HJPC shall notify the Office of the Disciplinary Counsel and submit to them all the information that arose from the verification for further action. (3) If the verification detects the irregularity referred to in paragraph (1) c) and d), the HJPC shall inform other competent authority and submit to them all the information that arose from the verification. The HJPC shall request feedback from these authorities about the outcome of the procedure.  The law on HJPC should be amended to allow a broad publication of financial disclosures of members of the judiciary: online publication should be authorized under certain conditions, in order to strike a balance between transparency and right to privacy. This balance could be ensured by redacting sensitive information from the form before publishing it; notably all information that would enable a third party to locate an asset (real-estate addresses, bank account numbers, etc.) and all information related to the identification of family members (information regarding their assets and interests should be made public but not their identity).  See below article 156 of the Draft amendments to the Law on HJPC Article 156 (Transparency) (1) Declaration referred to in Article 155 shall be published on the Council’s website. (2) Notwithstanding paragraph (1) of this Article, the following data from the declaration shall not be available to the public: a) First and last name of persons listed in the declaration, except for the names of judges and prosecutors, b) Personal identity number and residence address of the judge or prosecutor or other persons listed in the declaration, c) the name and number of street where the property listed in the declaration is located, d) bank account numbers and other financial ID numbers; e) The amount of cash owned by a judge or prosecutor, and other household members, f) Registration numbers of vehicles. (3) Annual income tax return, and other attachments that judges or prosecutors submit with their declarations, shall not be available to the public, unless judges or prosecutors agree to that. (4) First and last names of the relatives of judges/prosecutors referred to in Article 155, paragraph (3) of this Law shall be published on the Council’s web site.  Generally speaking, the HJPC should be more transparent about the financial disclosure process: First, the template of the disclosure form should be available to the public, so that interested parties are aware of what they can expect to find in a financial disclosure, and formulate freedom of access to information requests accordingly. All internal regulations regarding the financial disclosures should be written into a Book of rules, which should be publicly available. Finally, the HJPC should be more accountable on how the financial disclosures are processed and monitored. It should publish every year, for example in its activity report, the results achieved in that regard (compliance rates, number of disclosures monitored, outcomes of the monitoring).  See below article of the Draft Rulebook: 118 Article 24 (Processing of data contained in financial reports for monitoring and reporting purposes) (1) The HJPC shall publish on its website the following reports at least once a year: a. on the submission of financial statements, which shall contain at least: the number of judges or prosecutors who submitted financial statements; number of submissions, number of late submissions; number of self-corrections of financial statements; number of financial statements where spouse and/or children living in the same household with the judge or prosecutor refused to provide data; b. on conducted verifications, which shall contain at least: the number of verifications (formal, periodic and non-periodic checks); reasons for non-periodic check, as per Article 14 (1); short description of each performed check and data on the results thereof, pursuant to Articles 18 and 19; and information on the availability of data for verification purposes; c. on the HJPC’s actions following verification, which shall contain at least: number of cases of the Office of the Disciplinary Counsel relating to financial statements and number of disciplinary measures, by types of measures; number of cases of the tax administration, prosecutor’s office and/or other competent authority relating to financial statements, by type, and information about the outcomes thereof; d. on access to data in the financial statements that are not subject to publication, which shall contain at least: number of requests of other authorities for access to the data, pursuant to Article 23 (2) of this Rulebook, by types of the authorities. (2) The HJPC shall publish on its website the following information at least on a quarterly basis: a. list of judges and prosecutors obliged to submit financial statements, based on the Register referred to in Article 11 of this Rulebook; b. list of judges and prosecutors who failed to submit financial statements; c. list of judges and prosecutors who made self-corrections to financial statements; d. list of judges and prosecutors whose spouse and/or children living in the same household refused to provide data for the financial statement. (3) The Council will review the reports and information referred to in paragraphs (1) and (2) of this Article and propose measures to improve the implementation of this Rulebook. OTHER  Impose limitations on professional advancement in light of adjudicated judicial disciplinary misconduct (Diagnostic Analysis of the Integrity of the BiH Judicial Sector and Potential Risks of its Corrupt of Unethical Conduct, October 2015),  Effects of disciplinary sanction: Ban to be recruited to another court or prosecutors office or to become chief prosecutor or court president, for a given time (EC Peer Assessment of Disciplinary Procedured in BiH Judiciary, November 2016)  Introduce the right to appeal the judicial appointment decision and to that end improve the explanation provided in the judicial appointment decision (Diagnostic Analysis of the Integrity of the BiH Judicial Sector and Potential Risks of its Corrupt of Unethical Conduct, October 2015)  A legal remedy should be introduced, available to the candidates or other interested parties with regard to council decisions affecting them- such as Recruitment, appointment, reappointment and removal; appointment and removal of managerial position; assessment of performances; transfer; promotions; disciplinary measures; retirement / early retirement, reintegration (EC Peer Assessment on procedure and criteria for appointment of judges and prosecutors) 119 ANNEX VIII: SUMMARY OF CORRUPTION-RELATED LITERATURE REVIEW For the purpose of both the performance evaluation of the USAID/BiH’s Justice Activity and a brief assessment of the BiH justice sector, MEASURE-BIH reviewed corruption-related documents and reports produced by international and domestic organizations. Within this report, the evaluation team provides a summary of the literature review; the full text is available on request. The evaluation team conducted the MEASURE-BIH literature58 review to analyze the content of the corruption-related reports issued in the last three years. The team conducted this review as part of our preparatory work to perform the JA performance evaluation and assess the BiH justice sector. In addition, the evaluation team compared findings from the reviewed literature and reports with those from other sources of data. The team selected reports for review based on surveys and interviews; the team reviewed products of domestic and international legislatures, previous reports of the same kind, and best practices in the international community. They were focused mainly on corruption in the public sector, with an emphasis on judges, prosecutors and political figures; only one report included attorneys and police officers in the scope of the analysis. Some reports had a specific aim, such as EU integrations and the investment sector, so their analyses and findings were focused on the corruption risks affecting their target sectors. The literature review helped the evaluation team to learn that the international community and the domestic NGO sector have recognized the slow pace of the BIH government(s) in adopting and implementing anti-corruption reform. The main findings from all of the material reviewed for this report summarized below:  The complexity of the government and the overlapping jurisdictions and legislation allow the persistence of corrupt practices in governmental institutions.  There is a low level of trust in the judiciary by citizens and few incentives to become involved in anti-corruption efforts or to report corruption.  Political parties are considered inherently corrupt and have a major influence on all branches of government.  There is a lack of harmonization and enforcement of the legislation on corrupt acts of public officials. 58 The materials selected for the literature review included:  Transparency International National Integrity Systems in the Western Balkans and Turkey: Priorities for Reform (October 2016) (TI 2016)  USAID and APIK The Diagnostic Analysis of the Integrity of the Judiciary Sector in BiH and the Possible Risks of Corruption or Unethical Conduct in the Judiciary System (USAID/APIK 2015)  European Parliament Anti-Corruption Efforts in the Western Balkans Briefing (April 2017) (EP 2017)  U.S. Department of State 2017 Investment Climate Statement (August 2017) (USDS 2017)  OSCE Assessing Needs of Judicial Response to Corruption through Monitoring of Criminal Cases Project Report (February 2018) (OSCE 2018)  Greco FOURTH EVALUATION ROUND Corruption prevention in respect of members of parliament, judges and prosecutors COMPLIANCE REPORT BOSNIA AND HERZEGOVINA (February 2016) (GRECO 1)  Greco Third Evaluation Round Fourth Interim Compliance Report on Bosnia and Herzegovina (June 2017) (GRECO 2)  Greco FOURTH EVALUATION ROUND Corruption prevention in respect of members of parliament, judges and prosecutors COMPLIANCE REPORT BOSNIA AND HERZEGOVINA (May 2018) (GRECO 3) 120  The system of financial disclosures of public officials is not properly managed.  There is a lack of cooperation and coordination among the competent judicial and prosecutorial bodies in preventing, detecting, and prosecuting high-level corruption.  High-level corruption cases last too long and do not lead to satisfactory results.  The governmental oversight agencies do not have the necessary authority or scope to perform their duties in an appropriate manner.  The rules on ethics and conflict of interest in the judiciary are not consistent or harmonized. 121 ANNEX IX: DATA COLLECTION INSTRUMENTS INTERVIEW PROTOCOL FOR USAID/BIH DESIGN 1. Background: Can you give us a brief overview of environment in which the JA started to operate? What contextual factors influenced the JA design and how? 2. Could you please explain the JA design and its rationale, including the Activity purpose, theory of change/development hypothesis (and its connection to achievement of the Activity Purpose), Activity components (relationship and objectives) and expected results 3. What are major the JA activities which contribute to achievement of the expected results? PROBE: PPP/POAP: design of POAP activities, PO needs assessment, model used for benchmarking, PO selection (deviation from TO) PROBE: Special Analysis/Diagnostic Studies: selection of topics, rationale for conducting analyses and studies, to what results they contribute PROBE: Consensus Building Forums: identification of key partners, key features of cooperation between the JA and local key partners, and cooperation with other donors operating in the sector of the JA interest (i.e. Collegium of Chief Prosecutors, HJPC SCs) PROBE: Any other activities which you would determine as major for achievement of the expected results 4. Could you please elaborate if there was any consultations/coordination with other donors in the stage of design of the JA activities? 5. Who was the lead in designing activities: beneficiaries, the JA, or someone else? Who defined the beneficiaries needs, and how? Describe the beneficiaries’ and stakeholders’ participation and vetting processes in design of your major activities? To what extent designed activities fit beneficiaries’ needs? IMPLEMENTATION 6. Have you faced any management-related challenges through the JA implementation, internal and/or external (including MEL plan changes)? 7. Please explain coordination and cooperation with other donors during the JA implementation? Individual Features: 8. The JA applied the PPP/POAP mechanism as one of the central tools for supporting the work of line prosecutors on cases of corruption and economic crime; strengthening management practices within POs; enhancing the transparency and public relations of the POs. Could you please elaborate on the main components of the PPP/POAP, its approach and implementation? 122 PROBE: Did POAPs differ for different POs, and to what extent? PROBE: What resources the JA committed to implementation of POAPs (different experts providing support to different PO line of business)? PROBE: How was the sequencing of activities within POAP defined (periodical support or one-time support, was the sequencing the same for all POs)? Were the activities flexible enough to meet extended or prolonged needs of a particular PO? PROBE: PO PR public outreach activities implementation (including MEL, guidelines, approach to information – FOIA, websites, and strategy) PROBE: PGI: how many joint investigative teams have been established, when, what have been the challenges in establishing these teams, what strategies would you use to overcome current issues in creation of these teams? 9. The JA has conducted a number of Special Analysis/Diagnostic Studies during the implementation, could you please elaborate on the studies and their usefulness for the Activity implementation, as well as for the intended beneficiaries? (Diagnostic Study to Determine the Sources of Corruption in the Justice Chain (DA), Expert Witnesses, SAIs, Asset Forfeiture, Court Verdicts): PROBE: What resources the JA committed to their development? PROBE: How was the timing/sequencing of studies defined? Was there any cooperation with beneficiaries in selection, timing, and sequencing of studies development? PROBE: How were the analyses and studies recommendations implemented? 10. Could you please elaborate on the JA training programs implementation as part of TA, as well as on their selection and delivery (e.g. on-site training, off-site training, PGI, cooperation with Swiss/Norwegian project and JPTC)? PROBE: What are the main categories of the trainings you delivered to the JA beneficiaries? PROBE: How were the training programs selected? PROBE: What resources the JA committed to the training? PROBE: What was timing/sequencing of the trainings provided? PROBE: Cooperation with other donors in defining and delivering trainings PROBE: How were joint-teams within PGI formed and trained (how many of them?) 11. Could you please explain how the JA approached the Prosecutors’ performance appraisal system, and what TA was provided, including any changes that may have been formally or informally introduced? PROBLE: What is the current performance appraisal system in place in BiH POs, and when was the current one introduced? If introduced during the JA implementation, please elaborate on the JA assistance. Who else participated in development/adoption of that system? What are the improvements of the newly introduced system? 12. Could you please elaborate on what type of assistance has been delivered to the ODC, including HJPC disciplinary proceedings? PROBE: How have the ODC needs been defined? 123 What resources the JA committed to implementation of ODC-related activities? (different experts providing support to different ODC line of business)? How was the sequencing of activities defined? Were the activities flexible enough to meet extended or prolonged needs of ODC? PROBE: How is the assistance provided perceived by ODC? PROBE: Judicial Discipline Benchbook/Handbook: what is the status of this activity? Who are the counterparts for this activity? How is the JA’s assistance in this task perceived by these counterparts? 13. Could you please elaborate on the JA activities related to support to development/implementation Ethic Codes/Codes of Conduct (for judges and prosecutors, for court and PO personnel – expert associates). How have these been implemented? PROBE: Who were the main beneficiaries and stakeholders involved in this process? PROBE: How was this work valued by beneficiaries and stakeholders? 14. What type of activities have taken place in relation to the implementation of Consensus Building Forums? To what degree have these been useful to the Activity or the Beneficiaries? PROBE: What forums have been implemented (i.e. with APIK, SAI, Forfeiture Agency, HJPC SCs, HJPC WGs, POs, courts, Collegium of Chief Prosecutors, various conferences/events)? PROBE: Have other donors participated in these forums, and in what capacity? PROBE: Is it a correct statement that development and implementation of Integrity Plans is a result of consensus building forums, namely APIK, HJPC, courts and prosecutors’ offices, with the JA support? Please elaborate on development and implementation of Integrity Plans, and results achieved so far. 15. Could you please elaborate on the activities related to improving Judicial Statistics on Corruption Cases and their exchange with law enforcement agencies – implemented ( e.g. workflow redesign, CMS/TCMS improvements, exchange of statistical data among law enforcement – related to results 1.2.1, 2.2.2)? RESULTS 16. What are the results have been achieved under Component 1? To what extent do you assess the JA contributed to strengthening prosecution status and performance? How do we know this? What else remains to be done within the remaining period of TO? PROBE: C1.1 Strengthened organizational leadership, planning, and performance in POs PROBE: C1.2 Prosecutors perform their functions more efficiently through a balanced allocation of resources PROBE: C1.3 Prosecutors uphold trust and integrity through prosecution of corruption or other serious crime 124 PROBE: C1.4 Prosecutors provide appropriate and accurate information to citizens in order to strengthen transparency and responsiveness PROBE: C1.5 Prosecutors status improved through performance appraisal, merit-based career advancement or incentives to prosecute cases 17. What are the results achieved under Component 2? To what extent do you asses the JA contributed to strengthening justice sector institutions to uphold public integrity and combat corruption? How do we know this? What else remains to be done within the remaining period of TO? PROBE: C2.1 Prosecutors investigate and prosecute high profile corruption and economic crime cases free from political or improper influence PROBE: C2.2 Increased cooperation among state, entity, and local justice and other sector actors PROBE: C2.3 ODC is properly resourced to manage complaints procedures, autonomously reviews conduct of judges and prosecutors and recommends appropriate sanctions PROBE: C2.4 Disciplinary proceedings are processed expeditiously and fairly and decisions are subject to independent and impartial review PROBE: C2.5 Public trust in and respect for justice sector institutions are increased due to greater ability to act independently and impartially, and to be held accountable PROBE: C2.6 Prosecutors and judges are trained in identifying elements of corrupt activities and investigate and prosecute corrupt practices 18. In your opinion, how is the JA work perceived and valued by beneficiaries and stakeholders? PROBE: Which activities beneficiaries perceive as most useful? Why? PROBE: Which activities they consider unsuccessful? Why? PROBE: Are there activities with mixed results (the same activity had good results in one environment and poor in the other)? If yes, have you been able to identify the reasons for success and failure? 19. Could you please elaborate to us the following three components of the JA MEL Plan? a. The JA Component one relates to improvements on status and performance of prosecutor offices and/or their strengthening. This is measured through 1) POCM matrix, which continually shows improvements/increased performance, while on the, lower level results it is measured by indicators tracking the number of cases in judicial process at the stage of indictments and convictions for corruption cases, and those do not seem to show significant improvement or a clear pattern? Could you please elaborate why these two levels of indicators do not match in their direction? (Indicators 1.1; 1.1.1; & 1.1.2). b. Based on the JA reports there has been significant success in developing Integrity Plan model which resulted in further development of more than 100 125 individual Integrity Plans by courts and POs, however these are not adopted yet. The JA MEL Plan states that at least 20 will be adopted by the end of the Activity. So this has not materialized. Could you estimate if the target will be met? (Indicator 2.5.3) To our best knowledge, based on the JA MEL Plan on public and professionals perception related to disciplinary proceedings, vulnerability to corruption of the judiciary and POs public relations, there are no significant perception changes or patterns. Could you please elaborate on how these survey results have informed the Activity implementation? 20. Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 21. What are your thoughts on the prospects of meeting the JA life of activity targets, and what is your reasoning on this? 22. Based on the JA contract, it was envisioned that best practices and lessons learned would be continuously shared with relevant justice sector institutions for system-wide reforms and harmonization/unified procedure across all POs. Could you please To what extent have these best practices been institutionalized? LESSONS LEARNED 23. What lessons have you learned based on the JA implementation? RELEVANCE OF THE JUSTICE SECTOR ASSESSMENT TO USAID 1. Could you please elaborate a little bit more on the assessment questions, which are as follows: PROBE: What is the current context under which justice sector operates? How is the current context different from the one identified at the commencement of the JA? PROBE: What are the current most pressing issues in justice sector in BiH? How can they be addressed? To what extent are these issues addressed by current projects/interventions implemented by international and/or local organization? PROBE: What are identified gaps and windows of opportunity in terms of needs for further technical assistance? What are recommendations to the Mission in terms of further programming in justice sector? PROBE: Is there a need or potential benefit of continuing direct assistance to the HJPC in some of the significant areas of HJPC’s mandate or were these sufficiently improved over the past decade? 2. What type of information is relevant to USAID for future planning? Which (if any) particular results USAID is seeking to inform with the assessment? 126 ANNEX III – INTERVIEW PROTOCOL FOR the JA IP DESIGN 3. Background: Can you give us a brief overview of environment in which the JA started to operate? What contextual factors influenced the JA design and how? 4. Could you please explain the JA design and its rationale, including the Activity purpose, theory of change/development hypothesis (and its connection to achievement of the Activity Purpose), Activity components (relationship and objectives) and expected results 5. What are major the JA activities which contribute to achievement of the expected results? PROBE: PPP/POAP: design of POAP activities, PO needs assessment, model used for benchmarking, PO selection (deviation from TO) PROBE: Special Analysis/Diagnostic Studies: selection of topics, rationale for conducting analyses and studies, to what results they contribute PROBE: Consensus Building Forums: identification of key partners, key features of cooperation between the JA and local key partners, and cooperation with other donors operating in the sector of the JA interest (i.e. Collegium of Chief Prosecutors, HJPC SCs) PROBE: Any other activities which you would determine as major for achievement of the expected results 6. Could you please elaborate if there was any consultations/coordination with other donors in the stage of design of the JA activities? 7. Who was the lead in designing activities: beneficiaries, the JA, or someone else? Who defined the beneficiaries needs, and how? Describe the beneficiaries’ and stakeholders’ participation and vetting processes in design of your major activities? To what extent designed activities fit beneficiaries’ needs? IMPLEMENTATION 8. Have you faced any management-related challenges through the JA implementation, internal and/or external (including MEL plan changes)? 9. Please explain coordination and cooperation with other donors during the JA implementation? Individual Features: 10. The JA applied the PPP/POAP mechanism as one of the central tools for supporting the work of line prosecutors on cases of corruption and economic crime; strengthening management practices within POs; enhancing the transparency and public relations of the POs. Could you please elaborate on the main components of the PPP/POAP, its approach and implementation? 127 PROBE: Did POAPs differ for different POs, and to what extent? PROBE: What resources the JA committed to implementation of POAPs (different experts providing support to different PO line of business)? PROBE: How was the sequencing of activities within POAP defined (periodical support or one-time support, was the sequencing the same for all POs)? Were the activities flexible enough to meet extended or prolonged needs of a particular PO? PROBE: PO PR public outreach activities implementation (including MEL, guidelines, approach to information – FOIA, websites, and strategy) PROBE: PGI: how many joint investigative teams have been established, when, what have been the challenges in establishing these teams, what strategies would you use to overcome current issues in creation of these teams? 11. The JA has conducted a number of Special Analysis/Diagnostic Studies during the implementation, could you please elaborate on the studies and their usefulness for the Activity implementation, as well as for the intended beneficiaries? (Diagnostic Study to Determine the Sources of Corruption in the Justice Chain (DA), Expert Witnesses, SAIs, Asset Forfeiture, Court Verdicts): PROBE: What resources the JA committed to their development? PROBE: How was the timing/sequencing of studies defined? Was there any cooperation with beneficiaries in selection, timing, and sequencing of studies development? PROBE: How were the analyses and studies recommendations implemented? 12. Could you please elaborate on the JA training programs implementation as part of TA, as well as on their selection and delivery (e.g. on-site training, off-site training, PGI, cooperation with Swiss/Norwegian project and JPTC)? PROBE: What are the main categories of the trainings you delivered to the JA beneficiaries? PROBE: How were the training programs selected? PROBE: What resources the JA committed to the training? PROBE: What was timing/sequencing of the trainings provided? PROBE: Cooperation with other donors in defining and delivering trainings PROBE: How were joint-teams within PGI formed and trained (how many of them?) 13. Could you please explain how the JA approached the Prosecutors’ performance appraisal system, and what TA was provided, including any changes that may have been formally or informally introduced? PROBLE: What is the current performance appraisal system in place in BiH POs, and when was the current one introduced? If introduced during the JA implementation, please elaborate on the JA assistance. Who else participated in development/adoption of that system? What are the improvements of the newly introduced system? 14. Could you please elaborate on what type of assistance has been delivered to the ODC, including HJPC disciplinary proceedings? PROBE: How have the ODC needs been defined? 128 What resources the JA committed to implementation of ODC-related activities? (different experts providing support to different ODC line of business)? How was the sequencing of activities defined? Were the activities flexible enough to meet extended or prolonged needs of ODC? PROBE: How is the assistance provided perceived by ODC? PROBE: Judicial Discipline Benchbook/Handbook: what is the status of this activity? Who are the counterparts for this activity? How is the JA’s assistance in this task perceived by these counterparts? 15. Could you please elaborate on the JA activities related to support to development/implementation Ethic Codes/Codes of Conduct (for judges and prosecutors, for court and PO personnel – expert associates). How have these been implemented? PROBE: Who were the main beneficiaries and stakeholders involved in this process? PROBE: How was this work valued by beneficiaries and stakeholders? 16. What type of activities have taken place in relation to the implementation of Consensus Building Forums? To what degree have these been useful to the Activity or the Beneficiaries? PROBE: What forums have been implemented (i.e. with APIK, SAI, Forfeiture Agency, HJPC SCs, HJPC WGs, POs, courts, Collegium of Chief Prosecutors, various conferences/events)? PROBE: Have other donors participated in these forums, and in what capacity? PROBE: Is it a correct statement that development and implementation of Integrity Plans is a result of consensus building forums, namely APIK, HJPC, courts and prosecutors’ offices, with the JA support? Please elaborate on development and implementation of Integrity Plans, and results achieved so far. 17. Could you please elaborate on the activities related to improving Judicial Statistics on Corruption Cases and their exchange with law enforcement agencies – implemented ( e.g. workflow redesign, CMS/TCMS improvements, exchange of statistical data among law enforcement – related to results 1.2.1, 2.2.2)? RESULTS 18. What are the results have been achieved under Component 1? To what extent do you assess the JA contributed to strengthening prosecution status and performance? How do we know this? What else remains to be done within the remaining period of TO? PROBE: C1.1 Strengthened organizational leadership, planning, and performance in POs PROBE: C1.2 Prosecutors perform their functions more efficiently through a balanced allocation of resources PROBE: C1.3 Prosecutors uphold trust and integrity through prosecution of corruption or other serious crime 129 PROBE: C1.4 Prosecutors provide appropriate and accurate information to citizens in order to strengthen transparency and responsiveness PROBE: C1.5 Prosecutors status improved through performance appraisal, merit-based career advancement or incentives to prosecute cases 19. What are the results achieved under Component 2? To what extent do you asses the JA contributed to strengthening justice sector institutions to uphold public integrity and combat corruption? How do we know this? What else remains to be done within the remaining period of TO? PROBE: C2.1 Prosecutors investigate and prosecute high profile corruption and economic crime cases free from political or improper influence PROBE: C2.2 Increased cooperation among state, entity, and local justice and other sector actors PROBE: C2.3 ODC is properly resourced to manage complaints procedures, autonomously reviews conduct of judges and prosecutors and recommends appropriate sanctions PROBE: C2.4 Disciplinary proceedings are processed expeditiously and fairly and decisions are subject to independent and impartial review PROBE: C2.5 Public trust in and respect for justice sector institutions are increased due to greater ability to act independently and impartially, and to be held accountable PROBE: C2.6 Prosecutors and judges are trained in identifying elements of corrupt activities and investigate and prosecute corrupt practices 20. In your opinion, how is the JA work perceived and valued by beneficiaries and stakeholders? PROBE: Which activities beneficiaries perceive as most useful? Why? PROBE: Which activities they consider unsuccessful? Why? PROBE: Are there activities with mixed results (the same activity had good results in one environment and poor in the other)? If yes, have you been able to identify the reasons for success and failure? 21. Could you please elaborate to us the following three components of the JA MEL Plan? a. The JA Component one relates to improvements on status and performance of prosecutor offices and/or their strengthening. This is measured through 1) POCM matrix, which continually shows improvements/increased performance, while on the, lower level results it is measured by indicators tracking the number of cases in judicial process at the stage of indictments and convictions for corruption cases, and those do not seem to show significant improvement or a clear pattern? Could you please elaborate why these two levels of indicators do not match in their direction? (Indicators 1.1; 1.1.1; & 1.1.2). b. Based on the JA reports there has been significant success in developing Integrity Plan model which resulted in further development of more than 100 130 individual Integrity Plans by courts and POs, however these are not adopted yet. The JA MEL Plan states that at least 20 will be adopted by the end of the Activity. So this has not materialized. Could you estimate if the target will be met? (Indicator 2.5.3) To our best knowledge, based on the JA MEL Plan on public and professionals perception related to disciplinary proceedings, vulnerability to corruption of the judiciary and POs public relations, there are no significant perception changes or patterns. Could you please elaborate on how these survey results have informed the JA Activity implementation? 22. Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 23. What are your thoughts on the prospects of meeting the JA life of activity targets, and what is your reasoning on this? 24. Based on the JA TO, it was envisioned that best practices and lessons learned would be continuously shared with relevant justice sector institutions for system-wide reforms and harmonization/unified procedure across all POs. Could you please To what extent have these best practices been institutionalized? LESSONS LEARNED 25. What lessons have you learned based on the JA implementation? 26. What institutions are the champions in implementation of the JA activities PROBE: What are characteristics of those institutions? PROBE: In your opinion, what are the major factors determining their success? 27. What approaches/types of assistance are preferred by local counterparts? PROBE: Which one of those you find to be producing tangible and sustainable results? 28. How do you plan to integrate these lessons into your future activities? BiH JUSTICE SECTOR: CONTEXT 29. What is the current context under which justice sector operates? How is the current context different from the one identified in 2014 (at the commencement of the JA)? CURRENT ISSUES IN BIH JUSTICE SECTOR 30. What are the current issues facing BiH justice sector (in relation to courts, prosecutor offices, HJPC, MoJs or other stakeholders) PROBE: How can they be addressed? 131 PROBE: What are the current BiH institutions led initiatives to address these challenges, if any? PROBE: Are there any donors that have already been working on addressing these issues (including support to BiH institutions led initiatives, if any; or NGO led initiatives? GAPS IN BIH JUSTICE SECTOR 31. What would you say, what are the gaps in the BiH justice system? PROBE: How could these gaps be addressed and by whom? PROBE: What type of assistance could USAID provide in bridging these gaps? SUPPORT TO HJPC 32. Based on your experience, how would you rate the quality of HJPC’s work and their capacities? 33. In your opinion does HJPC need more support to improve the quality of their work? Please elaborate why, and in which areas of their work/mandate? 34. In your opinion is continued assistance to HJPC priority over assistance to other Justice Sector institutions? Please elaborate why? 132 INTERVIEW PROTOCOL FOR BENEFICIARIES/STAKEHOLDERS DESIGN AND IMPLEMENTATION 1. Background: Could you try to give us a brief overview of environment in which the JA started to operate back in 2014? 2. How did you get into contact with the JA? 3. How did you understand the JA’s tasks and goals (objective and development hypothesis)? PROBE: the JA task to contribute to the increased efficiency and effectiveness of BiH justice sector institutions, mechanisms and services by helping prosecutors to better conduct investigations and supporting justice institutions to uphold public integrity and combat corruption which will lead to increased public trust in the rule of law. The JA is aiming to implement these through three key features of the program, which includes PPP/POAP, diagnostic assessments, consensus building forums, and other activities. 4. What the JA activities have you participated in? Please describe the implementation from your perspective. PROBE: Who was the lead in implementation of these activities? PROBE: How were the needs for the specific assistance for your institution (PO, Standing Committee, ODC, etc.) defined? Have you been asked to vet the design of this activity? To what extent did the delivered activities fit your needs? PROBE: Could you elaborate on how the implementation of the JA activity you were involved in looked like? PROBE: What resources the JA engaged in that activity? Have they been sufficient? PROBE: How timing/sequencing of activities was? Was the JA flexible in its approach? 5. Overall, how would evaluate the usefulness of the JA assistance your institution (PO, Standing Committee, ODC, etc.) received? PROBE: Which activities were most useful, which were least useful? PROBE: What were the challenges in implementing these activities? PROBE: In your opinion, what are the major factors determining the success of these activities? 6. How would you describe the overall and/or day to day the JA management and communication with you? Have you faced any challenges through implementation? If any, what were your mitigation techniques for overcoming these challenges? IF, aware of or exposed to: 1. How would you assess the POAP/PPP usefulness and implementation (supporting the work of line prosecutors in cases of corruption and economic crime; strengthening management practices within POs, enhancing the transparency and public relations of POs)? PROBE: Please describe the POAP development and implementation for your PO. PROBE: To what extent has the POAP been tailored to your needs? Did/does it address you priorities in relation to corruption and economic crime cases? 133 PROBE: What type of assistance/resources have you received from the JA for the implementation of POAPs? PROBE: How was the sequencing of activities within POAP defined (periodical support or one-time support, was the sequencing the same for all POs)? Were the activities flexible enough to meet your needs? PROBE: Did you PO establish PGI? Please describe the processes, and challenges you faced. If not, why not? PROBE: What types of MEL assistance have you received from the JA? Has your MEL system improved as a result of this assistance? 2. The JA conducted several Special Analysis/Diagnostic Studies, are you familiar with them? How would you assess their usefulness and implementation? (Diagnostic Study to Determine the Sources of Corruption in the Justice Chain (DA), Expert Witnesses, SAIs, Asset Forfeiture, Court Verdicts)? PROBE: the JA conducted a number of analyses and diagnostic studies. Have your about these? To what extent have you been consulted on the need for such analyses? PROBE: Did you implement any recommendations from these analyses and studies? 3. Could you explain in which training programs you/your institutions has participated in? How would you assess training programs usefulness and implementation (on-site training, off-site training, PGI, cooperation with Swiss/Norwegian project and JPTC)? PROBE: How were the training programs selected? Did you participate in defining which training programs your institution/organization will participate? PROBE: What was timing/sequencing of the trainings provided? 4. Are you familiar with the work that the JA has been doing in relation to the Prosecutors’ performance appraisal system? If yes, please elaborate. PROBE: What is the current performance appraisal system in place in BiH POs, and when was the current one introduced? PROBE: If introduced during the JA implementation, please elaborate on the JA assistance that you are familiar with. Who else participated in development/adoption of that system? PROBE: What are the improvements of the newly introduced system? 5. How would you assess the usefulness and implementation of ODC-related activities (including HJPC disciplinary proceedings)? Could you please elaborate on each of them? PROBE: How have the ODC or HJPC bodies in charge of disciplinary proceedings needs been defined? PROBE: What type of assistance the JA delivered to ODC or relevant HJPC bodies? PROBE: How was the sequencing of activities defined? Were the activities flexible enough to meet extended or prolonged needs of ODC/HJPC bodies? 134 PROBE: What is the JA contribution in improvement of ODC work or HJPC disciplinary proceedings? PROBE: How is the JA assistance perceived by ODC or HJPC competent bodies? 6. What ethic codes for prosecutors/judges exist? Are you aware of the JA’s assistance to their revisions? Please elaborate. 7. Are you aware of any consensus building forums that the JA organized, and have you participated in any of them (i.e. with APIK, SAI, Forfeiture Agency, HJPC SCs, HJPC WGs, POs, courts)? Have they been useful? PROBE: If familiar to you, could you elaborate on development and implementation of Integrity Plans? If applicable to your organization: have you adopted the integrity plan? Please describe the process. Has it been implemented? How would you assess its usefulness? PROBE: Did you notice improvements across different jurisdictions as result of the JA consensus building forum activities? 8. What was your collaboration with the JA like in terms of improving judicial statistics on corruption cases and their exchange with law enforcement agencies – implementation: PROBE: PO workflow redesign PROBE: CMS/TCMS improvements PROBE: Exchange of statistical data among law enforcement agencies RESULTS 9. What have you achieved so far in cooperation with the JA, and what is left for you to do in the ongoing period of relevance for activities being implemented by the JA? 10. Would you say that the JA contributed to the strengthening prosecution status and performance including: a) Organizational leadership, planning, and performance of POs b) Efficiency of prosecutors’ performance in terms of resource allocation c) Public trust in prosecutors and perceptions on their integrity due to improved prosecution of corruption or other serious crime d) Transparency and responsiveness of prosecutors due to providing appropriate and accurate information to citizens e) Prosecutors status through performance appraisal, merit-based career advancement or incentives to prosecute cases f) Investigation and prosecution of high profile corruption and economic crime cases by prosecutors, free from political or improper influence g) Cooperation among state, entity, and local justice and other sector actors h) ODC’s resources to manage complaints procedures, autonomously review conduct of judges and prosecutors and recommend appropriate sanctions 135 i) Expeditious and fair processing of disciplinary proceedings, and decisions are subject to independent and impartial review j) Public trust in and respect for justice sector institutions due to greater ability to act independently and impartially, and to be held accountable k) Prosecutors and judges training on identifying elements of corrupt activities and investigating and prosecuting corrupt practices 11. What is needed to achieve a stronger contribution to these results? 12. Were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 13. To what extent have best practices identified through the JA implementation been institutionalized? PROBE: Have any recommendations been institutionalized, have any system-wide reforms been proposed or adopted as per the JA recommendations? PROBE: To what degree have they been harmonized/unified across all POs? LESSONS LEARNED 14. What lessons have you learned based on the JA implementation? 15. Based on this, what would you recommend USAID for continuation of the JA and future planning? BiH JUSTICE SECTOR: CONTEXT 16. What is the current context under which justice sector operates? How is the current context different from the one identified in 2014 (at the commencement of the JA)? CURRENT ISSUES IN BIH JUSTICE SECTOR 17. What are the current issues facing BiH justice sector (in relation to courts, prosecutor offices, HJPC, MoJs or other stakeholders) PROBE: How can they be addressed? PROBE: What are the current BiH institutions led initiatives to address these challenges, if any? PROBE: Are there any donors that have already been working on addressing these issues (including support to BiH institutions led initiatives, if any; or NGO led initiatives? GAPS IN BIH JUSTICE SECTOR 18. What would you say, what are the gaps in the BiH justice system? 136 PROBE: How could these gaps be addressed and by whom? PROBE: What type of assistance could USAID provide in bridging these gaps? SUPPORT TO HJPC 19. Based on your experience, how would you rate the quality of HJPC’s work and their capacities? 20. In your opinion does HJPC need more support to improve the quality of their work? Please elaborate why, and in which areas of their work/mandate? 21. In your opinion is continued assistance to HJPC priority over assistance to other Justice Sector institutions? Please elaborate why? 137 INTERVIEW PROTOCOL FOR NON-BENEFICIARIES/EXTERNALS DESIGN AND IMPLEMENTATION 11. Background: Could you try to give us a brief overview of environment in which the JA started to operate back in 2014? 12. Did you have a chance to get into contact with the JA since 2014? On which occasion or working on which task? Occasional or continuous interaction? 13. What is your understanding of the JA’s tasks and goals (objective and development hypothesis)? PROBE: If yes, please elaborate. PROBE: If no, explain and ask to elaborate: the JA seeks to contribute to the increased efficiency and effectiveness of BiH justice sector institutions, mechanisms and services by helping prosecutors to better conduct investigations and supporting justice institutions to uphold public integrity and combat corruption which will lead to increased public trust in the rule of law. The JA is aiming to implement these through three key features of the program, which includes PPP/POAP, diagnostic assessments, consensus building forums, and other activities. 14. Have you participated in any of the JA activities or has the JA participated in activities designed by your organization? If yes, please describe the implementation from your perspective. PROBE: Who was the lead in implementation of these activities? PROBE: What is your understanding about how were the needs for the specific assistance defined? How the beneficiaries where selected? Have you been asked to participate in designing or vetting of activities? To what extent did the delivered activities fit beneficiaries’ needs? PROBE: Could you elaborate on how the implementation of the JA activity you were involved in looked like? or, How has the JA contributed in activities designed by your organization? PROBE: What resources the JA engaged in that activity? Have they been sufficient? or, What was contribution of the JA in term of resources in your joint activities? PROBE: How timing/sequencing of activities was? Was the JA flexible in its approach? PROBE: Have any of these activities complemented on your activities or, on the contrary, the JA activities were overlapping with your activities? 15. Overall, how would you evaluate the usefulness of the JA assistance to local institutions (PO, Standing Committee, ODC, etc.)? PROBE: Which activities were most useful, which were least useful? PROBE: What were the challenges in implementing these activities? PROBE: In your opinion, what are the major factors determining the success of these activities? 16. How would you describe the overall and/or day to day the JA management and communication with you? Have you faced any challenges through implementation? If any, what were your mitigation techniques for overcoming these challenges? 138 IF, aware of or exposed to: 17. How would you assess the POAP/PPP usefulness and implementation (supporting the work of line prosecutors in cases of corruption and economic crime; strengthening management practices within POs; enhancing the transparency and public relations of POs)? 18. The JA conducted several Special Analysis/Diagnostic Studies, are you familiar with them? How would you assess their usefulness and implementation? (Diagnostic Study to Determine the Sources of Corruption in the Justice Chain (DA), Expert Witnesses, SAIs, Asset Forfeiture, Court Verdicts)? 19. Could you explain in which training programs you/your institutions has participated in? How would you assess training programs usefulness and implementation (on-site training, off-site training, PGI, cooperation with Swiss/Norwegian project and JPTC)? 20. Are you familiar with the work that the JA has been doing in relation to the Prosecutors’ performance appraisal system? If yes, please elaborate. 21. How would you assess the usefulness and implementation of ODC-related activities (including HJPC disciplinary proceedings)? Could you please elaborate on each of them? 22. What ethic codes for prosecutors/judges exist? Are you aware of the JA’s assistance to their revisions? Please elaborate. 23. Are you aware of any consensus building forums that the JA organized, and have you participated in any of them (i.e. with APIK, SAI, Forfeiture Agency, HJPC SCs, HJPC WGs, POs, courts)? Have they been useful? 24. What was your collaboration with the JA like in terms of improving judicial statistics on corruption cases and their exchange with law enforcement agencies – implementation: RESULTS 25. Would you say that the JA contributed to the strengthening prosecution status and performance? PROBE: a) Organizational leadership, planning, and performance of POs b) Efficiency of prosecutors’ performance in terms of resource allocation c) Public trust in prosecutors and perceptions on their integrity due to improved prosecution of corruption or other serious crime d) Transparency and responsiveness of prosecutors due to providing appropriate and accurate information to citizens e) Prosecutors status through performance appraisal, merit-based career advancement or incentives to prosecute cases f) Investigation and prosecution of high profile corruption and economic crime cases by prosecutors, free from political or improper influence g) Cooperation among state, entity, and local justice and other sector actors 139 h) ODC’s resources to manage complaints procedures, autonomously review conduct of judges and prosecutors and recommend appropriate sanctions i) Expeditious and fair processing of disciplinary proceedings, and decisions are subject to independent and impartial review j) Public trust in and respect for justice sector institutions due to greater ability to act independently and impartially, and to be held accountable k) Prosecutors and judges training on identifying elements of corrupt activities and investigating and prosecuting corrupt practices 22. What is needed to achieve a stronger contribution to these results? 23. If aware, were the activities planned and implemented directly with HJPC on the issues of integrity and discipline sufficient and can these be considered successes? 24. If aware, to what extent have best practices identified through the JA implementation been institutionalized? LESSONS LEARNED 25. What lessons have you learned based on the JA implementation? 26. Based on this, what would you recommend USAID for continuation of the JA and future planning? BiH JUSTICE SECTOR: CONTEXT 27. What is the current context under which justice sector operates? How is the current context different from the one identified in 2014 (at the commencement of the JA)? CURRENT ISSUES IN BIH JUSTICE SECTOR 28. What are the current issues facing BiH justice sector (in relation to courts, prosecutor offices, HJPC, MoJs or other stakeholders) PROBE: How can they be addressed? PROBE: What are the current BiH institutions led initiatives to address these challenges, if any? PROBE: Are there any donors that have already been working on addressing these issues (including support to BiH institutions led initiatives, if any; or NGO led initiatives? GAPS IN BIH JUSTICE SECTOR 29. What would you say, what are the gaps in the BiH justice system? 30. How could these gaps be addressed and by whom? 31. Which institutions in the BiH justice sector need further donor assistance? Why? 140 32. What are the current or planned activities of your organization in the BiH justice sector (in relation to courts, prosecutor offices, HJPC, MoJs or other stakeholders) 33. What type of assistance could USAID provide in bridging these gaps? SUPPORT TO HJPC 34. Based on your experience, how would you rate the quality of HJPC’s work and their capacities? 35. In your opinion does HJPC need more support to improve the quality of their work? Please elaborate why, and in which areas of their work/mandate? 36. In your opinion is continued assistance to HJPC priority over assistance to other Justice Sector institutions? Please elaborate why? 141 ANNEX X: LIST OF INTERVIEWEES # Interview Stakeholder # Participants/nam es Position USG Agencies 1 USAID/BiH 1 Jasna Kilalic COR the JA, Deputy Democracy Officer 2 Justice Activity IP 2 Biljana Potparic Chief of Party 3 Muhamed Susic Deputy Chief of Party 4 Sanela Paripovic Component 1 Lead 5 Elmerina Ahmetaj -Hrelja Component 2 Lead 6 Elmir Halebic Assistant to COP 7 Sanin Dzidic MEL Director 3 OPDAT 8 Goran Krnaich OPDAT representative 9 Elma Ahic OPDAT representative 4 INL 10 INL representative INL representative Prosecutor Offices (PO) 5 BiH PO 11 Gordana Tadic Acting Chief Prosecutor 12 Mersudin Pruzan Prosecutor 6 RS SPO 13 Miodrag Bajic Chief Special Prosecutor 14 Zivana Bajic Prosecutor 7 RS PO 15 Mahmut Svraka Chief Prosecutor 8 FBiH PO 16 Tihomir Jurko Deputy Chief Prosecutor 9 District PO/East Sarajevo 17 Rajko Colovic Chief Prosecutor 18 Neven Kramer Secretary of PO 10 District PO/Banja Luka 19 Zelimir Lepir Chief Prosecutor 20 Zoran Bulatovic Deputy Chief Prosecutor 21 Sead Zeric Deputy Chief Prosecutor 11 District PO/Doboj; HJPC (member); HJPC SC for efficiency of POs; HJPC SC for legislation; HJPC SC for international relations and EU integrations 22 Zeljka Radovic Chief Prosecutor 12 District PO/Bijeljina 23 Milorad Debeljević Deputy Chief Prosecutor 13 Cantonal PO/Bihac 24 Fadila Amidzic Chief Prosecutor 25 Husein Huzejrovic Deputy Chief Prosecutor 142 # Interview Stakeholder # Participants/nam es Position 26 Senad Ljubijankic Head of Department for Economic Crime 14 Cantonal PO/Zenica 27 Redzo Delic Deputy Chief Prosecutor 28 Azra Alic Senior Advisor 29 Adis Babic Senior Economic Advisor 15 Cantonal PO/Tuzla 30 Tomislav Ljubic Chief Prosecutor 31 Mirzeta Begic Deputy Chief Prosecutor 16 Cantonal PO/Sarajevo 32 Anika Kešelj Deputy Chief Prosecutor 33 Senad Osmić Prosecutor 34 Sanin Bogunic Head of Department for Economic Crime 17 Cantonal PO/Siroki Brijeg 35 Vesna Cavar Chief Prosecutor 36 Dejan Ruzic Secretary of PO 37 Josip Ancic Senior Advisor 18 Cantonal PO/Travnik 38 Senad Dautovic Chief Prosecutor 19 Cantonal PO/Mostar 39 Zdenko Kovac Chief Prosecutor 20 District Brcko PO 40 Pavo Radoc Deputy Chief Prosecutor 41 Samir Beganovic Secretary 42 Amira Imamovic Head of Administration Courts 21 Supreme Court FBiH 43 Milorad Novkovic Court President 44 Ilhana Tahmaz Jamak Spokesperson 22 Court of BiH 45 Minka Kreho Judge 46 Emira Hodzic Registry 23 RS Supreme Court 47 Obren Buzanin President of Criminal Cases Department 48 Smiljana Mrsa President of Administrative Cases Department 24 First Instance, Sarajevo 49 Janja Jovanović Court President 25 First Instance, Mostar 50 Divna Bosnjak Court President 51 Kimeta Ljeljak Secretary of Court 52 Amil Sukman Court Staff Member 26 First Instance Court, Banja Luka 53 Milenko Mrkic President of Court 54 Dragan Vuletovic Deputy President of Court 55 Vladimir Jurisic Secretary 143 # Interview Stakeholder # Participants/nam es Position 56 Lana Kovacevic Registry 27 Second Instance, Sarajevo 57 Jasmin Jahijaefendic President of Court 58 Danijela Mikic Head of Department of Administrative Cases 28 Second Instance, Mostar 59 Zoran Krtalic Deputy of Court President 60 Nurko Pobric Judge 29 Second Instance Court, Banja Luka 61 Sadeta Veladzic Judge 62 Bozana Vulic Judge 63 Snjezana Kudric Judge 64 Svjetlana Maric Judge 65 Boris Trifunovic President of Civic Department HJPC 30 HJPC/Office of Disciplinary Prosecutor 66 Mirza Hadziomerovic Deputy of Chief Disciplinary Prosecutor 67 Amra Asanovic Disciplinary Prosecutor 31 HJPC (member); HJPC SC for efficiency of POs; HJPC SC for judicial administration and budgets; HJPC SC for judicial ethics, independence and impartiality; HJPC disciplinary panels, PO RS 68 Mahmut Svraka PO RS Chief Prosecutor 32 HJPC (member); HJPC SC for efficiency of POs; HJPC SC for legislation; HJPC SC for conducting test and appointment procedures; WG for T-CMS; HJPC disciplinary panels, PO FBiH 69 Slavo Lakic PO FBiH Prosecutor 33 HJPC Secretariat 70 Hajro Poskovic Deputy Director of HJPC Secretariat 71 Adis Hodzic Deputy Chief of Budgets and Statistics Department 72 Svjetlana Vujakovic Department of Accounting and Financing 73 Azir Mrdjanovic Legal Advisor 74 Jasmin Calija Deputy Chief of Appointments Departmetn 144 # Interview Stakeholder # Participants/nam es Position 75 Ana Bilic￾Andrijanovic Project Manager 76 Radenka Lukic Deputy Chief of Court Administration Department Donors 34 Swiss Development and Cooperation Agency 77 Haris Lokvancic Representative of the Swiss Embassy 78 Amila Rahic Project Manager 35 OSCE 79 Francesco de Sanctis Legal Advisor on Anti￾Corruption 36 Norwegian Embassy 80 H.E. Guri Rusten Ambassador 37 Swedish Embassy 81 Nedim Bukvic National Program Officer 38 Italian Embassy 82 H.E. Nicola Minasi Ambassador 39 UK Embassy 83 Dalida Tanovic Project Manager 40 EU Delegation to BiH 84 Julien Berthound Advisor on Rule of Law 85 Johannes Hintzen Programme Manager for Justice Reform and Security Governme nt Institutions 41 BiH Ministry of Justice 86 Nezir Pivic Deputy Minister 87 Milana Popadic Assistant Minister 42 APIK 88 Hazim Sabanovic Head of Agency 89 Igor Corluka Deputy Head of Agency 43 FBiH Agency for Management of Forfeited Assets 90 Kenan Kapo Director 44 FBiH Supreme Audit Institution 91 Mia Buljubasic Senior Auditor for International Cooperation 92 Dragan Kolobaric Deputy Chief Auditor Other 45 FBiH Judicial Training Center 93 Almir Tabakovic Senior Advisor 46 FBiH/BiH Association of Prosecutors 94 Hajrija Hadziomerovic￾Muftic President/Prosecutor 47 FBiH/BiH Association of Judges and Association of Women Judges BiH 95 Lejla Konjic Dragovic President/Judge 48 Association of Spokespersons 96 Nina Hadzihajdarevic Senior Associate for Public Relations 49 RS Association of Prosecutors 97 Zivana Bajic President of Association 50 RS Association of Judges 98 Milenko Milekic President of Association 145 # Interview Stakeholder # Participants/nam es Position CSOs 51 ACCOUNT 99 Eldin Karic Executive Director 10 0 Mirela Pobric Layer 52 CIN 10 1 Aladin Avdagic Chief Editor 10 2 Lejla Bicakcic Executive Director 53 Transparency International 10 3 Ivana Korajlic Acting Executive Director 10 4 Emsad Dizdarevic Project Manager 54 LucidLinks 10 5 Samra Suskic Basic Senior Consultant 10 6 Nermin Katrubasic Consultant 55 NetCounsulting 10 7 Bojan Bajic CEO 146 ANNEX XI: DISCLOSURE OF ANY CONFLICT OF INTEREST 147 148 149 150 151 ANNEX XII: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM USAID/BIH MISSION AND THE IMPLEMENTING PARTNER (IP) 152 Date: March 22, 2019 To: Elma Bukvic Jusic Development Assistance Specialist / MEASURE-BiH COR USAID/BiH Subject: THE EVALUATION TEAM’S RESPONSES TO COMMENTS FROM USAID/BiH AND IP ON THE PERFORMANCE EVALUATION OF USAID/BIH’S JUSTICE ACTIVITY (JA) REPORT Dear Ms. Bukvic Jusic, The evaluation team would like to thank USAID/BiH and the IP for providing their comments on the Draft JA Performance Evaluation Report. The evaluation team carefully examined each comment and made changes in the Final Report when justified. In addition, included below are the evaluation team’s responses to USAID/BIH’s and the IP’s individual comments on the Performance Evaluation of USAID/BIH’s Justice Activity (JA) report in chronological order. THE EVALUATION TEAM’S RESPONSES TO THE USAID/BiH COMMENTS: USAID/BiH COMMENT #1: In general and for most parts, the report looks good and reads accurate, though more so as a compilation of statistical data and less as a subject matter experts evaluation. The Evaluation Team Response: We are glad that you find our report to be accurate. Regarding the comment that the evaluation reads “less as a subject matter experts evaluation,” please note USAID ADS 201 and USAID Evaluation Policy requirements for evaluations. We also note that the composition of the evaluation team and its expertise were included in the SoW, shared with and approved by USAID/BIH prior to beginning the evaluation. Consequently, the same team conducted both the JA performance evaluation and the Brief Assessment of the BiH Justice Sector. Regarding the comment that the evaluation reads “more so as a compilation of statistical data,” the Monitoring and Evaluation (M&E) section of the JA contract specifies that the performance indicators will be used “to measure the results for each project component and establishes baseline measurement to assess the impact of proposed interventions.” In the JA MEL Plan, 15 out of 25 indicators use the administrative and perception data. The evaluation team in its evaluation approach followed on the JA contract requirements and used data included in the JA MEL Plan. Those data were complemented by information which the evaluation team obtained from 107 key informants, the National Survey of Citizens’ Perceptions and a document review. Following the data collection process and analysis, the evaluation team presented its findings and conclusions on the JA’s progress towards achieving the interventions’ expected results. 153 USAID/BiH COMMENT #2: We have reviewed the report carefully and we would like to provide few general comments on the structure of the evaluation report. The executive summary is very important part of the evaluation, particularly when the full report is 100+ pages. Therefore, the executive summary needs to contain the most important segments of the report. The Evaluation Team Response: The USAID Evaluation Report Requirements (ADS 201mah) prescribe a 2-5 page long Executive Summary. Within the limited space, we provided the most important components of the report. In addition, in response to USAID/BiH’s and the IP’s comments, we made further adjustments to the Executive Summary. Those adjustments reflect changes made in the body of the report, as specified in the rest of our responses. USAID/BiH COMMENT #3: For the project that provides assistance to individual prosecutors offices, the status of these offices and impact that JA assistance had on their overall results is the most important part. These individual results are currently not included in the executive summary. Also, the focus on these individual results should prevail throughout the document, not just in the executive summary, versus very general information about the status of prosecution of corruption cases in BiH The Evaluation Team Response: The evaluation team carefully examined the JA contract and identified that major assistance to individual POs, through PPPs with groups of POs in different years, is a principal task of the JA Component 1. Results achieved in work with groups of PPPs are tracked through corresponding outcomes/outputs set in the JA contract, and indicators set in the JA MEL Plan (in absence of quantifiable indicators in the JA contract). In relation to the individual results achieved by POs in processing corruption cases in the period of the JA implementation, due to the overall length of the report, their detailed data are provided in the Annex II. In accordance with the law, the principal task of POs is to prosecute crimes. Given the priority of corruption cases, achieved results by any individual or all POs in BiH are observed through the number of prosecutions in corruption cases. The overall conclusion of the evaluation team is that no observable changes in processing corruption cases occurred in the period of the JA implementation, with only 10 more indictments for corruption filed in 2017 than in the year which preceded the JA implementation (more precisely there were 283 indictments in 2017 vs 273 indictments in 2014, or 3.7% change over 2014 – 2017 period). Consequently, and as Annex II presents, the results of individual POs are mixed, some performed better and some performed worse (most of them in a highly volatile patterns).These results suggest that changes in POs’ performance do not coincide with the JA Activity implementation in particular POs. 154 More specifically, POs that received JA TA across years and increased the number of indictments filed in 2017 compared to their baseline value in 2014 include the following eight (8): PO Zenica, PO Tuzla, PO Doboj, PO Bihac, PO Mostar, PO Sarajevo, PO Brcko District and PO Orasje. Individual POs which filled fewer indictments in 2017 than in 2014 include the following eight (8): PO Banja Luka, PO East Sarajevo, PO Bijeljina, PO Trebinje, PO Siroki Brijeg, PO Livno, PO Travnik and PO Gorazde. The table to the right contains data for all individual POs in BiH for a reader’s further reference. Given the overall length and structure of the report and data provided in Annexes and this response, the evaluation team believes that sufficient data on the results of individual POs during the JA implementation are provided. USAID/BiH COMMENT #4: The final part of USAID/BiH Comment #4 ends with a statement that the evaluation report “…include data from the POs that were not partners of the project, such as State PO.” The Evaluation Team Response: The evaluation team found and presented in the evaluation report that the JA did not partner with PO BiH. Consequently, the evaluation report mentions PO BIH only in the “Context in which JA operates” section, where an explanation is provided about why the JA did not partner with this PO. No data on the prosecution of cases by PO BiH are presented either in that section or throughout the rest of the evaluation report. 2014 2015 2016 2017 TREND 273 334 262 283 128 201 174 181 122 114 70 80 13 12 7 7 10 7 11 15 PO Bihac 12 23 29 22 PO Orasje 1 6 6 4 PO Tuzla 27 75 32 45 PO Zenica 12 32 35 43 PO Gorazde 1 3 1 0 PO Travnik 19 11 14 17 PO Mostar 11 10 13 15 PO Siroki Brijeg 13 8 12 3 PO Sarajevo 21 28 24 26 PO Livno 11 5 8 6 PO Banja Luka 70 53 34 22 PO Bijeljina 22 21 10 12 PO Doboj 5 15 12 20 PO Trebinje 8 3 1 2 PO East Sarajevo 16 16 12 5 PO BL Special Department 1 6 1 - PO RS Special Department - - - 10 PO Prijedor - - - 9 DATA LEVEL INDICTMENTS FILED - CORRUPTION CASES PO BiH PO Brcko District FBiH Total BiH Total FBiH Total RS 155 On the contrary, the evaluation team identified that JA included results of PO BiH in its JA MEL indictors. Namely, indicators sourced from HJPC administrative data, in the last row (“All POs in BiH”) contain PO BiH data as well. In the evaluation report, the team invited the JA to re-examine the validity of its calculation (please see page 19 footnote 33, and page 51 footnote 50). USAID/BiH COMMENT #5: Also, we suggest rewriting the recommendation section in the executive summary that provide set of activities that JA could/should attempt before the end of the contract. Since the time left in the contract is significantly short, and due to delays in submission of this report, most of the recommendations read unrealistic and therefore should probably be deleted. The Evaluation Team Response: The evaluation team agrees with facts presented in this comment. The recommendations provided are in line with presentation of the evaluation team’s preliminary findings in July 2018 and delivering of the draft evaluation report in September 2018. These recommendations were mainly provided for USAID/BIH’s consideration. Given the time between delivering the draft report and providing this response, the evaluation team modified the recommendation section. USAID/BiH COMMENT #6: Finally, please note that the termination of former Component 2 was done unilaterally by USAID for the reasons that have nothing to do with the way implementer performed the tasks under this component in the year 1 of the project. We believe this should be reflected in the report. The Evaluation Team Response: The evaluation made modifications to the report in response to this comment. Specifically, the evaluation team modified the report by adding a quote from the JA contract modification and provided additional information in the evaluation report to reflect the fact that USAID unilaterally terminated the former Component 2. USAID/BiH COMMENT #7: Please limit the Executive summary to 5 pages as prescribed by the USAID Evaluation Report requirements. The Evaluation Team Response: The Executive summary was reduced to 5 pages. 156 THE EVALUATION TEAM’S RESPONSES TO THE IPs’ REQUEST FOR CLARIFICATION IP GENERAL COMMENT Millennium DPI Partners, LLC, as the implementing partner contracted under the Task Order No. AID￾168-TO-14-000001 to implement the USAID Justice Activity (JA), would like to thank USAID/Bosnia and Herzegovina for the opportunity to provide comments on the Draft Report on Performance Evaluation of USAID Justice Activity (Draft Report or Report), dated September 18, 2019 and prepared by the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH). As mentioned in the opening paragraphs of the Draft Report, the JA was designed to support USAID’s broader democracy and governance goal of achieving more functional and accountable institutions that meet BiH citizen needs (Development Objective [DO] 1), as expressed in USAID’s Country Development Cooperation Strategy (CDCS). The JA’s goal is to contribute to the USAID/BiH Intermediate Result (IR) and Project 1.1 — More effective judicial, executive, and legislative branches of government. Specifically, the JA is also expected to further USAID’s IR of making government more responsive to citizens by strengthening the capabilities of justice sector actors (IR 1.1.1). Millennium DPI would like to add that the JA is designed to achieve the said IRs during the period of 2014– 2019 and help increase the efficiency and effectiveness of BiH justice sector institutions, mechanisms and services by helping: 1) prosecutors to better conduct investigations (Component 1), and 2) justice institutions to uphold public integrity and combat corruption, and ultimately increase public trust in the rule of law (Component 2). The Evaluation Team Response: The evaluation team made a change on page 1 reflecting the names of the JA Components, as specified in the JA Contract/Modification. IP’s Comment #1: In that regard, Millennium DPI’s first general comment is related to the content of the Executive Summary of the Draft Report, which should reflect the entire Draft Report and allow a reader to grasp all facts and key findings without having to read the full text in detail. It is our impression that the draft Executive Summary does not sufficiently convey the evaluator’s conclusions about the performance of the JA and its components, especially those that are clearly drawn and noted further in the text. An example of this is a conclusion given on p. 17 (EQI): The JA’s interventions have been appropriate and met beneficiaries’ needs, to the extent possible given available resources, in institutional strengthening for prosecuting and processing CEC cases. This is not reflected in the Executive Summary. The evaluator gives only a conclusion on the performance of the Ex-Component 2 in the following way: The evaluation team concludes that initial activities within Ex-Component 2 were performed correctly (p. 5). The Evaluation Team Response: Please see the response provided to USAID/BiH Comment #2. 157 The first sentence of the second paragraph on page 2 of the Executive Summary reads: “The evaluation team found that in light of the disparity between the justice sector’s limited available resources and beneficiaries’ extensive needs, the JA’s design reflected the justice sector’s need to strengthen institutions to prosecute and process corruption and economic crime (CEC) cases.” The evaluation team’s conclusion on page 17 reads: “The JA’s interventions have been appropriate and met beneficiaries’ needs, to the extent possible given available resources, in institutional strengthening for prosecuting and processing CEC cases.” We believe that the statements on pages 2 and page 17 are similar. Nevertheless, the evaluation team replaced the first sentence of the second paragraph on page 2 within the Executive Summary with the text on page 17. In terms of the comment related to the former Component 2, please see the response to the USAID/BiH comment #6. IP’s Comment #2: In the Findings and Conclusions (pp. 2–5) of the Executive Summary, there are many statements given by Key Informants (KIs) and stakeholders without the evaluator’s assessment of such statements. For example, on p. 2, it states that: KIs confirmed that the JA’s design appropriately tailored implementing activities to meet the beneficiaries’ needs, while POs and other stakeholder KIs positively noted the JA’s ability to adapt and tailor assistance to the diverse demands, needs, and operational dynamics of the POs. If this is an opinion shared by the evaluation team or represents a conclusion arrived at by the team based on the KI’s statements, this should be clearly stated since the purpose of this section is to summarize the evaluator’s findings and conclusions. The Evaluation Team Response: Statements provided in the Finding and Conclusions section of the Executive Summary reflect individual findings and summary conclusions in the body of the Report. When findings are presented, they are preceded by quoting the source of information for that finding. Such findings were not a subject of the evaluation team’s assessment as they present information received by the evaluation team. The evaluation team used qualitative analysis of KIIs data to arrive at its conclusions, which are presented in the Conclusion sections in the body of the Report. Due to the space limitation imposed on the Executive Summary, individual Findings and Conclusions throughout the body of the report are jointly presented in one Finding and Conclusion section in the Executive summary. For the sample provided in the comment, after addressing the IP’s comment #1, the quoted paragraph reads: “The JA’s interventions have been appropriate and met beneficiaries’ needs, to the extent possible given available resources, in institutional strengthening for prosecuting and processing CEC cases. KIs confirmed that the JA’s design appropriately tailored implementing activities to meet the beneficiaries’ needs, while POs and other stakeholder KIs positively noted the JA’s ability to adapt and tailor assistance to the diverse demands, needs, and operational dynamics of the POs. …” Therefore, the first sentence reflects the evaluation team’s assessment, while the second sentence backs it up with the finding established on our analysis of KIs statements. IP’s Comment #3: Further, Millennium DPI is satisfied with the recognition given by the evaluators of the highly challenging environment in which the JA operates (p. 2 and pp. 11–14) with all its limitations, and systemic issues, including political conditions, that significantly impact the JA's work. Contextualizing the JA's efforts and 158 interventions provides a clearer picture of the circumstances that are outside of JA’s control and that have affected implementation (such as other donor activities or the fact that the specialized departments of the FBiH PO and the FBiH Supreme Court were not established by the BiH government as anticipated). Millennium DPI believes it would strengthen the usefulness of the evaluation if both in the Executive Summary and in the chapter called “Context in Which JA Operates”, there was some analysis of how these circumstances and challenges affected or influenced the performance of the JA. We believe that this would add value to the Draft Report and provide a better understanding of the MEASURE-BiH opinion on the JA’s performance in the context of BiH’s complex legal and political environment. While the evaluation team concludes that the JA’s implementation required flexibility and adaptation, an assessment of the ability of the JA to provide this flexibility and achieve results in the existing context is not given. The Evaluation Team Response: The evaluation team made changes to page 2 paragraph 1 to insert the omitted conclusion related to operating context and to include the conclusion about the JA’s flexibility and adaptation. Furthermore, the “Context in which JA operates” section on page 10 paragraph 3 and 4 and page 11 paragraph 1 and 2 and Finding 6 explain how changes in circumstances and challenges affected the work of the JA and how the JA adjusted to unforeseen circumstances (namely, changes were made through Work Plans and their consequent implementation). IP’s Comment #4: Further, in the opinion of Millennium DPI, the Executive Summary’s Finding and Conclusions identify certain shortcomings that are not relevant to the mandate of the JA and, without being sufficiently explained, could lead to the incorrect conclusion that these were JA shortcomings. For example, the statements such as (T)he evaluation team identified a noticeable lack of communication among stakeholders in the judicial sector and KIs pointed out a number of issues that negatively influence the work of justice sector institutions, including disciplinary proceedings, appointments, performance appraisals, current legislative and regulatory initiatives, and knowledge of issues and needs among judicial institutions, judges, and prosecutors (p.4), without giving further explanation on how these circumstance impacted the JA’s work, could mislead the reader to conclude that they are to be attributed to the JA’s performance. The Evaluation Team Response: Changes were made to the report to reflect that this information was received from KIs and although not related to the JA mandate, provide valuable information for USAID/BiH and are therefore presented in the report. IP’s Comment #5: Below are Millennium DPI’s comments on specific parts of the Draft Report. Updates. A number of JA activities and achievements have taken place since the period when the Draft Report was produced and therefore certain parts of the text do not reflect the current state of affairs. The JA did not make interventions in this regard assuming that the date on the document will remain the same (September 2018). However, if the date were to be changed, updates in the text would also need to be made as they have bearing on the accomplishments and effectiveness of the JA. 159 The Evaluation Team Response: The date of the final evaluation report will be changed to reflect timeframe in which USAID/BiH and the IP submitted comments, and the date of the evaluation team’s response. Nevertheless, the report itself will continue to reflect data collected by the evaluation team in period June 13 – July 27, 2018. IP’s Comment #6: “Duplication” of donor efforts. In several places in the Draft Report it is noted that the JA and INL￾financed OSCE ARC project duplicated their efforts in corruption trial monitoring and verdict analysis (p. 3 para 1, p. 6 para 4 in the Executive Summary). There are also parts of the report which imply that the two projects failed to coordinate their efforts effectively (p. 16 para 2, p. 18 para 4, and p. 48 para 1). This is incorrect. Millennium DPI conducted an analysis of court verdicts in corruption, while the OSCE ARC project carried out trial monitoring. There is a significant difference in how data is derived from these two approaches. While both projects came to more or less similar conclusions regarding overall deficiencies in prosecuting corruption, each project used its own methodology and produced different sets of data. “There are significant differences are in the methodology used: 1. USAID’s report is more extensive and based on a larger sample of cases. It first provides a detailed quantitative (statistical) assessment of trends in corruption cases over the three year period (on a sample of 416 cases/614 decisions rendered in this period) and then gives a qualitative analysis of 56 selected finalized court cases (140 decisions of all instances). The total sample of decisions analyzed includes all corruption cases prosecuted in BiH during the referenced time (2015–2017). The OSCE Report provides a very brief mapping of the 153 cases currently monitored by the mission (from January to November of 2017 as part of the INL project), and then focuses on 67 cases finalized in the period of seven years in BiH (2010–2017) which were subject to monitoring by the OSCE mission in the past (random selection of cases, not based on any particular or adopted criteria), and based on which all findings and recommendations are made. 2. USAID’s analysis deals separately with the four most commonly prosecuted forms of corruption (bribery, abuse of office, lack of commitment in office and embezzlement in office) based on a representative sample of such cases and it does NOT use names of the accused in the cited cases. The OSCE Report gives joint findings for all types of cases, and provides a very brief summary of the 67 cases with names of the accused persons. 3. Findings and recommendations of the OSCE Report are directed at several groups of actors: 1) the legislature and the executive branch, 2) HJPC, 3) judges, 4) prosecutors. USAID’s Analysis is mostly directed at members of the judicial community, primarily prosecutors, since the purpose was to identify flaws in their investigative and prosecutorial approaches in proving corruption and increasing the overall success in such cases. It could therefore be used more as a practical guidebook to proving elements of crimes while the OSCE report serves a different and broader public. These efforts should be seen as complementary and equally important to the justice sector, rather than as a duplication of work or ineffective use of donor resources. Millennium DPI believes it should be noted that the two projects put maximum effort towards coordinating their activities to avoid duplication of efforts and that the purposes of each project are in fact different. The purpose of the USAID funded Analysis, for example, is to assist prosecutors identify flaws in their investigative and prosecutorial approaches in proving corruption and thus increasing their overall success in such cases. It could therefore be used more as a practical guidebook to proving elements of crimes while the OSCE report serves the 160 broader public by providing an analysis of the root causes of obstacles to fair and effective case processing at the legislative, institutional and individual levels. At the onset of the OSCE ARC project, its representatives met with the JA (in September 2016) announcing the start of the project in January 2017. At that time the JA’s Analysis of Court Verdicts was far into the drafting stage (data had been collected since 2015, and the trends analysis was already being analyzed and drafted). The JA did its best to inform the other donor on this already initiated activity (started in 2015) as well as its purpose and target audience. During the course of 2017 the two projects met again to make sure their approaches did not overlap. It was beyond the JA's power to influence interventions planned by the INL as the donor of the OSCE ARC Trial monitoring project. We believe that the Draft Report should acknowledge the JA's efforts to coordinate the two projects, and should also include an explanation of the difference between trial monitoring and verdict analysis. The Evaluation Team Response: Wording in the report was changed to inform USAID/BIH that there are similarities in tasks related to monitoring of trials and analysis of judgments/verdicts in corruption cases between an USAID and an INL financed project. This is based on the fact that both the JA and the OSCE sampled a similar pool of verdicts and that both organizations attempted to “capture the most prominent cases” within the sample they selected. The wording in the report was also changed to reflect the evaluation team’s conclusion that USAID/BiH and INL should further examine similarities and differences between efforts, but in particular, areas for improved joint work of the JA and the OSCE ARC project, and possibilities for joint presentation of results. IP’s Comment #7: The “most useful” JA activities. In different sections of the Draft Report a reference is made to the KIs’ impression of the most successful and useful JA activities. In two places in the Executive Summary this is presented differently: on p. 2 para 3 it is stated that KIs identified the most useful elements of the JA to be the provision of experts in forensic accounting, specialized training, material assistance, the development of Integrity Plans, and the Guidelines for Preventing Conflict of Interest, including Asset Declaration Forms, while on p. 4 para 3, the development and adoption of Integrity Plans is recognized as the most successful activity. In another place (p. 15 para 3) it is also the development of Integrity Plans, and the Guidelines for Preventing Conflict of Interest including Asset Declaration Forms that are recognized as the most successful JA activities. Further on p. 17 para 2, the same conclusion is presented in a different way, with (i) provision of experts in economics and forensic accounting at the first place of the most useful interventions, followed by (ii) specialized trainings for prosecutors and trainings for PR personnel; (iii) material assistance with IT, other equipment, and furniture; (iv) technical assistance in developing Integrity Plans; and (v) technical assistance in developing the Guidelines for Preventing Conflict of Interest. Finally, the evaluation team (on p. 17 para 3) makes its own estimate of the most relevant activities, also putting the provision of experts in economics and forensic accounting and specialized trainings for prosecutors and PR personnel at the first place of the most useful JA interventions. These inconsistencies should be addressed. It could be useful to specify whether different groups of KIs made these different estimates (i.e., those working only with Component 1 or only with Component 2, which could be a potential explanation for the discrepancies). The Evaluation Team Response: Through careful examination of the quotes in the JA comment, the evaluation team feels that the quotes are separated from the context in which they are referred to or disconnected from related paragraphs explaining different types of assistance. For example, the evaluation team finds that the JA quote from the 161 comment above, which reads: “while on page 4 paragraph 3, the development and adoption of Integrity Plans is recognized as the most successful activity,” is different from the original text, which reads: “In the development and adoption of Integrity Plans, the JA exceeded expectations stipulated in the contract and the MEL Plan; KIs consistently cited this as one of the most successful JA activities.” Furthermore, when the evaluation team is providing findings or conclusions related to the overall JA intervention, the most valuable types of assistance are quoted in the same place (or connected paragraphs). The part of the report that refers to results of the JA Components (EQ2) is organized by the outcomes, which belong to separate JA components. When providing findings or conclusions related to individual outcomes, the evaluation team singled out only the most useful types of assistance as quoted by KIs, for the outcome discussed. Additionally, to clarify that the most useful types of assistance are not in any way ranked by KIs (and as such presented in the report), we removed their numbering (i) through (v) where it appeared in the report. Finally, the report has been adjusted to note which institutions/bodies included in the KIIs are considered direct beneficiaries and which are considered stakeholders. Furthermore, when describing findings based on qualitative data throughout the report, we provided a more details about the number of KIs describing each item. IP’s Comment #8: Translation of diagnostic assessments. It is noted in the Draft Report that the JA’s major diagnostic studies were not translated into English, which prevents non-local-language speakers from understanding the documents. This is stated first in the Executive Summary (p. 4 para 4), and subsequently on p. 44, para 1, and p. 48 para 4 under Opportunity no. 4, where it is also noted that the dissemination of diagnostic reports was not effective because only a limited number are available on the JA web site. The report recommends that USAID should translate major diagnostic studies prepared by the JA into English and make them available to USAID/BIH, at a minimum. Corrections should be made to these statements considering that all of the JA analyses have been put on the JA website since the moment of their finalization, making them widely available to stakeholders, and to USAID BiH as well. In addition, the JA organized public events for the promotion of each of its analyses inviting the most relevant stakeholders. All the analyses were printed and sent to the most important stakeholders and target audience. Complete translations were not made of these documents due to the excessive costs involved and because the JA primary partners and beneficiaries did not require a translation. It was determined that it was sufficient to have the executive summaries of each analysis (containing all major findings) and recommendations translated for the donor community. The statements made in the draft report should be amended to reflect these facts. The Evaluation Team Response: The evaluation team’s conclusion is also based on a donor KI statement which pointed out that members of the international community in BiH who are tasked with influencing local policy decision making, are deprived of information contained in the JA studies. In return, they cannot help support implementation of the JA’s recommendations. We also found that the non-local-language speakers are unable to access information on the JA’s web site (see screen shot of ENG and BCS version below). While the local language version contains all studies (their names, and content), the English version of the JA site, does not, posing a limitation on non-local-language speakers. 162 The JA web site - BCS version: The JA web site - ENG version: Finally, it is the evaluation team’s understanding that the JA contract requirement in the section A.14 spells out that all deliverables shall be produced in English. The JA contract does not specify a mechanism for exceptions to these requirements. IP’s Comment #9: JA work related to transparency. The draft report does not contain information about the third major activity conducted by the JA which involved improving the transparency of the PO’s public communications. Specifically, p. 26 under Finding 9 omits a description of the extensive assistance JA provided to the POs to improve the quality and transparency of their public relations. The JA developed the Draft Plan for Monitoring and Evaluation of Public Relations of POs with the public, which was finalized in cooperation with the SDC, and subsequently adopted by the HJPC in 2018. This mechanism, which represents a system wide change, monitors both the quantity and quality of information provided to the public by Public Information Officers of the courts and POs. To reinforce these efforts NGOs have been engaged as an assessment body to measure the quality of the information being disseminated by POs. Indicators related to the Public Relations of POs became part of the POs’ strategic plans and as such they became a regular topic of the Collegiums of the Chief Prosecutors. The only mention of these activities in the Report (on p. 30 under Finding 20) incorrectly states that the JA developed the PR MEL plan for HJPC, which was adopted by the HJPC in September 2017. There is no explanation given as to the nature of this document, implying that it is an M&E plan for the HJPC instead of for POs, endorsed /passed by the HJPC. Furthermore, the report does not mention templates for publishing information via POs official websites that were developed by JA in the first half of 2018. These templates are considered particularly useful for PR officers (that are not professional spokespersons), since it enables them to prepare quality information for publishing via official POs websites as well as via other communication channels. The Evaluation Team Response: The evaluation team made adjustments to findings 8, 9 and 20 of the report. IP’s Comment # 10: Ex-Component 2. The Research Questions contain an inquiry related to the termination of the original Component 2 and the overall impact that the JA had in this area of justice sector reform. We believe that 163 the report should take into account the extraordinary efforts invested in developing the Diagnostic Assessment on System of Enforcement of Final Claims in BiH and the impact that this document has had on other donor activities, such as the HJPC’s project on Improving Court Efficiency and Accountability of Judges and Prosecutors in BiH, which was designed based on the findings and recommendations of the JA analysis. The Evaluation Team Response: The IP’s comment quote: “The Research Questions contain an inquiry related to the termination of the original Component 2 and the overall impact that the JA had in this area of justice sector reform.” The evaluation team finds that the research questions do not contain this inquiry. More specifically, EQ2 states (on pages 6 and 97 (the SoW)): “To what extent has the JA achieved expected results under both components as outlined by the contract, and what are the prospects of meeting life-of-activity targets specified by the monitoring and evaluation plan?”. Related to the former Component 2, the evaluation team presented its findings and conclusions in accordance with the information obtained through KIIs and the document review. Please see the evaluation team’s response to USAID/BIH comment #6. IP’s Comment # 11: Reference to INL training. With regard to the conclusion that beneficiaries, among others, found the provision of on-site and off-site trainings as one of the most useful elements of the JA, including those provided by experts on financial/forensic accounting, the JA cannot identify clear reasons for the statement made under Findings and Conclusions in the Executive Summary (p. 4 para 5, repeated on p. 45 para 2), that the upcoming training from INL in forensic accounting presents an opportunity for staff from POs and courts to improve knowledge and skills in this area. The same is repeated under the Recommendations in the Executive Summary (p. 6 para 2) where an additional assertion is put forth that formal trainings should be organized for police officers and conducted by prosecutors (it is not clear why the upcoming INL program is in the executive part of the report dealing with key recommendations for the JA's further work). There seems to be a lack of understanding that the core of the JA's on-the-job trainings provided to prosecutors by JA experts was in the area of financial investigations and forensic accounting. The statement/recommendation made in the Draft Report implies that the JA omitted or failed to provide training to prosecutors on forensic accounting, and that the INL will be introducing such support for the first time. Furthermore, the lack of explanation as to how the upcoming JA activities in the final year of its implementation could be incorporated into the INL program, leaves the above remarks all the more puzzling. It should also be noted that the JA, according to the clear mandate of this project and its area of activities in relation to other USAID projects and US Embassy departments (such as the US Department of Justice program ICITAP) is prohibited from working with the police, i.e., training them, which is being proposed in the Draft Report. The Evaluation Team Response: Page 5 paragraph 2 of the draft report reads: “Despite the extensive onsite and offsite training provided by the JA, which beneficiaries welcomed and praised, the team’s analysis of KIIs showed that prosecutors, judges, and law enforcement agencies still lack expertise related to processing economic crime and corruption cases, as well as cases related to cybercrime. Further training in CEC is needed for prosecutors, judges, and police to address issues that have emerged systematically when processing CEC cases. Given the lack of expertise for providing CEC training in BiH, continued donor engagement in this area may be necessary. Upcoming training from INL in forensic accounting represents an opportunity for staff from POs and courts to improve their knowledge and skills in this area.” 164 We disagree that this statement could lead to the IP’s conclusion that: “The statement / recommendation made in the Draft Report implies that the JA omitted or failed to provide training to prosecutors on forensic accounting, and that the INL will be introducing such support for the first time. Furthermore, the lack of explanation as to how the upcoming JA activities in the final year of its implementation could be incorporated into the INL program, leaves the above remarks all the more puzzling.” To make the evaluation team’s implications more clear, we changed the last sentence in the quoted paragraph to read: “To build upon the JA delivered training, upcoming training from INL in forensic accounting represents an opportunity for staff from POs to further improve their knowledge and skills in this area.” (Note: the reference to this paragraph in the final report is on page 4 paragraph 5). IP’s Comment # 12: Lack of awareness on ODC activities. On p. 4 para 6 a statement is made that Judges and prosecutors are unaware of ODC’s activities and practice, and POs are unaware of good practices in processing CEC cases within the existing regulatory framework. These statements are not accurate. The Evaluation Team Response: Please see the evaluation team’s response to IP’s comment #4. IP’s Comment # 13: PO lack of awareness of good practices. In addition, the statement that POs are unaware of good practices is far too strong in light of the JA’s continued efforts to provide specialized prosecutorial trainings (praised by the evaluators as the most useful activity) where best practices are discussed and useful tools for their dissemination created. The Evaluation Team Response: This language was adjusted as presented in the final report. IP’s Comment # 14: HJPC field visits to POs. In addition, in the same paragraph it is stated that HJPC members rarely visit individual POs to disseminate information on ongoing HJPC activities and receive information on PO needs. It is not clear why this statement is in the Executive Summary, considering that visits of the HJPC members are not something the JA could influence. Moreover, the JA made considerable effort to keep both sides informed about developments on each side, such as changes in the HJPC Law and the POs’ needs. The Evaluation Team Response: The language was adjusted as presented in our response to IP’s comment #4. Please see the evaluation team’s response to IP’s comment #4 and modifications in the report. 165 IP’s Comment # 15: Recommendations. In the Recommendations section of the Executive Summary it is unclear how some of the recommendations could be implemented during the remainder of the JA project (such as extending support to courts, focusing on certain regions, and working with police). This is unrealistic, both with regard to the limited time remaining to finish the project, but also from the perspective of the existing contract, with its legal limitations and mandate. This could potentially be remedied by focusing such recommendations not on the JA in Year 5 but rather on USAID’s overall planned interventions in the justice sector. The Evaluation Team Response: Please see response to the USAID/BiH comment #5. IP’s Comment # 16: Sustainability of strategic planning within POs. Although in Finding 13, the MEASURE found that the JA assistance helped POs to substantially increase their technical capacity to manage CEOCC cases — referring to the progress made in strategic planning efforts — in the same finding it points to a statement by an HJPC representative that without further support, the sustainability of strategic planning will be questionable. The JA questions a conclusion based on one person's comment and provides support for a conclusion that sustainability has been achieved. On October 10, 2016, the HJPC adopted the Instruction on Implementation of the Book of Rules on Criteria for Measuring Work of Prosecutors in POs and related Guidelines (adopted in November 2017). This introduced an obligation for all POs to include in their Annual Reports their achievements in implementing strategic objectives in accordance with the strategic planning methodology which was developed by the JA. This important systemic change ensures long term sustainability of strategic planning and the JA considers it one of its major successes. It also marks the achievement of one of the defined contractual milestones for the Activity 1.1. The Evaluation Team Response: The evaluation team modified Finding 13, by deleting this sentence. IP’s Comment # 17: Response bias. On p. 8, under Limitations, it is stated that there is a Response bias: The majority of the key informants had direct interaction with the JA and therefore may overstate the positive effects and understate the negative effects of its interventions. It is not clear how this conclusion can logically be made. The fact that KIs worked directly with the JA can theoretically influence the opinions of KIs both positively and negatively, depending on the JA’s performance. The report should either omit the reference to response bias or indicate the possibility that KIs could also have a negative bias and understate the JA’s achievements. The Evaluation Team Response: The quoted Response bias was replaced with Social desirability bias. In addition, Lack of complete information limitation was added on page 7. 166 IP’s Comment # 18: Support of the JA in implementation of the EU peer review recommendations. The work of the JA on amendments to the Law on the HJPC, in line with EU Peer Review recommendations, is not accurately portrayed in the Report (e.g., p. 5 para 3 in the Findings and Conclusions section) as an accomplishment which sets the foundation for future justice sector reform. In the Report this work is tied to three sets of guidelines, which the JA helped develop in the first two years. These are the guidelines on integrity plans, the guidelines on conflict of interest, and the guidelines on disciplinary sanctioning (p. 42 para 2 in Finding 47). As indicated, this a quote from the HJPC letter. Please see footnote 37! While the process of the development of these guidelines and advocacy for their adoption and subsequently implementation, may have served as a way to identify some of the deficiencies of the current Law on the HJPC, the work of the JA on amendments to the Law on the HJPC was larger in scope and relied on other JA work as well, such as the Diagnostic Analysis of the Integrity of Judicial Sector and Potential Corruption Risks, the Functional Analysis of the ODC, and the asset declaration reform. Finding 1, at the beginning of the report establishes that three sets of guidelines are consequent studies of the DA. While the DA is not formally adopted by HJPC, the three sets of guidelines, as presented in Finding 4, are formally adopted by the HJPC. This support to the HJPC in implementing the EC Peer Review recommendation also includes the drafting of the Book of Rules on the submission, verification, and management of asset declaration of judges and prosecutors (BoR). The work of the JA on reforming the asset declaration system for judges and prosecutors was mentioned only in passing in Finding 43 of the Report (p. 41 para 1), in conjunction with guidelines on conflict of interest. The need to reform the asset declaration system of judges and prosecutors was first recommended in the JA Diagnostic Analysis of the Integrity of Judicial Sector and Potential Corruption Risks, and was singled out as one of the priorities of USAID during the workplan development for Year 3 of the JA. As a result, the JA invested significant effort in working with the HJPC on reforming the asset declaration rules by drafting the BoR, which will enable the use of asset declarations to combat corruption within the framework of the current Law on the HJPC. That said, the JA would like to use this opportunity to inform USAID that the BoR was adopted by the HJPC on September 26, 2018. Finally, the JA helped the HJPC draft amendments to the Law on the HJPC which will further reform the asset declaration system. The Evaluation Team Response: The JA support for implementation of the EU peer review recommendations is thoroughly discussed throughout the report. We did make additional changes in findings 43 and 47 to further describe the related TA. IP’s Comment # 19: Support to the ODC. The support of the JA to the ODC was evaluated as successful. The report notes that the achievements were not always recognized by some KIs (p. 37 para 2 in Finding 37). Similarly, the report notes that KIs are rarely aware of JA activities related to changes to ethics codes (p. 40, Finding 46). The reasons for the lack of recognition of the JA’s work in the field of disciplinary reform and professional ethics are manifold and deserve mention in the Report, as follows: 167 a. The work of the ODC, which is responsible for investigating disciplinary breaches and bringing disciplinary charges against judges and prosecutors, is by its own nature not popular among judges and prosecutors; b. The HJPC, which decides on disciplinary charges through its disciplinary panels, has not been sufficiently transparent (one example was the resistance of the HJPC to publicize its Disciplinary Sanctioning Guidelines [adopted in 2016] which the JA helped develop); c. In the past the HJPC did not invest sufficient effort in education on professional ethics. The Evaluation Team Response: Please see response to IP’s comment #4. IP’s Comment # 20: The JA has been fully aware of these issues, and has been working hard with the HJPC (the HJPC Working Group on Integrity and Accountability and the HJPC Secretariat legal department) and the ODC to strengthen the disciplinary system. To that end: a. The JA developed the draft Judicial Discipline Benchbook and draft Ethics Handbook, in partnership with the HJPC, which was not mentioned in the Findings but was mentioned in Annex 1 – 2.4.1 (p. 65 para 2) of the Report. The adoption of the Benchbook and the Handbook by the HJPC is planned in the course of 2019 (the HJPC target adoption date is February 2019). Both are the first such publications in BiH and follow the EU Peer Review recommendations. It should be noted that the Ethics Handbook was adopted by the HJPC in January 2019. b. The JA supported the HJPC in the development of three sets of guidelines (on integrity plans, conflict of interest and disciplinary sanctioning), which were adopted in 2016, and recognized by the Report as achievements. c. The JA monitors the implementation of the disciplinary sanctioning guidelines, to make sure the disciplinary sanctioning is clear, fair, and consistent, which has been the biggest stated concern of judges and prosecutors about the disciplinary system. d. The JA worked with the HJPC and the ODC on improving the transparency of the disciplinary system by: producing case summaries of all disciplinary decisions since establishment of the HJPC and the ODC, which will form part of the Judicial Discipline Benchbook, proposing ways to improve the publication of disciplinary decisions on the HJPC website, helping the ODC to develop a communications strategy and to improve its website, as well as providing communications training to the ODC. Finally, the JA proposed amendments to the Law on the HJPC relevant to improving the transparency of the disciplinary system. e. The JA supported the drafting of amendments to the Law on the HJPC to improve the enforcement of the professional ethics standards. f. The JA provided first ever trainings to members of the HJPC, who also serve on disciplinary panels, on the application of Disciplinary Sanctioning Guidelines and on how to write an explanation of a disciplinary decision, which will contribute to the transparency of the disciplinary system. Millennium DPI Partners believes that the JA’s efforts to overcome ingrained attitudes towards judicial ethics through an improved disciplinary system have been effective and that this needs to be acknowledged in the Draft Report. 168 The Evaluation Team Response: The evaluation team believes that the second part of the JA comment in the “Support to ODC” section is associated with output 2.4 and 2.5, thus the evaluation team will respond to it in a separate comment. We find that “Judicial Discipline Benchbook” and draft “Ethics Handbook” were mentioned as part of the second group of the JA MEL indicators discussed on page 22. This text further states: “These indicators cover efforts that the JA directly controls, and results indicate that there has been consistent improvement over the JA implementation period. The JA was on track to achieve the activity’s expected results in 2018 or to reach their 2019 (the life-of-activity) results. In some cases the JA has already met the 2019 (the life-of-activity) targets. Annex I provides details on these indicators as well.” Furthermore, this group of MEL indicators is mentioned in the Executive Summary on page 3 paragraph 3. Regarding outcome 2.4, we made changes to finding 40 by adding a new paragraph. Regarding outcome 2.5, we made changes to findings 46 and 47. In addition, please see the team’s response to IP’s comment #4. Finally, although the delivered the JA assistance was valued by ODC, the JA performance indicator (Indicator 2.4.2) that measures judges’ and prosecutors’ agreement that the disciplinary process is fair and impartial has a low value and experienced a downturn in Y3 compared to its baseline value. Consequently, the evaluation team can acknowledge that the TA delivered was useful to the ODC, but it did not receive the recognition of judges and prosecutors to be considered as effective. IP’s Comment # 21:  Judicial Discipline Benchbook and Ethics Handbook. In addition to the Judicial Discipline Benchbook and Ethics Handbook, the JA drafted the Universal Benchbook on how to prosecute and adjudicate corruption, and organized and economic crime (Universal Benchbook). Its publication is expected in the first quarter of Year 5. The Universal Benchbook was not mentioned in the Evaluation Report even though Millennium DPI provided the evaluation team with information about the drafting process and with the draft itself. The Universal Benchbook incorporates the discussions, recommendations, and lessons learned from the specialized trainings and as such presents a valuable instrument to secure the sustainability of this educational effort. The development of the Benchbook was preceded by a comprehensive training needs assessment, and the development and implementation of a long-term specialized training — the first of its kind in BiH. The Evaluation Team Response: Changes were made in findings 48 and 49. IP’s Comment # 22:  Implementation of Integrity Plans. The JA wishes to correct the statement in the Executive Summary Recommendations section that the JA will support full implementation of integrity plans in pilot courts and POs (p. 6 para 4 and p. 41 para 1 in Finding 43). As noted in the Year 4 JA workplan, the JA provided assistance to two pilot institutions (one court and one PO) in 169 developing integrity plans, as well as supported the implementation of a selected measure from the integrity plan of a pilot court. The JA also supported HJPC efforts to monitor the development of integrity plans in courts and POs and will help the HJPC monitor their implementation. As part of the latter, the JA will help the HJPC disseminate good practices, as recommended in the Report. The Evaluation Team Response: Page 6 paragraph 3 of the draft report reads: “USAID should consider continuing support for HJPC in implementing Integrity Plans and, to the extent possible, replicate the achievements from two pilot locations where the JA will support full implementation of Integrity Plans in as many other judicial institutions as possible.” Finding 43 on page 39 paragraph 1 the draft report reads: “In addition, the JA committed itself to support the implementation of Integrity Plans in two locations (one court and one PO).” The evaluation team does not find any discrepancy between the content of the evaluation report and the IP’s comments. In accordance with the response to USAID/BiH’s comment #5, the recommendation stated in the first paragraph of this response is removed. 170 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) Fra Anđela Zvizdovića 1 UNITIC Tower B, Floor 21 71000 Sarajevo Bosnia and Herzegovina PHONE: + (387) 033 941 676 contacts@measurebih.com www.measurebih.com