PERFORMANCE EVALUATION OF THE SEPARATION OF POWERS (SPP) PROGRAM Final Evaluation Report 20 November 2014 This report was produced for review by the United States Agency for International Development. It was prepared independently by Marilyn Zelin and Robert Lochary of Social Impact, Inc. Cover drawing: Illustration by Novica Kocić in a May 19, 2014 Politika article about the “Spring Cleaning” in the Nis Basic Court. Courtesy of SPP. PERFORMANCE EVALUATION OF THE SEPARATION OF POWERS (SPP) PROGRAM FINAL EVALUATION REPORT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. November 20, 2014 This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc.to the United States Agency for International Development under Analytical Services IQC - Contract AID-OAA-I-10-000003 and Task Order number AID-169-TO-14-00001. 1 TABLE OF CONTENTS ACRONYMS.................................................................................................................2 EXECUTIVE SUMMARY ................................................................................................3 INTRODUCTION ..........................................................................................................1 Evaluation Purpose ............................................................................................................1 Evaluation Questions..........................................................................................................1 Project Background ............................................................................................................1 Evaluation Methods & Limitations......................................................................................3 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ......................................................4 EVALUATION QUESTION 1: .................................................................................................5 EVALUATION QUESTION 2: ...............................................................................................10 EVALUATION QUESTION 3: ...............................................................................................15 CHALLENGES.............................................................................................................19 REFERENCES .............................................................................................................21 APPENDICES .............................................................................................................23 ANNEX A: Evaluation Statement of Work..........................................................................23 ANNEX B: Persons and Agencies Contacted.......................................................................39 ANNEX C: Semi-Structured Interview Questions................................................................40 ANNEX D: Individual Court Statistics for USAID, March 2013..............................................42 ANNEX E: Court Efficiency Table.................................................................................43 ANNEX F: Comparative Statistics.......................................................................................44 ANNEX G: Evaluation Matrix.............................................................................................48 ANNEX H: Findings, Conclusions, Recommendations (FCR) ................................................50 ANNEX I: Final Success Story.............................................................................................57 ANNEX J: Comparative Statistics Spreadsheet ...................................................................60 2 ACRONYMS AVP Case management system used by the courts BPMIS Budget software used by the HCC CMS Case Management System EU European Union HCC High Court Council JA Judicial Academy MOJ Ministry of Justice NBRP National Backlog Reduction Plan NJRS National Judicial Reform Strategy PMP Performance Monitoring Plan QA Quality Assurance SCC Supreme Court of Cassation SI Social Impact, Inc. SSI Semi-Structured Interview SOW Scope of Work SPP Separation of Powers Program TPM Team Planning Meeting UBRP Uniform Backlog Reduction Program of the Supreme Court of Cassation USAID U.S. Agency for International Development 3 EXECUTIVE SUMMARY Evaluation Purpose From September to October 2014, Social Impact (SI) conducted a performance evaluation for USAID/Serbia of the backlog reduction and prevention work performed by the Separation of Powers Program (SPP). The purpose of this evaluation is to measure the results of work with the 10 SPP partner courts from 2010-2013 and provide recommendations to inform future court administration efforts in Serbia. The Social Impact (SI) Team conducted this evaluation in parallel with an assessment of the main case management software systems used by the Serbian judiciary (CMS Assessment Report). Therefore, this evaluation report draws on the findings included in that report. Project Background Recognizing the importance of court efficiency for genuine rule of law in Serbia, in 2008 USAID launched SPP to support improvements in court administration as well as judicial and legislative independence. SPP is a six-year program structured around three tasks. The focus of this evaluation is the backlog reduction and prevention activities included in Task 2 (specifically sub-task 2.2). During the SPP, the Serbian judiciary experienced significant reforms. The most important reforms for the purposes of this evaluation were implemented pursuant to the National Judicial Reform Strategy (2006-2011)—namely, reorganization of the judges’ network in 2009, in which 30% of the judges were dismissed, and reorganization of the court network in 2010. As a result of these changes, SPP had to select new partner courts in 2010, and during the first three years of the program partner courts were led by acting court presidents. After the elections in May 2012, the Government initiated another set of reforms for the judiciary, including preparation of a new, five-year, 2013-2018 National Judicial Reform Strategy (NJRS) and Action Plan, which the National Assembly passed in July 2013 and includes a National Backlog Reduction Plan (NBRP). The NBRP was prepared with the support of—and incorporates the achievements of—SPP. USAID amended the contract for SPP in December 2013 to extend SPP to consolidate best practices and transition them to host country partners and other donors, particularly the European Union (EU). The extension covers all three SPP tasks and the focus is to identify and address weaknesses of host country partners. With USAID’s assistance, the EU Delegation developed a 9 Million Euro judicial efficiency project that was intended to begin in 2014. That project has been delayed and the Evaluation Team was informed that the project will not begin for another six months. Evaluation Design, Methods and Limitations The Evaluation Team applied a mixed-methods approach using the standard rapid appraisal methods of materials review and semi-structured interviews of key informants. The Team 4 conducted approximately 30 interviews and also made site visits to six partner courts and seven non-partner courts in nine cities. A mixed-method approach and reliance on semi￾structured interviews, in particular, involves methodological limitations. These limitations include recall, response, and selection bias, which are often exacerbated by the technical nature of rule of law programs. Nonetheless, the Evaluation Team took several steps to mitigate such bias. As set forth in the workplan for this evaluation, the Team had planned to compare results and impact of backlog reduction and prevention in partner and non-partner courts based on quantitative data available to the Team. However, given difficulties in obtaining data as described later in this report and the limited time and resources allocated for this evaluation, the Team was only able to compile limited statistics. Summary of Findings, Conclusions and Recommendations Note: All recommendations are included under evaluation question 3. Evaluation Question 1: What were the results and impact of SPP’s interventions for case backlog reduction and prevention in partner courts, compared to courts that were not part of the program, from 2010 to 2013? Finding/Conclusion: The Evaluation Team found that partner and non-partner courts adopted SPP measures for backlog reduction and prevention to varying degrees. The Team concluded that there was marked difference between partner and non-partner courts in terms of the interventions they employed to reduce backlogs and increase efficiency. Both partner and non-partner courts employed obvious measures such as identifying old cases, labeling those case files, prioritizing them, and monitoring them. However, the partner courts also included more creative and pro-active interventions that linked the court to stakeholders in the community. Finding/Conclusion: The Team found that SPP’s interventions for case backlog reduction and prevention in partner courts were successful overall, although there were variations in year-to-year results. The Team concluded that the degree of improvement varied from court to court and year to year not only as a result of the extent to which individual courts implemented the measures proposed by SPP, but also to circumstances outside the court’s control. Finding/Conclusion: SPP’s partner courts, like all basic and higher courts, use AVP software. The system was installed in 2010 when the new court network was established. The Team found that numerous sources, including the SCC, court presidents, the World Bank, and other experts, maintain that there are errors in the AVP data and that the lack of reliable statistics is a major obstacle to measuring progress in achieving efficiency and backlog reduction. This is referred to in the CMS Assessment Report as GIGO (garbage in garbage out1 ). Notwithstanding concerns about data quality, the Evaluation Team did compile limited comparative statistics from data for 2011 and 2013 provided by SPP after the Team’s departure from Serbia. The statistics suggest that there was improvement in backlog reduction and backlog prevention (as measured by clearance rates) at both SPP partner and non-partner basic and higher courts as a whole between 2011 and 2013. However, given the concerns about data quality, limitations on analysis due to time and resources available for this evaluation, and differences in the profile and circumstances in 1 See discussion of data integrity in SI’s CMS Assessment Report. 5 individual courts, the Team has concluded that these statistics should not be taken as an indication that SPP’s interventions did not yield results. Evaluation Question 2: What were the key factors and lessons learned in generating these results, if any? Finding/Conclusion: The Team found that five key factors account for the successful results of SPP partner courts: 1) SPP selected partner courts that had challenges and thus would provide good examples of how interventions could work; 2) SPP introduced practical techniques for backlog reduction and prevention, and provided tools for successful implementation; 3) SPP allowed partner courts to select and own the improvement techniques; 4) SPP worked closely with partner courts and provided hands-on assistance; and 5) SPP closely monitored progress and followed up on any apparent obstacles to progress. The Evaluation Team concluded that the methodology SPP followed was key to its success in introducing new backlog reduction and prevention techniques in partner courts. Forgoing a “one size fits all” approach allowed partner courts to select methods they thought would best address efficiency and backlog problems in their courts. In turn, this resulted in partner courts investing in the success of the measures introduced. Finding/Conclusion: The Team found that the working relationship SPP developed with the MOJ and the SCC was an important factor in extending the SPP model of backlog reduction and prevention to all Serbian courts. SPP documented and leveraged the results of its work with the partner courts through publication and dissemination of the Best Practices Guide, assistance to the MOJ and SCC in drafting new Court Rules and procedural codes to reflect those practices, and finally through the NBRP. The Team concluded that using a top-down and bottom-up approach allowed partner courts to test and refine backlog reduction and efficiency interventions, disseminate successful measures to all courts, and achieve a measure of sustainability through rules mandating that all courts devise backlog reduction and efficiency plans, including targets, and report upon them. Finding/Conclusion: SPP employed a variety of techniques to share experiences between partner courts and non-partner courts. The Team found that for partner and non-partner courts, court-to-court cooperation was one of the most common and valued methods for court’s to share experience and lessons learned. The Team concludes that court-to-court cooperation is a key to successful sustainability of court efficiency and backlog reduction measures generated in partner courts through the SPP program. A regular exchange of information between partner courts that are proven champions of efficiency and backlog reduction and courts that are struggling to reduce backlogs is an effective and inexpensive method to disseminate SPP’s court efficiency and backlog reduction interventions. Finding/Conclusion: The Evaluation Team found that pursuant to the project design, SPP initially followed dual tracks: focused backlog reduction with one set of partner courts, and case management/backlog prevention activities that involved the full scope of court operations with a second set of courts. After approximately two years (the fourth year of the project), all partner courts began to follow both tracks. The Team concluded that SPP correctly identified that backlogs are a symptom of court inefficiency and instructed partner courts to work on backlog reduction and prevention. The Team further concluded that an initial focus on backlog reduction at all courts and then extending activities to include backlog prevention in all courts may have been more productive. 6 Finding/Conclusion: The Team finds that the Serbian courts are struggling to keep pace with the ongoing reforms, both administrative and procedural, in the Serbian judicial system since SPP began. As noted in the CMS Assessment Report, the sheer velocity of changes in the Serbian judiciary is extraordinary. All respondent court presidents stated that recent reforms, including new procedure codes, have resulted in confusion among judges in the basic, higher and appellate courts about how to apply the new laws and codes. The Team found that to address these problems, SPP has provided support for harmonization of jurisprudence and case management training based on the new procedural laws, including developing new training modules and conducting training for trainers. The Team learned that the case management training is not continuing. The Team has concluded that both training and harmonization efforts need to continue. Evaluation Question 3. Building upon key lessons learned, how can Serbian and international stakeholders expand and scale up these results nationally, in order to meet the implementation objectives outlined in Serbia’s National Judicial Reform Strategy? Based on the above findings and conclusions, the Evaluation Team offers the following recommendations: Recommendation: To address the problem of unreliable court statistics, a major obstacle to measuring progress in achieving efficiency and backlog reduction in Serbia which undermines Serbia’s efforts to comply with one of the principles of the NJRS—Efficiency, and achieve EU accession, the MOJ and Working Group should immediately develop a plan for manual verification of data to correct errors resulting from migration of cases in the 2014 court network reorganization. This should be followed by implementation of the recommendation in the CMS Assessment Report to develop reliable unified statistical reporting system at the national level. Further, the Working Group should immediately resume its duties to monitor and provide guidance on implementation of the UBRP. In that regard, the Working Group, in cooperation with the Judicial Academy (JA), should organize training for the UBRP Working Group and court presidents on how to run statistical reports, analyze data, set benchmarks, identify trends, assess staff performance, adjust staffing plans, and make other management decisions that will improve court efficiency based upon the data analysis. Training should also be organized for the SCC Statistics Department and court statisticians. Both trainings should be based upon the Guide to Statistical Reporting for Court Presidents: Strategies and Tools for Success (Statistics Manual) developed by SPP in 2013. Recommendation: To address the concern about whether the UBRP Working Group has the capacity to adequately monitor and guide implementation of the UBRP after SPP closes, the Evaluation Team recommends that pending initiation of the new EU IPA project, USAID consider engaging a local, rule of law-oriented organization to support continuation of the initiatives started by SPP during its final year. This could be accomplished through the USAID Serbia Civil Society Partners program. Recommendation: To have a positive impact on a large number of backlogged cases, the Evaluation Team recommends that the UBRP Working Group continue to focus on reducing the number of backlogged enforcement cases in the basic courts, which account for over 90% of all backlogged cases. A national program modeled on the Infostan settlement at the Belgrade First Basic Court would be a significant step toward clearing the existing 7 backlog, which is an obstacle to continued progress on Chapter 23 of the EU Accession process. Recommendation: Rules for changing territorial jurisdiction can be found in the Criminal and Civil Procedure Codes. Both allow the SCC, or in some criminal cases, a higher court, to reassign a case. The Team recommends that the SCC use existing law to reassign cases from courts with large backlogs to those with smaller backlogs. Changes to the law should be considered that would allow for temporary backlog reduction units of judges charged with solving old cases in overburdened courts. The use of plea bargaining, settlement and mediation in all courts should be encouraged. Further, the Team recommends that the UBRP Working Group require courts to include annual staffing plans that are based upon court efficiency and backlog reduction as part of their annual backlog reduction plans. Overall Conclusion: SPP initiated major progress in reducing backlog and improving efficiency in Serbia. Through focused work with ten partner courts, SPP was able to test and refine measures that were simple and could be implemented without additional human or financial resources, By documenting best practices, working with key judicial institutions to amend legislation to include those practices and develop a plan for replicating SPP’s success with partner courts on a national level, SPP has provided the Serbian judiciary with a framework for ongoing backlog reduction and improved efficiency. However, as with many projects, SPP’s national partners have come to rely on SPP and the Evaluation Team concludes that without some level of ongoing outside support, there could be a setback. 1 INTRODUCTION Evaluation Purpose The purpose of this performance evaluation is twofold: • to conduct a full, evidence-based and independent review of the results of the Separation of Powers Program (SPP)’s work from 2010 to 2013 in reducing and preventing backlog of cases in partner courts, and • to produce a report that provides a qualitative as well as quantitative evaluation of those activities and recommendations that will inform future court administration initiatives in Serbia. The Social Impact (SI) Team conducted this evaluation in parallel with an assessment of the main case management software systems used by the Serbian judiciary, and therefore draws on the findings included in the assessment report. The audience for this evaluation is USAID/Serbia, High Court Council (HCC), Ministry of Justice (MOJ), other donors, local and international judicial reform stakeholders, and USAID’s Europe and Eurasia Bureau and Center for Democracy, Rights and Governance. Evaluation Questions The evaluation addresses three questions: 1. What were the results and impact of SPP’s interventions for case backlog reduction and prevention in partner courts, compared to courts that were not part of the program, from 2010 to 2013? 2. What were the key factors and lessons learned in generating these results, if any? 3. Building upon key lessons learned, how can Serbian and international stakeholders expand and scale up these results nationally, to meet the implementation objectives outlined in Serbia’s National Judicial Reform Strategy? Although the sample size was not sufficient to conduct a true impact evaluation in response to evaluation question 1, SI employed techniques designed to provide proxies for standard impact information, as described below. At the in-brief on September 24, 2014 the Mission posed an additional question, “What accounted for the variation in results of partner courts?” Project Background Recognizing the importance of court efficiency for genuine rule of law in Serbia, in 2008 USAID launched SPP to support improvements in court administration as well as judicial and legislative independence. SPP is a six-year program structured around three tasks: 2 Task 1. Judicial branch financial independence: Developing the capacity of the Serbian judiciary to allocate, acquire, and manage its resources; Task 2. Improved court administration: Assisting the Serbian judiciary in making its administration of justice more efficient, transparent, and responsive to the needs of its users; and, Task 3. Legislative branch financial independence: Building the financial capacity of Serbia’s National Assembly. Together, these tasks support the Government of Serbia (GoS) in its effort to build the necessary institutional and democratic practices required for integration with the European Union. According to the Country Development Cooperation Strategy, 2013-2017, SPP activities fall under Development Objective 1: “Accountability of Key Democratic Institutions Strengthened and Intermediate Results,” IR 1.1: “Parliament and Local Governments are More Responsive to Citizens,” and IR 1.2: “Rule of Law and Oversight Improved.” During SPP, the Serbian judiciary experienced significant reforms. The first wave of reforms was implemented pursuant to the National Judicial Reform Strategy (2006-2011). This included establishing the High Court Council 2009, which undertook a major reconstruction of the judges’ network, dismissing almost 30% of all judges and a reconstruction of the court network pursuant to the 2008 Law on Court Organization. Under this law, the new Serbian court network reorganized the 138 Municipal Courts into 34 Basic Courts, the 30 District Courts into 26 Higher Courts, and created four Appeals Courts with general jurisdiction. In July 2012, the Constitutional Court of Serbia annulled the HCC decision on dismissal of judges and the approximately 300 non re-elected judges were reinstated. As a result of these changes, SPP had to select new partner courts in 2010, and during the first three years of the program, partner courts were headed by acting court presidents. After the elections in May 2012, the Government prepared a new set of judicial legislation, including the new court network law, which took effect in January 2014. There are now 25 Higher Courts and 66 Basic Courts. As part of that reform, in July 2013 the National Assembly passed the new, five-year 2013-2018 National Judicial Reform Strategy (NJRS) and Action Plan, which includes a National Backlog Reduction Plan (NBRP). According to the Action Plan, the immediate priorities are harmonization of jurisprudence, reduction of backlog of courts cases and equal distribution of the workload. The NBRP was prepared with the support of—and incorporates the achievements of—SPP. In another significant development, the European Council formally agreed on 28 June 2013 to continue open accession negotiations with Serbia. The 2013 EU Progress Report noted the following: in 2012, the backlog of courts cases was not reduced significantly (more than three million cases were still pending at the end of 2012); reform of the court network will require a comprehensive analysis—notably in terms of cost, efficiency and access to justice; and the quality of court statistics needs to be improved. The 2014 Report released on 8 October, 2014 again noted the backlog problem (with 2.8 million cases pending at the end of 2013) and pointing out that the situation with cases over two years old was particularly worrying (1.7 million cases, of which 1.2 million are enforcement cases); the current system of collecting court statistics is not efficient and does not allow making a meaningful analysis 3 of the performances of the Serbian Judicial system; and inconsistency in case law continues to be a concern, especially in the appeal courts. USAID amended the contract for SPP in December 2013 to extend SPP to consolidate best practices and transition them to host country partners and other donors, particularly the EU. The extension covers all three SPP tasks and the focus is to identify and address weaknesses of host country partners. With USAID’s assistance, the EU Delegation developed a 9 Million Euro judicial efficiency project that was intended to begin in 2014. That project has been delayed and the Evaluation Team was informed that the project will not begin for another six months. Evaluation Methods & Limitations The Evaluation Team initially reviewed the following background documents provided by the Mission: • SPP SOW • SPP annual and quarterly reports • SPP annual work-plans • SPP PMP reports and data • SPP Mid-term evaluation • SPP Best Practice Guide • NJRS and Action Plan • NBLRS and Action Plan • Relevant laws During the course of the Team’s field visit, SPP provided other documents requested by the Team, including SPP Monthly Reports for Year 6, materials prepared by SPP experts, and the Guide to Statistical Reporting for Court Presidents: Strategies and Tools for Success (Statistics Manual). The Team was able to view the BPMIS (budget software used by the HCC) database at the SPP office. The Team also conducted a limited review of the SCC Annual Reports for 2011 and 2012 since these reports are in Serbian, and examined AVP (case management system used by the MOJ) data and quarterly reports submitted by the courts at both the SCC statistical office and during site visits to courts. Finally, the Team obtained and reviewed the World Bank Multi-Donor Trust Fund Aide Memoir for the Functional Review of the Serbian judiciary and European Commission 2014 Progress Report for Serbia (2014 Progress Report). See References for a complete list of materials reviewed. The Evaluation Team conducted semi-structured interviews with direct and indirect beneficiaries, as well as with external observers to obtain information needed to answer the three evaluation questions posed by USAID. The Team also made site visits to six partner courts and seven non-partner courts located throughout Serbia. Finally, the Team attended the Judges’ Association Annual Meeting in Vrnjacka Banja from October 5-7, 2014 to conduct semi-structured interviews with more judges and attend presentations made by direct and indirect beneficiaries and external observers. 4 Semi-structured interview questions that the Team drafted in advance of fieldwork2 were slightly revised for each interviewee so as to tailor the scope of the interview to the position of the particular interviewee, as well as to reflect information learned during the course of the evaluation thus far. In addition, potential interviewees that the Team identified during the course of the evaluation were incorporated into the data collection schedule to the extent possible. The Evaluation Team initially prepared a draft survey to be distributed at the Judges Association Annual Meeting, although after further consideration and discussion with SPP, the Team concluded that it would be difficult to obtain accurate data from the survey. The Evaluation Team endeavored to limit recall, response and selection bias to the extent possible. To limit recall and response bias from partner courts, the Team asked specific questions about the SPP backlog reduction and prevention activities and about examples of experiences with new case management procedures. The Team probed into broad responses to verify respondent familiarity with the material discussed. The same was true with regard to questions about the Uniform Backlog Reduction Program (UBRP). Furthermore, the Team stressed that lessons learned from SPP activities would benefit non-partner courts. In terms of selection bias, although the Evaluation Team did consult with SPP on which partner and non-partner courts to visit, the Team made the final decisions and included courts that had not been suggested by SPP. Further, the Evaluation Team used multiple sources of data to triangulate on each evaluation issue. By combining information from multiple sources (i.e., documents, interviews, and reports), the effect of biases on the analysis will be mitigated as much as possible. With regard to gender equity, more than half of the interviewees were female, including Court Presidents and judges at the SCC. As set forth in the workplan for this evaluation, the Team had planned to compare results and impact of backlog reduction and prevention in partner and non-partner courts based on quantitative data available to the Team. The Team had expected that the data provided in the SCC Annual Reports for 2011 and 2012 provided by the Mission would be supplemented by data in a format that would be easy for the Team members to analyze as the time and resources available for the evaluation did not encompass data compilation. With the assistance of SPP, the Team obtained Excel spreadsheets with data for 2011 and 2013 after the Team had already departed from Serbia. The Team used this data to prepare limited comparative statistics, which are discussed under Evaluation Question 1 and included in Annex G. FINDINGS, CONCLUSIONS & RECOMMENDATIONS The following section presents the Evaluation Team’s findings, conclusions and recommendations. 2 See Annex C. 5 EVALUATION QUESTION 1: What were the results and impact of SPP’s interventions on case backlog reduction and prevention in partner courts, compared to courts that were not part of the program, from 2010 to 2013? Finding: The team found that partner and non-partner courts adopted SPP measures for backlog reduction and prevention to varying degrees. A summary of the use of these measures based upon interviews conducted by the Evaluation Team follows Backlog Reduction Team: All six partner courts that the Team visited had a backlog reduction team in place that was formal, active, and contained representatives from the judiciary as well as from the registry office. In five of seven non-partner courts visited, court presidents stated that teams were in place. However, upon further inquiry, it appeared that the Teams were informal in all but two of the non-partner courts, and that the court president used staff meetings (as opposed to formal teams) to promote backlog reduction and efficiency measures. Identifying and labeling old cases. The Team found that the most efficient technique for reducing backlog was to identify and label old cases and then for the court president to monitor progress on these cases. All six partner courts visited by the Evaluation Team used this technique. The presidents of those six courts also required the oldest cases to be given priority. The court presidents in partner courts acknowledged that judges avoided working on old cases because they are difficult to resolve. As a result, these cases simply aged and the backlog increased. The President of the Nis Basic Court reported that due to labeling, judges know these cases are under scrutiny and that “they must be handled.” Judges in two partner courts who were not court presidents noted that peer pressure served as motivation, since judges would be embarrassed to be seen with case files marked as “old”. The President of the Basic Court in Pancevo, a non-partner court, immediately took measures to reduce backlog after her appointment in September 2013. She has a bulletin board in her office with a separate posting for each judge listing his or her backlog cases and the status of the cases. Several courts have successfully implemented “backlog reduction weeks.” The Nis Basic Court’s “Spring Cleaning” was the first example of this. Improving Delivery/Service of Process: Five of six court presidents in partner courts where site visits were conducted stated that they had reformed the process of delivering court documents as a result of SPP’s program, streamlining service of process and cutting down on delays. In three courts (the Higher and Basic Courts in Subotica and the Basic Court in Uzice), this involved direct cooperation with the post office. In one non-partner court, the president implemented improved service of process through the post office after learning about this from partner courts. The remaining non-partner courts did not implement reforms in their service of process. Cooperation with External Partners: All but one of the six partner courts visited for this evaluation had communicated with external partners, such as prosecutors, defense attorneys, the police, the prisons, or centers for social welfare to coordinate and better facilitate hearings and to cut down on continuances. Three partner courts had contacted each of these partners and reached agreements with each of them that accelerated the court process, 6 although in one of these partner courts, the agreements lapsed after a new president was appointed. All five partner court presidents who had used these agreements cited them as effective methods for ensuring the presence of accused and of witnesses; only two of the non-partner courts visited used them and did so after learning about them from partner courts. The Team learned that cooperation with external institutions (social work center, psychiatric hospital and police) as well as public outreach programs (brochures and open court hours) that contributed to an increase in efficiency and reduction in backlog at one of the partner courts (Vrsac) was terminated when the court replaced its president. As this happened late in 2013, the Team did not have data to determine if efficiency or backlog reduction was affected. Efficient Scheduling: All partner and non-partner courts visited stated that they monitor the scheduling of backlogged cases. All partner courts visited stated they did so as a result of SPP interventions. All non-partner courts stated that they did so as a result of the requirements imposed by the NBRP. E-Justice: Five of six partner court presidents listed some form of electronic assistance or communications to parties or citizens who use the court. This usually involved email communications with parties and witnesses, as well as web sites where court documents and schedules can be found. All five implemented the measures as a result of the SPP program. One of the six had stopped using its E-justice measures when a new court president was appointed. None of the non-partner courts listed use of E-justice measures as an intervention that they use in their courts. Use of New Criminal & Civil Procedure Codes: All partner courts visited had tried to take advantage of new provisions in the civil and criminal procedure codes, such as use of plea bargaining. However, pursuant to the law, in all courts visited, use of plea bargaining appeared to be largely left to individual judges. Both partner and non-partner courts noted that plea bargaining is only applicable in new cases. This will lead to a reduction in future criminal cases, but has not been an effective tool for current backlogged cases. Conclusion: There was marked difference between partner and non-partner courts in terms of the interventions they employed to reduce backlogs and increase efficiency. Both partner and non-partner courts employed obvious measures such as identifying old cases, labeling those case files, prioritizing them, and monitoring them through meetings that included an agenda item requiring discussion of backlogs. However, the partner courts also included more creative and pro-active interventions that linked the court to stakeholders in the community, such as agreements with the post office, police, prosecutors, advocates, prisons and centers for social care. These measures are largely aimed at ensuring the conditions necessary for hearings and trials to take place rather than be continued, thus pushing cases through to final resolution. Some partner courts have established points of contact with outside stakeholders that help streamline and ensure future communications. As a result of these proactive efficiency interventions in partner courts, there is greater transparency in the work of the courts and strengthening of the partnerships within the community the court serves. The interventions in partner courts have caught the attention of reform-minded presidents of non-partner courts, such as the presidents of the basic courts in Valjevo and Pancevo, and Novi Sad, who have initiated court to court contact with the presidents of partner courts to learn about the SPP interventions and how to apply them in their courts. 7 The SPP program has thus facilitated communication necessary for the spread of best practices throughout the Serbian court network without the expense of formal training. Finding: SPP’s partner courts as a whole exceeded targets for the period 2010-2013. As reflected in the individual court statistics for the period 2010-2013 (see Annex D), the results at the partner courts were not uniform—they varied by court and year. All partner courts improved case processing efficiency from 2010 to 2013 (see Annex E). However on a year to year basis the results are: all courts improved case processing efficiency in 2011; eight improved in 2012; and the two that declined in 2012 improved in 2013, while efficiency at four other courts declined. Eight courts reduced their total backlog from 2010 to 2013, while backlog at the other two courts increased. On a yearly basis, in 2011 backlog increased in two courts and declined in eight; in 2012 backlog increased at four courts; and reductions occurred in all courts in 2013. During its site visits, the Team learned that each court is subject to specific conditions that affect its overall backlog reduction and prevention results as measured by SPP, including size and location of court, human resources (both judges and staff), types of cases and major inflows, attitude of court presidents and judges regarding reform, and extent of cooperation of external partners.3 A few illustrative examples follow. • The Uzice Basic Court began its backlog reduction and prevention program with criminal cases. The Team learned that the average caseload for a criminal judge decreased from about 400 cases in 2010 to 46 in 2014; there are only 22 backlog cases and no case is older than five years. However, although the backload reduction and prevention program was expanded to include the civil department in 2011, that department is still struggling with its backlog and has eight cases over 10 years old. The Team learned that the department recently received a large number of new cases in which farmers in the Zlatibor region are suing the state for restitution. Although the overall efficiency rate for the Uzice Basic Court increased from an 89% case completion rate in 2010 to a 102% completion rate in 2013, the rate for the criminal department alone is even higher. • In the Basic Court in Sremska Mitrovica, following a dramatic increase of investigative cases, SPP presented the court president with a projection showing that backlogged cases would continue to grow unless two judges were reassigned to the investigative department. SPP based these projections upon statistics that the program had refined through work with the court registry department. Although initially reluctant to reassign judges, the projections finally convinced the court president to reassign two judges to the investigative department. The backlog was finally reduced, although not before the overall number of backlogged cases increased from 2,601 in 2010 to 5,468 at the end of 2012. However, as a result of this intervention, the backlog declined to 3,040 by the end of 2013. • Three court presidents in partner courts noted that a lack of physical space for judges to work in and to hold hearings compromised their ability to move cases forward. Site visits confirmed these statements. • The Higher Court in Subotica received a large number of cases in the 2010 migration, including first instance cases. By the end of 2010, the court had very few 3 Although SPP kept data disaggregated by case type (civil and criminal) for 2011, the Team learned that such data was not compiled for 2012 or 2013. 8 first instance cases as a result of using plea bargaining in criminal cases and assigning 15 judicial trainees to a new preparatory department. Conclusion: The Team concluded that although backlog reduction and efficiency improved in the partner courts as a whole, and exceeded the SPP targets, the degree of improvement varied from court to court and year to year not only as a result of the extent to which individual courts implemented the measures proposed by SPP, but also to circumstances outside the control of the courts, such as large and sudden inflows of certain types of cases. The Team further concluded that if the measures described above had not been implemented, partner courts such as Uzice and Sremska Mitrovica would almost certainly have had larger backlogs. Finding: SPP’s partner courts, like all basic and higher courts, use AVP software. The system was installed in 2010 when the new court network was established. The Team found that numerous sources, including the SCC, court presidents, the World Bank, and other experts, maintain that there are errors in the AVP data and that the lack of reliable statistics is a major obstacle to measuring progress in achieving efficiency and backlog reduction. This is the result of errors in recording the migration of cases in 2010, compounded by errors in data entry—particularly during the period when AVP was new, 2010-2011. The Head of the SCC’s IT department reported that courts do not have unified versions of the AVP software, so data may not be comparable. Furthermore, the Team learned that the statistics included in the SCC’s Annual Reports are not extracted from AVP as there is no connectivity between individual courts and the SCC’s statistics department. The statistics are based on data in Excel sheets completed by the individual courts, signed by the court presidents and forwarded to the SCC. Interviews with the SCC Statistics Department revealed that quality controls imposed upon AVP data submitted by individual courts are limited. Interviews with court presidents and IT personnel in partner courts revealed that SPP worked closely with its partner courts to verify and correct data. In interviewing court presidents in non-partner courts, the Team found that this verification and correction of data was not done with the non-partner courts, although respondents in non-partner and partner courts agreed that case data in AVP began to improve after 2011 as they became familiar with the new AVP system, the new network, and new reporting requirements. Additional discussion of the AVP system is included in the CMS Assessment Report. Notwithstanding concerns about data quality, the Evaluation Team compiled limited comparative statistics from data for 2011 and 2013 provided by SPP after the Team’s departure from Serbia (See Annex G). The data provided by SPP includes information on all case types separately, as well as categories of cases, such as the most common types of cases in the basic and higher courts. The Team compiled statistics on the most common types of cases because that category is intended to exclude enforcement cases, which were not within SPP’s mandate. The following statistics compare the data for partner and non-partner courts as of January 1, 2011 and December 31, 2013. Basic Court Backlog 1/1/11 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Clearance Rate (%) 2011 Clearance Rate (%) 2013 Uzice 322 469 45.7% 91.5% 103.1% Cacak 1217 1,256 3.2% 98.6% 90.8% 9 Vranje 3307 2,173 (34.3)% 90.2% 118.5% Nis 1322 1,484 1.2% 103.0% 96.2% Vrsac 265 129 (51.3)% 86.8% 117.9% Basic Court Backlog 1/1/11 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Clearance Rate (%) 2011 Clearance Rate (%) 2013 Sremska Mitrovica 2947 2,264 (23.2)% 95.0% 137.0% Subotica 2008 621 (69.1)% 106.4% 117.1% Basic Court Backlog 1/1/11 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Clearance Rate (%) 2011 Clearance Rate (%) 2013 Total SPP Partner Basic Courts 11,388 8,396 (26.3)% 97.78% 111.12% Total Non￾Partner Basic Courts 50,908 37,240 (26.8)% 95.02% 108.32% Comparing 2011 and 2013, both SPP partner and non-partner basic courts as a whole decreased backlog by approximately 26% and improved clearance rates by approximately 13 percentage points. However, as the table shows, there were significant differences in the results among partner courts. Higher Court Backlog 1/1/11 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Clearance Rate (%) 2011 Clearance Rate (%) 2013 Belgrade 2,741 673 (75.4)% 105.5% 138.2% Novi Pazar 47 11 (76.6)% 110.3% 110.8% Subotica 28 8 (71.4)% 121.1% 91.3% Total SPP Partner Higher Courts 2,816 692 (75.4)% 106.7% 133.81% Total Non￾Partner Higher Courts 1,099 514 (53.2)% 102.7% 103.04% Comparing 2011 to 2013, both the SPP partner and non-partner higher courts as a whole decreased backlog significantly and improved clearance rates. The backlog reduction at SPP partner courts was higher than at the non-partner higher courts as a whole (75.4% vs. 53.2%). Similarly, the improvement in the clearance rate for SPP partner higher courts was greater than for non-partner courts (27.11 percentage points vs. 0.34). However, the large improvement at SPP Partner Higher Courts was due, at least in part, to the transfer of cases out of the Belgrade Higher Court. 10 Conclusion: The Team concluded that the statistics presented above suggest that there was improvement in backlog reduction and backlog prevention (as measured by clearance rates) at both SPP partner and non-partner basic and higher courts as a whole between 2011 and 2013. However, these statistics should not be taken as an indication that SPP’s interventions did not yield results. First, by 2012, some of the measures introduced by SPP at partner courts were already applicable to all courts: e.g. amendments to the criminal and civil procedure codes; recommendation by MOJ that all courts use the SPP template for backlog reduction planning; and publication of the Best Practices Guide. Second, as previously noted, each court is affected by specific factors that it may not have control over, such as large inflows of cases. Third, the number of cases filed in the Serbian courts has been declining, which should have a positive effect on clearance rates and backlog reduction. Finally, based on the concerns regarding data quality generally and the limits on data analysis discussed earlier, the Team notes that these comparative statistics are a doubtful quantitative measure of the relative progress in backlog reduction and prevention at partner and non￾partner courts. EVALUATION QUESTION 2: What were the key factors and lessons learned in generating these results, if any? Finding: The Team found that there were five key factors that account for the successful results of SPP partner courts: • SPP selected partner courts that were facing challenges and thus would provide good examples of how interventions could work in other courts. • SPP introduced practical techniques for backlog reduction and prevention and provided tools for successful implementation • SPP allowed partner courts to select and own techniques • SPP worked closely with partner courts and provided hands-on assistance • SPP closely monitored progress and followed up on any apparent obstacles to progress. The Team learned that SPP announced a call for court applicants, received 25 applications, visited the candidate courts and then selected partner courts. The selection was designed so that the partner courts manifested a variety of characteristics to test the success of backlog reduction and prevention techniques in different environments. This was corroborated by site visits to partner courts. The presidents of three partner courts selected in 2010 confirmed that SPP required a formal application process and also examined the population and ethnic make-up in their territory prior to selecting them as partners. The Team found that SPP selected courts with “real” backlog and efficiency problems to demonstrate that the techniques were effective even under difficult conditions. The Team learned that early in the program, SPP staff reviewed documents related to the previous USAID-supported backlog reduction program implemented by NCSC to determine lessons learned, and concluded that a practical approach was necessary. Four court presidents (in the Higher and Basic Court in Subotica and the Basic Courts in Uzice and Vrsac) interviewed for this evaluation stressed that a key to their success at improving 11 efficiency and reducing backlogs was that SPP offered a menu of possible solutions to partner courts, asked them to focus their efforts on case types that were most problematic in their courts, and then allowed the individual partner courts to choose which measures the court would test. The Team interviewed representatives from the court registry offices in these four locations, who confirmed the statements of their court presidents. In a fifth court (the Basic Court in Nis), although the court president was not present in 2010, the court manager confirmed that the menu of options offered by SPP and the latitude to pursue them was a key to the court’s success, allowing for local solutions for local problems. The techniques offered by SPP were simple solutions that courts could implement without a significant investment or additional resources. The Team found that these solutions had not previously been considered by partner courts. The President of the Nis Basic Court said that “SPP widened my horizons”, although he noted that some of the measures could not be used under the current system. A detailed discussion of these techniques and their use by partner courts appears in the next finding. All SPP partner court judges and staff reported that SPP spent significant time with them over the course of the project, making at least quarterly site visits of several days to all partner courts. The Team learned that at the outset of the program, the SPP Chief of Party joined the site visits, but after that, site-visits and monitoring were conducted by national staff. Court presidents and registry staff at all partner courts visited noted that SPP staff provided hands-on assistance to them in an attempt to resolve inaccuracies and ensure reliable reporting. In an effort to refine reporting on case backlogs and efficiency, SPP developed tracking and reporting templates that partner courts used to gather relevant case reporting data and submit to SPP on a monthly basis. All court presidents and judges of partner courts interviewed for this evaluation who worked with SPP cited this approach to monitoring as a key to their successes in reducing backlogs and increasing efficiency. The Team also found that SPP helped the partner courts to set realistic targets. For example, respondents at the Nis Basic Court reported that the original backlog reduction target that it set in 2010 was 50%, but SPP convinced the backlog reduction working group to set a reasonable standard of 20%. Conclusion: The Evaluation Team concluded that the methodology SPP followed was key to its success in introducing new backlog reduction and prevention techniques in partner courts. Forgoing a “one size fits all” approach allowed partner courts to select methods they thought would best address efficiency and backlog problems in their courts. In turn, this resulted in partner courts investing in the success of the measures introduced. Further, SPP taught the courts how to analyze their problems to develop targeted solutions, a skill that they can use to resolve future problems. Finding: The Team found that the working relationship that SPP developed with the MOJ and the SCC was a key factor in extending the SPP model of backlog reduction and prevention to all Serbian courts. Early in the project, SPP participated in the working group that revised the Book of Court Rules as well as the NJRS sub-group tasked with developing an action plan to support the strategy’s efficiency pillar. All court presidents interviewed for this evaluation stated that they reference the Book of Court Rules provisions related to backlog reduction and prevention in their everyday work. The MOJ incorporated SPP’s ideas into new criminal and civil procedure codes. 12 In further support of the NJRS, SPP created a draft NBRP in July 2013 that provides clear guidance on implementation of the Plan. SPP also created a Statistics Manual, which explains the importance of gathering and maintaining statistics in the courts, as well as concepts of creating and using case management indicators, and three new reporting modules (old cases, average duration of pending cases, and closed cases) for the AVP system. In December 2013, the President of the SCC passed the UBRP that was based upon lessons learned from SPP’s work with its 10 partner courts and SPP’s Draft National Backlog Reduction Plan. The UBRP provides backlog reduction objectives for all the courts of Serbia and incorporates the reporting requirements articulated in the Statistics Manual. It requires the President of the SCC to chair a Working Group that is charged with analyzing quarterly reports from individual courts and proposing measures necessary to improve the process of reducing the number of backlogged cases. Every non-partner court president respondent noted that the requirement to prepare backload reduction and prevention plans imposed by the SCC, along with specific instructions on how to deal with old cases, was a key motivation for their programs. However, two non-partner court presidents reported that they had initiated backlog reduction activities as far back as 2010 and that they were motivated by a sense of responsibility to properly manage their courts. The President of the Basic Court in Pozega reported that he had successfully adopted measures that were outlined in the SPP Best Practices Guide. He noted that the key to backlog reduction and prevention was proper organization of the court and hard work. The Working Group was established in December 2013 and held its first meeting in January 2014. It includes representatives from the SCC, the Administrative Court, the Commercial Appellate Court, the Higher Courts and the Basic Courts. There are no representatives of the HCC in the Working Group. Most members were proposed by SPP and selected on the basis of their previous contribution to SPP backlog reduction and prevention activities. The Team learned that two of the original members had already resigned and that the SCC was trying to expand the Working Group and ensure that there was sufficient representation from all of the appellate regions. The Working Group met again in February to review backload reduction plans and statistics submitted by all courts. However, due to the poor quality of many of the individual plans received by non-partner courts and of the statistics (due in part to problems with migration of cases in the AVP software, a responsibility of the MOJ), SPP cancelled the Working Group’s March meeting and organized a retreat for the group in April. The Working Group has not met since then and is focused on having the MOJ resolve the migration problem so the Working Group has accurate statistics. Conclusion: The Team concludes that using a top-down and bottom-up approach allowed partner courts to test and refine backlog reduction and efficiency interventions, disseminate successful measures to all courts, as well as achieve a measure of sustainability through rules mandating that all courts devise backlog reduction and efficiency plans, including targets, and report upon them. However, the Team also concluded that SPP provides all of the technical and analytical support to the Working Group and that the Working Group is reliant on SPP for this fundamental support. Because SPP is a driver of reform efforts at the 13 top-down level, the Evaluation Team questions whether efforts are sustainable after SPP closes at the end of the year. This is discussed further under evaluation question 3. Finding: The Team found that for partner and non-partner courts, court-to-court cooperation was one of the most common methods for court’s to share experience and lessons learned. SPP employed a variety of techniques to share experiences between partner courts and non-partner courts. This included formal measures such as publication of the Best Practices Guide, court-to-court workshops and roundtables, as well as less formal methods such as inter-court meetings. Five of six partner courts visited by the Evaluation Team cited a need for future court to court exchanges that would promote backlog reduction and prevention through judges sharing lessons learned. Three of the five stated that progress in implementing efficiency measures in courts nationwide were likely to falter once the SPP project ended, and that the technique of having judges from partner courts share their experiences with judges from non-partner courts would help maintain momentum for these reforms. The presidents of five of seven non-partner courts interviewed by the Evaluation Team agreed. They cited court to court roundtables and exchanges between judges from partner courts and those from non-partner courts as necessary in the future if progress is to be made in court efficiency and backlog reduction nationwide. Two court presidents in non-partner courts who knew about SPP and had undertaken efficiency measures contained in the Best Practices Guide stated that they did so because they had contacted their counterparts in nearby partner courts, and those partner court presidents had helped them devise and implement efficiency interventions. Court presidents in partner courts and their team members confirmed during interviews that they had aided presidents in non-partner courts with lessons learned from their SPP partnership. For example, the President of the Nis Basic Court reported that Basic Courts in nearby towns (Leskovac, Pirot and Aleksinac) had requested advice via telephone. However, he noted that the small courts needed different techniques than the larger courts, and that the Nis Basic Court cooperated with the Basic Court in Novi Sad. During the site-visit to the Basic Court in Novi Sad, a non-partner court, the Team learned that a delegation from that court had visited the Cacak Basic Court to consult on enforcement cases. Many court presidents and members of backload reduction working groups noted that they now reach out to other courts to discuss specific issues. Conclusion: The Team concludes that court-to-court cooperation is a key to successful sustainability of court efficiency and backlog reduction measures generated in partner courts through the SPP program. A regular exchange of information between partner courts that are proven champions of efficiency and backlog reduction and courts that are struggling to reduce backlogs is an effective and inexpensive method to disseminate SPP’s court efficiency and backlog reduction interventions. Court to court roundtables are an effective way of introducing non-partner judges to efficiency interventions. These can be followed up by having the efficiency teams from struggling courts visit more proficient courts to observe techniques, then apply them in their courts. Teams from more proficient partner courts can then conduct site visits and offer assistance and guidance as the reforms take hold. Finding: After the reorganization of the court network in 2010, SPP selected new partner courts—five for backlog reduction and five for backlog prevention—and continued with the dual track approach incorporated into the original project design. However, after working with the partner courts for two years, it became clear to SPP and the courts that in fact 14 measures applied to reduce backlogs also served as efficiency measures that helped prevent backlogs. Therefore, after approximately two years (the fourth year of the project), SPP started to work on both backlog reduction and prevention in all 10 partner courts. All respondent court presidents in partner and non-partner courts listed efficiency measures as the key to reducing backlogs. These included more efficient service of process, court presidents requiring judges to prioritize difficult and old cases, and holding judges’ meetings to discuss and resolve issues in difficult cases. A judge at the Belgrade Higher Court, which was in the original backlog prevention group, noted that backlog reduction should have begun earlier in SPP. Conclusion: The Team concluded that SPP correctly identified that backlogs are a symptom of court inefficiency. The Team further concluded that an initial focus on backlog reduction at all courts and then extending activities to include backlog prevention in all courts may have been more productive. Thus, to reduce backlogs and prevent them, courts must become more efficient. Given the state of the Serbian judiciary, with a large number of new judges and courts, common sense solutions to practical efficiency problems were adequate to increase efficiency in partner courts. Given their practical nature, most SPP interventions are applicable in non-partner courts as well. Finding: The Team finds that the Serbian courts are struggling to keep pace with the ongoing reforms in the Serbian judicial system. As noted in the CMS Assessment Report, the sheer velocity of changes in the Serbian judiciary is extraordinary. Many court presidents and judges stated they were negatively impacted by constant changes in law, judges and the court network. All respondent court presidents stated that recent reforms, including new procedure codes and laws, have resulted in confusion among judges in the basic, higher and appellate courts about how to apply the new laws and codes. Two judges from partner high courts, the head of the Criminal Department of the SCC, the head of the Civil Department of the SCC, two appellate court judges in Nis, the president of the Basic Court in Valjevo, and the presidents of SPP partner courts specifically mentioned that jurisprudence in Serbia must be harmonized to reduce uncertainty and thereby reduce the amount of time it takes for lower court judges to make decisions as well as reduce the number of cases returned on appeal. The appellate court judges and MOJ representative interviewed for this evaluation stated that the four appellate courts struggle to consistently decide cases that are factually similar. The Team found that SPP started to support harmonization efforts in the final year of the project and is facilitating meetings between appellate court judges. SPP has urged that these efforts continue in the future, especially in regards to guidance in the large number of backlogged enforcement cases. During the 2014 Judges’ Association Annual Meeting, the SCC acknowledged the importance of addressing the problem and recognized SPP’s help as crucial to initiating the process. The lack of harmonization is a significant problem as noted in the 2014 EU Progress Report and must be addressed for Serbia to progress toward EU integration. The Team also found that the training available to judges has not kept pace with legislative reforms. One department head noted that as the judges were being trained on revisions to the Criminal Procedure Code in 2012, additional revisions had already passed changing provisions they had just received training about. The Team notes that at the request of the JA, SPP updated its case management training curricula in 2012 to reflect changes in the civil 15 and criminal procedure laws. In conjunction with updating the curricula, SPP organized a training of trainers for eight appellate court judges, two from each appellate district. The Evaluation Team learned that some training for trainees and sitting judges was conducted in 2013, but no case management training has been conducted for sitting judges since that time. Conclusion: The Team has concluded that constant upheavals associated with judicial reform efforts since the 2006 Constitution have kept courts off-balance and focused on basic issues related to new courts, new judges, case migration and application of new laws. This has negatively impacted their ability to fully benefit from backlog prevention and reduction measures introduced by SPP and incorporated in the NBRP. The Team has concluded that additional support for harmonization efforts as well as training is needed. With respect to training, based on the relatively low level of capacity of the JA, the Team has concluded that additional support to the JA is needed4 . Further support for harmonization and training are discussed under evaluation question 3. EVALUATION QUESTION 3: Building upon key lessons learned, how can Serbian and international stakeholders expand and scale up these results nationally, in order to meet the implementation objectives outlined in Serbia’s National Judicial Reform Strategy? Finding: As discussed under Evaluation Question 1, the lack of reliable court statistics is a major obstacle to measuring progress in achieving efficiency and backlog reduction in Serbia, which undermines Serbia’s efforts to comply with one of the principles of the NJRS— Efficiency and achieve EU accession. The Team learned that the UBRP Working Group tasked with monitoring implementation of the NBRP and proposing measures to improve implementation (the main body for sustaining the work of SPP in backlog reduction and prevention) cancelled its March meeting (the third monthly meeting) due to the poor quality of many of the individual plans received by non-partner courts and of the statistics. The Working Group maintains it cannot continue its work until the MOJ corrects the errors in data resulting from the migration of cases as a result of the 2014 court network reorganization and courts can prepare the required periodic reports correctly. A representative of the Working Group reported that the data provided by non-partner courts after the 2014 reorganization was unusable and that court presidents need training in data collection and reporting. Further, at the Judges Association Annual Meeting, she stated that looking at this data was like being in a fog. A related issue that is discussed in the CMS Assessment Report is the limited IT capacity in the judicial sector5 . Conclusion: The Team has concluded that addressing the first problem, correcting data to accurately reflect migration of cases, requires manual verification of data as described in the CMS Assessment Report, while the second problem, poorly prepared backlog reduction plans, requires training for court presidents and staff involved in preparation of annual backlog reduction plans and the periodic reports the courts submit to the SCC. Further, the 4 See SPP’s Organizational Capacity Assessment of the Judicial Academy, 2014. 5 See discussion of human resources challenges in the CMS Assessment Report, 16 Team agrees with the conclusions and recommendations in the CMS Assessment Report regarding development of a reliable unified statistical reporting system at the national level and the need to build the capacity of judicial sector IT personnel. Recommendation: The Evaluation Team recommends that the MOJ and UBRP Working Group immediately develop a plan for manual verification of data. The plan should consider what can be accomplished with internal resources and to what extent external resources are necessary. The Team recommends that the Working Group consider the techniques SPP employed to help partner courts “cleanse” data after the 2010 court network reorganization and replicate useful techniques to the extent practicable. To the extent external resources are necessary, USAID, together with other donors, should consider supporting that effort. If this cannot be accomplished prior to the launch of the EU IPA project, then the data cleansing process should be one of the first activities in that project. The Team further recommends that the UBRP Working Group develop a plan, in coordination with the JA, for providing training on report preparation to court presidents and staff using the SPP Statistics Manual. Given the limited capacity of the JA as noted in the 2014 Judicial Academy Organizational Capacity Assessment prepared by SPP, this effort will require donor technical and financial support. In that regard, the Working Group could seek assistance from the local organization discussed below if USAID or another donor elects to support such project. Finding: The Evaluation Team found that consistent with the terms of the extension of SPP for an additional year, SPP undertook several activities to consolidate best practices and transition them to host country partners and other donors (e.g. EU) during the past year. With respect to backlog reduction and prevention and harmonization of jurisprudence, this was the subject of a Memorandum of Understanding between SPP and the SCC signed in March 2014. Notwithstanding SPP’s efforts, the Team found that there is considerable concern about whether the UBRP Working Group can maintain the progress SPP has made to date in implementing the NBRP. One of the key members of the Working Group stated that the Working Group has relied on SPP up to this point to gather and refine data on court efficiency and backlogs, and to prepare reports that contain analyses. This member stated that SPP acts as a secretariat for the Working Group, helping them collect and analyze data and focus on next steps. She expressed concern regarding whether Working Group members, who have full time jobs as judges, would have the time or special skills needed to prepare these reports once SPP closes, and more importantly, to analyze them and draw the correct conclusions based upon the analysis. The Team found similar concerns among presidents of partner and non-partner courts. Three presidents of SPP partner courts who had worked with SPP since 2010 and had implemented most of the successful interventions that resulted in the Best Practices Guide expressed concern that when SPP closes, the Working Group will need assistance to liaise with courts to gather accurate statistics and make decisions based upon them. The presidents of two non-partner courts, both of whom had actively communicated with partner courts to learn about SPP interventions and implement them in their courts, stated that they had not received assistance from the Working Group and were concerned about how much the Working Group could accomplish without SPP’s assistance. These sentiments were echoed 17 by representatives of the MOJ during interviews and at the final session of the 2014 Judges’ Association Annual Meeting. Further, as discussed earlier, the Team found that the UBRP Working Group has not met since April 2014, and it is continuing to rely on the support of SPP. Although the IPA project was intended to continue to support backlog reduction and prevention efforts, that project is behind schedule. The Team learned that the EU Delegation expects the project to begin within six months; however there is no guarantee. Conclusion. The Team has concluded that the UBRP Working Group will suffer without SPP support, particularly due to the delay in the IPA project. Further, as non-partner court presidents cited the requirement from the SCC President that they focus on efficiency and backlog reduction as the main reason for their efforts at implementing these reforms, the Team has concluded that if the Working Group fails to closely monitor and provide individual guidance to courts in the near term, it is possible that further progress will not be made. The Team recognizes that meeting backlog reduction and prevention targets are not evaluation criteria pursuant to the Rule on Performance Evaluation of Judges adopted on July 22, 2014 and therefore, there are no direct consequences for individual judges or court presidents if backlog reduction and prevention targets are not met. In this regard, members of the UBRP Working Group raised the possibility of initiating disciplinary actions based on the failure to comply with backlog reduction and prevention requirements. Recommendation: Pending initiation of the new EU IPA project, USAID should consider providing a grant through the Serbia Civil Society Partners program to a local rule of law￾oriented organization to support continuation of the initiatives started by SPP during its final year. This would include support for the UBRP Working Group and the SCC President to maintain momentum on backlog reduction and prevention, as well as for harmonizing jurisprudence. Some illustrative activities are described below. • Facilitate cooperation between courts and external partners on initiatives under consideration by the Working Group, including coordinating with the Ministry of the Interior to gain access to property registers using national personal identification numbers would facilitate service of process and concluding a new agreement with the Postal Service of Serbia to improve the delivery of court documents.6 • Assist the Working Group to implement the SCC President’s order of May 2014 requiring court presidents to resolve cases that have lasted more than 10 years. At the end of 2013, there appeared to be 150,978 cases in this category. Of these, 133,780 were enforcement cases in the basic courts. Given the unreliability of data in AVP, the Working Group will likely need assistance identifying these cases, facilitating communication with all the responsible court presidents, and monitoring progress. • Facilitate court-to-court activities, which have a multiplier effect as court presidents, judges and staff in partner courts share their experience with colleagues. There should be a regular exchange of information between proven champions of efficiency and backlog reduction measures and courts that are struggling to reduce backlogs. • Organize training, in cooperation with the JA, for court presidents and relevant staff on preparation of backlog reduction plans and periodic reports using the SPP 6 Report of the Working Group for the Implementation of the Uniform Backlog Reduction Program of the Supreme Court of Cassation. 18 Statistics Manual. Assist the JA in resuming case management training using the training materials on the Criminal and Civil Procedure Laws developed by, and the appellate court judge trainers trained by, SPP. The Team also suggests that USAID consider whether any of these activities could be assigned to the Judicial Reform and Government Accountability (JRGA) program. The Team further recommends that the Working Group invite representatives of the HCC to join the Working Group. This would give the Working Group additional leverage and might help to secure changes that would support the Working Group such as granting Working Group members a reduction in their caseload quotas in recognition of time devoted to the Working Group; including satisfaction of backlog reduction targets in the criteria for evaluation of judges; and obtaining resources to establish a small secretariat. Finding: According to the Annual Statistics Report of the SCC, at the end of 2013 the courts had 1,772,149 backlogged cases. 1,687,875 of these cases were in the basic courts, of which, 1,615,830 were enforcement cases7 . The Evaluation Team found that basic court presidents were preoccupied with enforcement cases Creditors in enforcement cases often have several cases pending against single debtors. Many of these are utility cases. The Belgrade First Basic Court, with SPP support, tackled this problem when Infostan, the largest utility, entered into payment agreements with 80,000 debtors, resulting in the removal of 60,000 backlog cases from the court’s records. Conclusion: The Team has concluded that while enforcement cases do not require much work on the part of judges, closing them will allow a clearer picture to emerge of the true extent of the backlog problem and have a positive impact in terms of how outside stakeholders assess the problem of case backlogs in Serbia. The law regarding private bailiffs will likely have a positive impact on resolving new enforcement cases, but does not apply to old backlogged enforcement cases. Recommendation: To have a positive impact on a large number of backlogged cases, the Evaluation Team recommends that the UBRP Working Group continue to focus on reducing the number of backlogged enforcement cases in the basic courts. Implementing a nationwide program modeled on the Belgrade Basic Court’s settlement of backlogged utilities cases would significantly reduce the number of backlogged cases. Further, the MOJ should examine the possibility of approaching public utility companies to encourage them to write off small amounts of debt (e.g., up to 500 RSD) and withdraw their request for enforcement in these cases, an initiative proposed by the SCC in May 2014. Amendments of the Law on Enforcement allowing private bailiffs to collect on debts in old cases as well as new cases, and amnesty for debtors that would allow them to make a partial payment of the amount they owe in return for closing the cases against them should also be considered Finding: Respondents at partner and non-partner courts noted techniques to reduce backlog and improve efficiency that are rarely used, including transferring cases from very busy courts to less busy courts and use of alternative measures such as settlement, plea bargaining and mediation. For example, after the 2010 reorganization of the court network, cases from the Higher Courts in Belgrade and Novi Sad were transferred to the Higher 7 Report of the Working Group for the Implementation of the Uniform Backlog Reduction Program of the Supreme Court of Cassation. 19 Courts in Subotica and Pancevo. Based on that experience, judges at the Higher Court in Pancevo suggested transferring old cases to experienced judges who could devote all their time to those cases. Another possibility would be to engage retired judges, but this would require specific authorization from the HCC. Within courts, the Team found a lack of uniformity in assignment of staff to relieve judges of administrative procedures, such as assigning judicial assistants to a specific judge or judges or establishing preparatory departments. Judges in all partner courts noted that the use of novelties in procedural codes, such as plea bargaining, settlement and mediation, were welcome, but that prosecutors and advocates were slow to use them. In addition, the Law on Mediation passed in 2014 has not been implemented Conclusion: The Team concludes that certain opportunities to reduce backlog and promote efficiency are not fully utilized. Although judges may not be reassigned to locations with large backlogs, cases from courts with large backlogs may be reassigned to courts without large backlogs. Within courts, court presidents should make staffing decisions that remove duties from judges with backlogs to skilled staff. The Team further concludes that increased usage of plea bargaining, settlement, and mediation as foreseen in the NJRS and Action Plan is needed to increase court efficiency and reduce backlogs. The Bar Association, Prosecutors’ Association and Mediators’ Association should be engaged in promoting the use of these alternatives. Recommendation: Rules for changing territorial jurisdiction can be found in the Criminal and Civil Procedure Codes. Both allow the SCC, or in some criminal cases, a higher court, to reassign a case. The Team recommends that the SCC use existing law to reassign cases from courts with large backlogs to those with smaller backlogs, as was done recently in Pancevo. Within courts, staffing plans that are based upon court efficiency and backlog reduction should be submitted to the Working Group as a requirement of each court’s backlog reduction plan. The MOJ should consider changes to the law that would allow for the creation by the Working Group of temporary backlog reduction units of judges charged with solving old cases in overburdened courts. The use of plea bargaining and settlement and mediation in all courts should be encouraged, especially in overburdened courts, to reduce the number of new cases entering the system. Finally, the Team recommends that donors provide targeted support for implementation of the Mediation Law as foreseen by the NJRS and Action Plan. CHALLENGES The primary challenge in this evaluation was obtaining data that could be used to compare results and impact in partner and non-partner courts. After being informed by several sources, including SPP staff and experts, World Bank experts, representatives of the MoJ and the courts themselves, that such statistics did not exist, the Evaluation Team submitted a formal letter to Supreme Court Justice Snezana Andrejevic, who is a member of the Unified BLR Program Working Group, requesting verified statistics that could be used for purposes of comparing SPP partner courts with non-partner courts. The Evaluation Team understands that the letter was forwarded to a SPP staff member who obtained data that was finally submitted to the Team in Excel format on 14 October, 2014. Although this data is voluminous and would require a significant amount of additional effort for the Team to 20 extract and fully analyze, the Team did compile limited comparative statistics that are included in this report. Subsequently, after the Team conducted this limited analysis and was finalizing this report, on 7 November 2014, the SPP Deputy Chief of Party forwarded data she received from Justice Andrejevic to the Team. Another challenge was travel to partner and non-partner courts for site visits. Overall Conclusion: SPP initiated major progress in reducing backlog and improving efficiency in Serbian courts. Through focused work with ten partner courts, SPP was able to test and refine measures that were simple and could be implemented without additional human or financial resources. By documenting best practices, working with key judicial institutions to amend legislation to include those practices and develop a plan for replicating SPP’s success with partner courts on a national level, SPP has provided the Serbian judiciary with a framework for ongoing backlog reduction and improved efficiency. However, as with many projects, SPP’s national partners have come to rely on SPP, and the Evaluation Team concludes that without some level of ongoing outside support, there could be a setback. 21 REFERENCES Contract Documents - SPP Statement of Work, 08.02.07USAID Request for Proposal - USAID Modification of Contract, December 19, 2013 SPP Documents - SPP Workplans Years 1-5 - SPP Annual Reports Years 1-5 - SPP Monthly Reports 10/13-8/14 - SPP PMBP Reports 9/30/2010-3/14/2014 - SPP Best Practices Guide: Backlog Prevention & Reduction Measures for Courts in Serbia, July 2012 - SPP Guide to Statistical Reporting for Court Presidents: Strategies and Tools for Success, June 2013 - SPP National Backlog Reduction Plan (2013-2018) - Individual Partner Court Statistics (2010-12/31/2013) Backlog Reduction and Prevention - Statistics for civil and criminal cases at 12/31/2011 and 9/31/2012 - SPP Website (through 10/10/14) - SPP Training Materials - Trainer’s Manual: Dynamic Individual Case Management for the Civil Trial Judges of Serbia (2012 and 2013 versions and power point presentations) - Trainer’s Manual: Dynamic Individual Case Management for the Criminal Trial Judges of Serbia (2012 and 2013 versions and power point presentations) - Backlog Reduction Plans for Partner Courts for 2011 and 2012 (Serbian) - Backlog Reduction Plan and Analysis Templates (English) - Monitoring Report Templates for Basic and Higher Courts (English) - SPP Short-Term Consultant’s Trip Reports - Judge Carolyn Engel Temin, July 1-7, 2012 - David Egar, 24 August 27- 9 September, 2012 - Ken Stewart, September 2012 - Pim Albers, November 2012 - SPP Assessment Reports - Judicial Academy Organizational Capacity Assessment, September 2014 - High Court Council Organizational Capacity Assessment, 2014 SPP Evaluation Reports - USAID/Serbia Separation of Powers Program Evaluation Final Report, January 13, 2012 22 - USAID/Serbia Separation of Powers Program Evaluation Final Report, June 2013 Supreme Court of Cassation/MOJ Documents - National Judicial Reform Strategy 2013-1018 - Action Plan for Implementation of National Judicial Reform Strategy 2013-1018 - National Backlog Reduction Plan 2013-2018 - Report of the Working Group for the Implementation of the Uniform Court Backlog Reduction Program of the Supreme Court of Cassation - 2011 and 2012 Annual Reports of the Supreme Court of Cassation on the Work of the Courts in Serbia and accompanying analysis ( Serbian) - Book of Court Rules Other Documents - EU Progress Reports 2013 and 2014 - Multi-Donor Trust Fund for Justice Sector Support, Aide Memoir, 20 January to 1 February 2014. Laws: translations from the originals as found on the web site of the Ministry of Justice located at www.mpravde.gov - Constitution of the Republic of Serbia, Published in the Official Gazette of RS No. 98/06 - Criminal Code, Published in the Official Gazette of RS No. 85/05 - Law on Amendments to the Criminal Procedure Code, Published in the Official Gazette of RS No. 32/2013 - Law on High Judicial Council, Published in the Official Gazette of RS No. 101/2010 - Law on Judges, Published in the Official Gazette of RS No. 101/2010 - Law on Organization of Courts, Published in the Official Gazette of RS as amended No. 101/2010 - Law on the Seats and Territorial Jurisdictions of Courts and Public Prosecutors Offices, Published in the Official Gazette of RS as amended No. 101/2013 23 APPENDICES ANNEX A: Evaluation Statement of Work PURPOSE The goal of the Serbian Rule of Law Efficiency Review (ROLER) award is to review USAID assistance in improving court efficiency through two objectives: 1) To measure the results of the Separation of Powers Program (SPP)’s work in case backlog reduction and prevention in partner courts from 2010 to 2013. 2) To assess the performance framework and functionalities of the main case management software systems used by the Serbian judiciary, identify opportunities to optimize their efficient use, and provide recommendations for stakeholder action. The review, which will be realized through two components, will inform future programing and provide actionable recommendations for USAID, the Ministry of Justice (MOJ), High Court Council (HCC), Delegation of the European Union (EUD), and other relevant stakeholders. The first component will focus on the USAID-funded Separation of Powers Program the intention is to measure the impact of its work to reduce case backlog in selected partner courts (as compared to courts which did not partner with the SPP program). Recognizing sample size limit a full impact assessment, the contractor should measure results with an eye for any information on impact that can be found. The second component will review the case management IT systems in the Serbian court network, providing the MOJ and other actors with information and recommendations to optimize court efficiency. The contractor may propose implementing these two components separately or concurrently in order to achieve the required results. The teams implementing the two components should work together and share information where feasible and relevant, in order to add to the value and quality of the respective deliverables. The Mission anticipates two separate deliverables for this activity: one report for each component. The first deliverable will be a performance evaluation report of SPP. The second deliverable will be an assessment report of case management systems in Serbia and recommendations for future action. 24 COMPONENT 1: PERFORMANCE EVALUATION OF SEPARATION OF POWERS PROGRAM The contractor will conduct a performance evaluation of the Separation of Powers Program (SPP)’s Task 2 implemented by the East-West Management Institute, Inc. (EWMI) under contract 169-C-00-08-00102-00. The purpose is to evaluate the results of SPP’s work from 2010 to 2013 in reducing and preventing backlog of cases in partner courts. This evaluation is meant to serve a dual purpose: (1) to identify the impact the program has had in improving court efficiency; and (2) to provide recommendations that will inform future court administration initiatives in Serbia. An impact evaluation was preferred however sample size may limit the methodology. Contractor should employ techniques that can provide impact information, as possible. The audience of this evaluation will be USAID/Serbia, HCC, MOJ, other donors, local and international judicial reform stakeholders, and USAID’s Europe and Eurasia Bureau and Center for Democracy, Rights and Governance. Background and Relationship to Mission Strategy Recognizing the importance of court efficiency for genuine rule of law in Serbia, in 2008 USAID launched the Separation of Powers Program (SPP) to support improvements in court administration as well as judicial and legislative independence. The six-year program, implemented by the East-West Management Institute, is structured around three tasks: Task 1. Judicial branch financial independence: Developing the capacity of the Serbian judiciary to allocate, acquire, and manage its resources; Task 2. Improved court administration: Assisting the Serbian judiciary in making its administration of justice more efficient, transparent, and responsive to the needs of its users; and Task 3. Legislative branch financial independence: Building the financial capacity of Serbia’s National Assembly. Together, these tasks support the GoS in its effort to build the necessary institutional and democratic practices required for integration with the European Union. Under the Country Development Cooperation Strategy 2013-2017, SPP activities fall under the Development Objective 1: Accountability of Key Democratic Institutions Strengthened and Intermediate Results (IR) 1.1: Parliament and Local Governments are More Responsive to Citizens, and IR 1.2: Rule of Law and Oversight Improved. 25 During the Separation of Powers Program, Serbian judiciary went through two major reconstructions, in 2010 and 2014. The evaluation team is expected to assess the impact of SPP’s interventions in backlog reduction during the period 2010-2013. The courts of general jurisdiction: · Basic courts · High Courts · Appellate courts · Supreme Court of Cassation. The courts of special jurisdiction: · Misdemeanor courts · High Misdemeanor court · Commercial courts · High Commercial court · Administrative court. Program Approach and Results SPP works with ten partner courts8 of general jurisdiction to improve court efficiency and effective case management. The performance objectives for Task 2 included the creation of a position, training and implementation of court managers; improvements in case processing efficiency; and increasing transparency and openness. A substantial piece of Task 2 focused on backlog reduction, the focus of this performance evaluation, including the following two Performance Objectives: · After four years, the average number of cases pending for more than two years has been reduced, and the average number of cases pending for more than four years in select courts has been reduced. (Performance Objective 10) · After five years, the average case processing time in selected courts has been reduced. (Performance Objective 15) With the assistance of SPP, the partner courts identified key internal and external problems, which led to increased backlog and reduced case processing efficiency. SPP provided potential measures and techniques to achieve higher clearance rates, which included: 1) creation of backlog reduction team; 2) new procedures of monitoring and labeling backlog cases; 3) improved delivery/service of documents; 4) creation of preparatory department and new procedures in court registry; 5) cooperation with external partners of the courts; 8 The ten basic and higher partner courts are the Basic Courts in Cacak, Nis, Sremska Mitrovica, Subotica, Uzice, Vranje, and Vrsac, and the Higher Courts in Belgrade, Novi Pazar, and Subotica. 26 6) efficient scheduling of court Hearings; 7) E-justice measures; and 8) new policies regarding court experts. SPP’s approach with partner courts was to focus on one case type, apply selected techniques, analyze the results and then gradually spread successful efforts to other case types. SPP had only an advisory and monitoring role and did not invest any funds in equipment or facilities. By end of 2012, all 10 partner courts were working simultaneously on backlog prevention and reduction on civil and criminal cases. In agreement with USAID, SPP excluded old enforcement cases from the overall number of backlog cases. In 2013, half of the acting court presidents of partner courts were replaced. SPP continued to monitor their work and to cooperate with some of them. SPP’s analysis shows that the courts continued to have above average results and the techniques used are now deeply incorporated in the courts’ procedures. The new National Judicial Reform Strategy (2013-2018) calls for the creation of a national backlog reduction plan, and, in the second half of 2013, SPP developed a National Backlog Reduction and Prevention Plan, calling for nationwide improvements, based on results achieved by partner courts. SPP presented the Plan to the MOJ, HCC and SCC. The Supreme Court of Cassation took the lead in the backlog reduction and adopted the National Backlog Reduction Plan on December 25, 2013. The Supreme Court of Cassation formed a Working Group, which will use SPP’s model to spread it nationwide and to monitor the implementation. In 2014, SPP continues to support the backlog reduction by switching from an individual to the national level. SPP funded the development of a tracking report within the Case Management System (“AVP”) database that the working group will use in monitoring backlog reduction results. Supreme Court of Cassation has been collecting individual court reports, and issuing annual overall reports. According to the SCC’s report for 2013, there are more than 2.8 million pending cases out of which nearly 1.7 million cases are considered “old”. The vast majority of “old” cases are enforcement cases. With the new Law on enforcement adopted in 2011, the private enforcement officers were introduced into the system. The positive effect is yet to be seen. SPP memorialized this work in the Best Practice Guide, released in mid-2012, providing practical guidance to other Serbian courts. The statistics for each court, as of September 2013, are available for the evaluators. Project activities include the following results: • The six courts participating in SPP’s backlog reduction program reduced their combined backlog from over 23,000 cases in 2010 to less than 12,000 27 cases as of September 30, 2012, a 49% decrease in backlog (well in excess of SPP’s targeted 25%-30% reduction). • The average clearance rate for the five courts participating in SPP’s backlog prevention program exceeded 113%, significantly higher than the clearance rate target of 95%. • SPP developed a template for backlog reduction planning that was initially used by its 10 partner courts. The MOJ subsequently recommended that all Serbian courts use the SPP template for 2012 backlog reduction planning. SPP also provided tools for analyzing and monitoring performance against backlog reduction goals. • SPP successfully worked with court managers to reduce the enforcement backlog. At the Belgrade First Basic Court, Infostan, the largest utility, entered into payment agreements with 80,000 debtors with multiple enforcement actions against them, resulting in the removal of 600,000 cases from the system. • Several of SPP’s recommended backlog reduction and prevention techniques were incorporated into the new civil and criminal procedure codes, including the use of preliminary hearings to set case schedules and hearings, and improvements to service of process and delivery of court documents. Evaluation Questions: The evaluation team will measure SPP’s impact in reducing and preventing case backlog in partner courts, reviewing activities implemented from 2010-2013. Evaluators should identify what SPP has achieved, whether expected results have occurred according to the project’s design, and whether those results could have occurred without the program assistance. Evaluators should answer the following questions, listed in order of priority: 1. What were the results and impact of SPP’s interventions for case backlog reduction and prevention in partner courts, compared to courts that were not part of the program, from 2010 to 2013? 2. What were the key factors and lessons learned in generating these results, if any? 3. Building upon key lessons learned, how can Serbian and international stakeholders expand and scale up these results nationally, in order to meet the implementation objectives outlined in Serbia’s National Judicial Reform Strategy? Data Collection Evaluators should use a mix of quantitative and qualitative data collection and analysis methods to generate findings. USAID Serbia will provide evaluators with project related documents and will make available data collected by SPP. This includes the following documents: - SOW for the Separation of Powers Program 28 - SPP Project quarterly and annual reports, and work plans - SPP PMP reports and data - SPP Mid-term Evaluation Report - AVP reports 2010-2013 - National Judicial Reform Strategy (2013-2018) - National Backlog Reduction and Prevention Plan/ SPP Best Practice Guide - Annual Reports from the Supreme Court of Cassation Evaluators should also review the data provided through these reports, verifying the information through site visits to courts and review of data available through the MOJ’s case management system (AVP). Finally, evaluators should also access the High Court Council’s budget software, (BPMIS) to help collect court profile data. The Evaluation Team should develop data collection tools that are consistent with the evaluation questions to ensure high quality analysis. The Evaluation Team is required to share data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. The Evaluation Team will start work on a desk review of all the documents listed above prior to arriving in Serbia. The local evaluation team members should complete the paper review prior to the international team’s arrival. Evaluation Methodology The evaluation team will first complete a desk study that will be used to establish an understanding of SPP’s activities and environment before arrival in Serbia. Based on this understanding, the evaluation team will prepare a work plan that it will present to the mission at the evaluation in-brief. The methodology will include a mix of tools appropriate to the evaluation’s research questions. Prior to the start of data collection, the evaluation team will develop and present, for USAID/Serbia review and approval, a data analysis plan that details how data received from control and test groups will be analyzed; what procedures will be used to analyze qualitative and quantitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data from indicators and project performing monitoring records to reach conclusions about the effectiveness and efficiency of the activities implemented. Evaluators should, where feasible, use experimental or quasi-experimental methods. All findings and conclusions/qualitative analysis will be supported by quantitative data, when available, and/or representative examples. The Evaluation Team Leader 29 will provide a draft of all interview protocols to USAID prior to starting interviews. USAID reserves the right to approve/suggest additional questions. The evaluation will be supported by relevant data and information gathered from meetings with program partners and all the relevant stakeholders. The evaluation team will clearly demonstrate links between the recommendations/conclusions and data available. Deliverables: A team planning meeting will be held in Serbia at the outset of the team’s field work. This meeting will allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will: • Clarify team members' roles and responsibilities; • Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion; • Review and develop final evaluation questions (work out realistic expectations of the team within each of the topic areas; • Review and finalize the assignment timeline and share with USAID; • Present data collection methods, instruments, tools, and guidelines (materials should be developed prior to this meeting); • Review and clarify any logistical and administrative procedures for the assignment; • Develop a preliminary draft outline of the team's report; and, • Assign drafting responsibilities for the final report Work Plan: During the team planning meeting the evaluation team will prepare a detailed work plan, which will include the methodologies to be used in the evaluation, timeline, budget and Gantt chart. The work plan will be submitted to the USAID Evaluation Program Manager for approval no later than the sixth day of work. Methodology Plan: A written methodology and data analysis plan (evaluation design, data analysis steps and detail, operational work plan) will be prepared during the team planning meeting and discussed with USAID prior to implementation. List of Interviewees and Schedule: USAID will provide the Evaluation Team with an initial list of interviewees, from which the Evaluation Team can work to create a more comprehensive list. Prior to starting data collection, the Evaluation Team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. 30 Data collection tools: Prior to starting fieldwork, the Evaluation Team will share the data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion and approval. In-briefing and Mid-term brief with USAID: The Evaluation Team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in-brief, the partner should have the list of interviewees and schedule prepared, along with the Gantt chart that map out the evaluation through the report drafting, feedback and final submission periods. At the mid-term brief, the partner should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. Discussion of Preliminary Draft Evaluation Report: The team will submit a rough draft of the report to the USAID Evaluation Program Manager, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Serbia through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project design approaches, results, or activities. The team will consider USAID/Serbia comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the team leader’s departure from Serbia. The written report should clearly describe findings, conclusions, and recommendations. Where possible, the draft report should include graphic representation of key program accomplishments and impact. These graphs and other visualizations shall be suitable for use/incorporation in Mission presentations and communications materials. USAID will provide comment on the draft report within two weeks of submission. Final Report: The team will submit a final report that incorporates the team responses to Mission comments and suggestions no later than ten days after USAID/Serbia provides written comments on the team’s draft evaluation report (see above). The format will include an executive summary, table of contents, methodology, findings, and actionable recommendations. The report will be submitted in English, electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. 31 Success story: Based on the evaluation findings, the team will prepare a success story on the backlog reduction along with the evaluation report. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this evaluation will be presented to USAID on CD Rom to the Evaluation Program Manager. All data on the CD will be in an unlocked, editable format. Reporting Guidelines • The evaluation report should represent a thoughtful, well-researched and well￾organized effort to objectively evaluate the program. • Evaluation reports shall address all evaluation questions included in the scope of work. • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations should be supported by a specific set of findings. • Recommendations should be action-oriented, organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline need to be agreed upon in writing by USAID. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) and what is being done to mitigate the threats to validity. Evaluation report requirements The format for the evaluation report is as follows: 1. Executive Summary—concisely state the most salient findings and recommendations; 2. Table of Contents; 3. Introduction—purpose, audience, and synopsis of task; 32 4. Background—brief overview of SPP, USAID project strategy and activities implemented in response to the problem, brief description of SPP project, purpose of the evaluation; 5. Methodology— describes evaluation methods, including threats to validity, constraints and gaps; 6. Findings/Conclusions/Recommendations—for each evaluation question; also include data quality and reporting system that should present verification of spot checks, issues, and outcome; 7. Challenges—provide a list of key technical and/or administrative, if any; 8. References (including bibliographical documentation, meetings, interviews and focus group discussions); 9. Annexes—annexes that document the evaluation methods, schedules, interview lists and tables—should be succinct, pertinent and readable. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/policy/evalweb/evaluation_resources.html). The final evaluation report will conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy. The USAID Evaluation Program Manager will determine if the criteria are met. This evaluation will not conclude until the Evaluation Program Manager has confirmed, in writing, that the report has met all of the quality criteria. The final version of the evaluation report will be submitted to USAID/Serbia electronically. The report format should be restricted to Microsoft products and 12- point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 30 pages, excluding references and annexes. COMPONENT 2: ASSESSMENT OF CASE MANAGEMENT SOFTWARE IN SERBIAN COURT NETWORK At present, there are a variety of case management software applications operating in the Serbian justice sector. The National Judicial Reform Strategy calls for a comparative overview of all information technology (IT) systems working in the area of case management and states that the GoS seeks to unify the management systems “in order to achieve a centralized management and operation of the systems, which would enable their intercommunication.”9 The purpose of this assessment is to deliver high-quality, independent analysis to assist the MOJ with that comparative overview by reviewing the currently existing case management software utilized throughout the court network in Serbia, assess their 9 National Judicial Reform Strategy 2013-2018, Page 19. 33 technical capacities to be further developed towards a unified system and to develop recommendations for consideration by the Ministry and other interested parties. Background The Serbian court system currently utilizes two main types of case management software: AVP and SAPS. AVP was first implemented in 2008 for 20 commercial courts with USAID assistance through the Commercial Court Administrative Strengthening Activity (CCASA), implemented by Booz Allen Hamilton. AVP is a product of Mega Serbia, a local software firm. After AVP was piloted in the commercial courts, the MOJ later expanded the rollout of AVP throughout the court system. Today AVP is available and operates in all Basic, Commercial and Higher Courts. A similar system (also developed by Mega Serbia) is being considered for the Misdemeanor Courts. In 2009, the European Union made a grant to the Serbian Government to deploy a case management system called Standardized Software Application (SAPS) for the Serbian Judiciary (the project was implemented by the business technology firm Atos). The goal was to improve the efficiency and transparency of the Judicial System of Serbia by developing and introducing a centralised information system. The project began in March 2010 and by January 2013 SAPS was developed, users trained and deployed in the Supreme Court of Cassation; Appellate Courts in Belgrade, Novi Sad, Nis and Kragujevac; the Administrative Court; and as a pilot in the Basic and Higher Courts in Sremska Mitrovica. In addition, some courts may be using their own customized case management systems, which have been developed internally. Purpose The purpose of this assessment is to inform the implementation of the National Judicial Reform Strategy by providing independent high-quality, comprehensive expert analysis that will help Serbian decision makers achieve key benchmarks as negotiations towards EU accession commence. The National Judicial Reform Strategy, passed in 2013, emphasizes the importance of transparency, accountability and efficiency for effective rule of law. Effective and unified case management system is important to implement that vision for several reasons: first, it improves internal court management and ensures more efficient and cost-effective services to the public; second, it provides the MOJ, High Court Council and others with information necessary for evidence-based decision making, resource allocation, and annual budgeting. Given the existence and usage of different case management systems, the MOJ - in partnership with the courts, EUD, USAID, and other stakeholders – repeatedly sought 34 for a comprehensive expert analysis that can provide options for identifying efficiencies and paths forward. Therefore, the Assessment Team will analyze the current case management systems operating throughout the Serbian court network and develop recommendations of ways forward for consideration by the MOJ. The focus for this independent assessment should be on case management systems operating in the courts – primarily AVP and SAPS. However, the assessment should provide a review of all existing case management systems in the judiciary, including links of the case management with prosecution and prison administration. The recommendations developed as part of this assessment should be practical and cost effective in nature, reflecting the budgetary and bureaucratic realities facing the MOJ as well as the donor community–yet without sacrificing overall system reliability /availability, data integrity and access control management. Therefore recommendations should be concrete, and where possible, intended to improve what already exists rather creating a new “perfect” but cost prohibitive solution. USAID doesn’t seek recommendations that pick one system versus another, per se, but rather considerations for how the existing applications could be integrated or unified. This implies the preparation of a specific and detailed Total Cost of Ownership analysis for the nation- wide running (i.e. development/installation/implementation cost, training and maintenance costs, total licensing costs, and any costs related to additional hardware and/or software upgrades or licenses needed, interconnectivity cost, human resources needed for maintaining the application) of the currently available systems. As with all research, and given the sensitive background of two local IT companies providing two 'competing' systems, independence is paramount for the findings and recommendations to be valuable and trusted. The Assessment Team will adhere to strict standards of independence, ethics, and analytical rigor. USAID intends to conduct this assessment in partnership with the MOJ and the EU Delegation to Serbia throughout this process. All final assessment findings will be provided to USAID, MOJ, and EU Delegation. Assessment Questions: The Assessment Team will answer the following questions: 1. What are the relative strengths/weaknesses of AVP and SAPS? [Analysis should include at a minimum consideration of technical functionality, addressing information security (data integrity, system reliability, availability, access control and auditing etc.), TCO (Total Cost of Ownership) over the period of five 35 (5) years, experience to date, and human resource requirements for maintaining/supporting each system; other criteria may also be proposed.] 2. Provide five (5) recommendations to MOJ and other stakeholders for ways to optimize the utilization of current case management software applications, in order to meet the relevant goals in the National Judicial Reform Strategy. (Include estimates of costs for these recommendations, including hardware, software, connectivity, and human resource requirements.) 3. Based on answers to (1) and (2), provide three (3) recommendations to inform the development of the MOJ’s IT Strategy for the judiciary. Data Collection As highlighted earlier, the value of this assessment is directly linked to the independence and rigor of the team and methodology. By conducting an independent evaluation, the credibility of its findings and recommendations will be elevated for all stakeholders. The Assessment Team will first conduct a desk study, reviewing all materials provided by USAID, the MOJ, EU Delegation or other stakeholders prior to arriving in Serbia. The local team members should complete the paper review prior to the international team’s arrival. The Assessment Team should then develop data collection tools that are consistent with the assessment questions to ensure high quality analysis. The Team is required to share data collection tools with the USAID Program Manager for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. Documents to be provided include: - National Judicial Reform Strategy (2013-2018) - ICT Strategy Report/Concept Note for the Judicial Functional Review (being conducted by MDTF WB) - National Backlog Reduction and Prevention Plan/ SPP Best Practice Guide - CCASA program Final Report - SAPS Project documentation - Annual Reports from the Supreme Court of Cassation The Assessment Team will be given access with test databases to AVP and SAPS application systems and should travel throughout Serbia’s court network for interviews and review of systems. A team planning meeting will be held in Serbia at the onset of the Assessment’s field work. This meeting will allow USAID to present the team with the expectations of the assignment. In addition, the team will: 36 • Clarify team members' roles and responsibilities; • Review final assessment questions to confirm shared understanding and expectations • Review and finalize the assignment timeline and share with USAID; • Present data collection methods, instruments, tools, and guidelines (materials should be developed prior to this meeting); • Review and clarify any logistical and administrative procedures for the assignment; • Develop a preliminary draft outline of the team's report; and, • Assign drafting responsibilities for the final report The data collection methodology will be comprised of a mix of tools appropriate to the research questions. These tools may include a combination of the following: • Conducting a functional analysis to identify courts’ needs and operations • Reviewing each of the case management systems to identify their capabilities • Key informant interviews and focus groups of users of AVP, SAPS, and other existing systems, to understand needs, identify functionality of each system, user-friendliness, and requirements on application supporting/maintenance staff (including requirements for training) Data Analysis Methods Prior to the start of data collection, the assessment team will develop and present, for USAID/Serbia review and approval, a research plan that details how stakeholders will be included in the process (including MOJ, USAID, European Union), how information will be collected and analyzed; and how the assessment will weigh and integrate qualitative data from these sources with quantitative data received from analysis of the software to reach conclusions about costs and benefits of the various software solutions. Analysis should lead the development of five recommendations for consideration by the MOJ on ways forward to improve effective case management in Serbia. Deliverables Work Plan: During the team planning meeting the Assessment Team will prepare a detailed work plan, which will include the methodologies to be used in the evaluation, timeline, budget and Gantt chart. The work plan will be submitted to the USAID Evaluation COR for approval no later than the sixth day of work. 37 Methodology and Data Collection Plan: A written methodology and data analysis plan (design, data analysis steps and detail, operational work plan) will be prepared during the team planning meeting and discussed with USAID prior to implementation. List of Interviewees and Schedule: USAID will provide the Assessment Team with an initial list of interviewees, from which the Assessment Team can work to create a more comprehensive list. Prior to starting data collection, the Assessment Team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Assessment Team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. In-briefing and Mid-term brief with USAID: The Assessment Team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in￾brief, the partner should have the list of interviewees and schedule prepared, along with the Gantt chart that maps out the assessment through the report drafting, feedback and final submission periods. At the mid-term brief, the partner should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. Discussion of Preliminary Draft Assessment Report: The team will submit a rough draft of the report to the USAID Program Manager, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the Assessment Team’s departure. Debriefing with USAID: The Assessment Team will present the major findings to USAID/Serbia through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project design approaches, results, or activities. The team will consider USAID/Serbia comments and revise the draft report accordingly, as appropriate. Draft Assessment Report: A draft report of the findings and recommendations should be submitted to the USAID Program Manager prior to the team leader’s departure from Serbia. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide comment on the draft report within two weeks of submission. Final Report: The team will submit a final report that incorporates the team responses to Mission comments and suggestions no later than ten days after USAID/Serbia provides written comments on the team’s draft assessment report (see above). The format will include an executive summary, table of contents, methodology, findings, and at least two actionable options for consideration by the 38 MOJ. Where possible, the report should use easy to read tables or graphic representation of key findings. The report will be submitted in English, electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this assessment will be presented to USAID on CD Rom to the Program Manager. All data on the CD will be in an unlocked, editable format. All data sets will be shared with the MOJ and the EUD. 39 ANNEX B: Persons and Agencies Contacted List of Interviewees NAME AND TITLE Brian LeDuc, SPP Chief of Party Sonja Prostan, SPP Deputy Chief of Party Milan Nikolic; Dimitije Sujeranovic , SPP Staff members Snezana Andrejevic, Supreme Court Justice, member of the Unified BLR Program Dragana Boljevic, Judge, Belgrade Appellate Court Zorica Bulajic, Judge, Belgrade Appellate Court Biljana Djukic, Deputy Head of Registry Office, Belgrade Appellate Court Andjelka Stanojevic, President, Novi Sad Basic Court Marija Rakic, Judge Subotica Higher Court Rozalija Tumbas, Court President, Subotica Basic Court Tamara Trajkovic, Judge, Subotica Basic Court Goran Nikolic, President, Vrsac Basic Court Cveta Kajevic Grubisic, Judge, Vrsac Basic Court Branka Jankovic, Court President, Uzice Basic Court Goran Spasic, Court President, Nis Basic Court Sasa Dincic, Court Secretary, Nis Basic Court Sladjana Djurickovic, Deputy President, Nis Appellate Court Vidosava Danilovic, Judge, Uzice Basic Court Ljiljana Karac, President, Valjevo Basic Court Radoje Milovanovic, Court Secretary, Valjevo Basic Court Mirjana Puzovic, Chief of Cabinet of the President of the Supreme Court of Cassation, Secretary of the UBRP Working Group Majda Krsikapa, Secretary General, High Court Council Tanja Sobat, Court President, Belgrade I Basic Court, Member of the UBRP Working Group Cedomir Backovic, Assistant Minister, Ministry of Justice Aleksandar Nikolic, Court Secretary, Uzice Basic Court Dragan Skontra, IT Administrator, Vrsac Basic Court Ljubinko Petrovic, President, Pozega Basic Court Stanka Simonovic, President, Pancevo Basic Court Natasa Lazovic, President, Cacak Basic Court Izet Suljovic, President, Novi Pazar Higher Court Aleksandar Stepanovic, President, Belgrade Higher Court Dijana Jankovic, Judge, Nis Appellate Court 40 ANNEX C: Semi-Structured Interview Questions (Questions are illustrative and not comprehensive) Qs for SPP • What were the criteria for selecting the backlog reduction (BR) and backlog prevention (BP) pilot partner courts? • What was the difference in SPP’s initial assistance to the BR vs. BP pilot courts? • How did you assist the pilot courts in expanding into BR or BP, respectively? • What were the key factors affecting results in the partner courts? Lessons learned? • In your opinion, what is the quality of the data received from the MoJ? • What was SPP’s role in expanding BR/BP to non-partner courts? • What further support is needed? Will IPA provide that? Qs for MoJ and HCC • What support has SPP provided to the MoJ in connection with BR and RP efforts? • What is the MoJ’s responsibility in terms of backlog reduction? What other institutions are involved, such as the Supreme Court of Cassation and the HCC? • How does the MoJ monitor progress on BR and BP? How do you ensure integrity of the data collected? • In your opinion, what were the major successes/disappointments in the SPP BR and BP program? • Thinking about the National Backlog Reduction Plan, what assistance has SPP provided in scaling-up the effort? • In your opinion, are the targets in the BR plan reasonable? What obstacles, if any, do you foresee in realization of the targets? • What other donors or projects are supporting BR and BP? What assistance is still needed? Qs for SPP Partners • What were SPP’s interventions in your court? • What was the composition of your backlog reduction working group? Were there any outside members, such as representatives of the bar or prosecution office? • Which SPP interventions had impact/caused you to make changes? Why? • What specific changes did you make? • With benefit of hindsight, what were the lessons learned? • What process does your court use for: collecting data, entering data, verifying data and producing reports? Can you verify this data from the AVP reports and Supreme Court reports for us? In your opinion, what changes are needed? • What changes did your court make based on successes/failures of other partner courts. • Has your court worked with other international or Serbian organization on BR/RP? • In your view, what assistance do non-partner courts need to implement BR/BP? Who should provide that? Q’s for Non-Partner Courts • What is your court doing in terms of BR and RP (e.g. backlog reduction team: plan; what specific steps are you implementing)? When did the court start those activities and why? What has been effective and what has not? 41 • Are you familiar with the SPP Best Practices Guide? Have you used it to develop solutions for your court? • Have you worked with any internationals/Serbians on BLR & P? Describe. • What assistance does your court need to implement BR/BP? Who should provide that? • In your opinion, are the targets in the BR plan reasonable? What obstacles, if any, do you foresee in realization of the targets? • What process does your court use for: collecting data, entering data, verifying data and producing reports? In your opinion, what changes are needed? Q’s for External Interlocutors (Bar, attorneys, prosecutors, etc) • Can you describe how implementation of backlog reduction measures has affected your practice at pilot courts? • Were you asked to participate in the planning process? If so, can you please describe your role. In your opinion, what were the greatest difficulties in implementing the measures, both internal and external? 42 ANNEX D: Individual Court Statistics for USAID, March 2013 43 ANNEX E: Court Efficiency Table10 Court Efficiency Rate, 2010 Efficiency Rate, 2013 High Court Belgrade 69.25% 120% High Court Subotica 89.17% 97.11% High Court Novi Pazar 100.85% 106.23% Basic Court in Subotica 96.90% 112.14% Basic Court in Vrsac 94.85% 112.67% Basic Court in Sremska 78.18% 125.10% Basic Court in Cacak 81.74% 107.84% Basic Court Uzice 89.08% 102.06% Basic Court Nis 89.12% 97.79% Basic Court in Vranje 73.40% 114.34% 10 Efficiency, also known as the clearance rate, is the number of cases received divided by cases solved. Therefore, an efficiency rate of over 100% indicates that the court is solving more cases than received. 44 ANNEX F: Comparative Statistics (Based on the most common types of cases in the basic and higher courts) The comparison below is based on the following data furnished by SPP on October 1411. (Refer to Annex K, "Comparative Statistics Spreadsheet" -- Excel spreadsheet attached with report) • 2013--Excel sheets for all basic and higher courts individually and as a group. • 2011- Excel sheets for all basic and higher courts individually. The team prepared the compilation of data for all basic and higher courts for 2011, then extracted data on the most common types of cases for SPP courts for 2011 and 2013. As no data was provided for 2012, the following analysis compares statistics for 2011 and 2013. The most common types of cases in all courts of the same level are: • Basic courts: civil, investigation, criminal, inheritance, labor. Statistics available at Annex F. • Higher courts: criminal, criminal investigation, criminal juvenile, criminal investigation juvenile, and second instance civil, labor and family. Statistics available at Annex F.3. 2013 Basic Courts Court Backlog 1/1/2013 Backlog 12/31/ 2013 % Change in Backlog Increase/ (Decrease) Total Cases Received Total Cases Solved Clearance Rate (%) Uzice 488 469 -3.8 5,949 6,137 103.1 Cacak 1,380 1,256 -9 8,247 9,084 101.5 Vranje 3,642 2,173 -40.3 11,582 13,723 118.5 Nis 1,737 1,484 -14.6 27,724 26,660 96.2 Vrsac 188 129 -31.4 3,603 4,249 117.9 Sremska Mitrovica 4,740 2,264 -52.2 11,461 15,702 137 Subotica 1,155 621 -34.3 10,670 12,493 117.1 Total SPP Partner Basic Courts 13,330 8,396 -37% 79,236 88,048 111.12% Total Serbian Basic Courts 62,559 45,636 -27% 398,968 434,367 108.87% As a whole, in 2013 the SPP partner basic courts outperformed the Serbian basic courts in both backlog reduction and prevention. SPP partner basic courts reduced backlog by 37% vs. 27% for the Serbian basic courts and SPP partner courts achieved a 111.12% clearance rate compared to 108.87% at the Serbian basic courts. Although backlog 11 The out-brief was on Oct. 10 and the evaluation team departed Serbia on Oct. 11. 45 decreased in Nis, the clearance was less than 100%, which can lead to future increases in backlog as cases age. The Team learned that the Nis basic court has aggressively pursued backlog reduction, but has had less success introducing backlog prevention techniques such as cooperation with outside agencies. 2011-Basic Courts Court Backlog 1/1/11 Backlog % Change in Backlog Increase/(Decrease) Total Cases Received Total Cases Solved Clearance Rate (%) Uzice 322 457 41.9 7532 6728 91.5 Cacak 1217 1218 -- 11464 11299 98.6 Vranje 3307 3123 -5.6 13438 12124 90.2 Nis 1322 986 -10.2 27244 28048 103 Vrsac 265 204 -23 4392 3812 86.8 Sremska Mitrovica 2947 4409 49.61 11840 11249 95 Subotica 2008 1942 -3.2 11206 11919 106.4 Total SPP Partner 11,388 12,339 8.3 87,116 85,179 97.78% Basic Courts Total Serbian 62,296 63,090 1.3 422,937 404,386 95.61% Basic Courts In 2011, the backlog increased in both SPP partner basic courts (8.3%) and Serbian basic courts as a whole (1.3%). The increase was higher at the partner courts due to a nearly 50% at the Sremska Mitrovica basic court. Further, although neither SPP partner basic courts or Serbian basic courts as a whole resolved more cases than received, the clearance rate at the SPP partner basic courts was higher (97.78% vs 95.61%.). In addition, while Vrsac had the best results for backlog reduction, it had the worst clearance rate of the partner courts. 46 2011 vs. 2013 Basic Court Backlog 1/1/11 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Clearance Rate (%) Clearance Rate (%) 2011 2013 Uzice 322 469 45.7 91.5 103.1 Cacak 1217 1,256 3.2 98.6 90.8 Vranje 3307 2,173 -34.3 90.2 118.5 Nis 1322 1,484 1.2 103 96.2 Vrsac 265 129 -51.3 86.8 117.9 Sremska Mitrovica 2947 2,264 -23.2 95 137 Subotica 2008 621 -69.1 106.4 117.1 Total SPP Partner Basic Courts 11,388 8,396 -26.3 97.78% 111.12% Total Serbian Basic Courts 62,296 45,636 -26.7 95.61% 108.87% Comparing 2011 and 2013, both SPP partner and non-partner basic courts as a whole decreased backlog by approximately 26% and improved clearance rates – resolving more cases than received. The clearance rate for partner basic courts as a whole was slightly higher than for non-partner courts (111.12% vs. 108.87%). However, there were significant differences in the results among partner courts. . 47 2013 Higher Courts Court Backlog 1/1/13 Backlog 12/31/13 % Change in Backlog Increase/(Decrease) Total Cases Received Total Cases Solved Clearance Rate Belgrade 2,519 673 -73.3 17,701 24,597 138.20% Novi Pazar 52 11 -78.8 1,033 1,145 110.80% Subotica 19 8 -58 1,586 1,448 91.30% Total SPP Partner Higher Courts 2,590 692 -73.30% 20,320 27,190 133.81% All Serbian Higher Courts 3,476 1,206 -65.30% 67,822 76,199 112.35% As a whole, in 2013 the SPP partner higher courts outperformed the Serbian higher courts in both backlog reduction and prevention. SPP partner higher courts reduced backlog by 73.3% vs. 65.3% for the Serbian higher courts and SPP partner courts achieved a 133.81% clearance rate compared to 112.35% at the Serbian higher courts. The Team learned that the dramatic backlog reduction at the Belgrade Higher Court was due to the transfer of abuse of power cases to the basic court as a result of amendments to the criminal procedure code. These cases were transferred back to the Belgrade Higher Court in the 2014 court network reorganization increasing the backlog again. The Team learned the criminal department backlog increased after the 2010 court network reorganization and that the court has not been able to eliminate the backlog up due to judicial vacancies. 48 ANNEX G: Evaluation Matrix Documents/Research Semi-Structured Interviews Donor Implementer Secondary USG Personnel Implementer Staff Direct Participants or Beneficiaries Indirect Beneficiaries External Mini-Survey Backlog Reduction: 2.2.1 A. Preparation and implementation/monitoring of backlog reduction plans by individual partner courts  SPP SC; MoJ   PC Presidents, Dept Heads, Members of BR working groups & Court Administrators SC, MoJ, HCC Attorneys (local bar assn.); Prosecutors BR working group members; Attorneys/Pros ecutors 2.2.1 B Overall backlog reduction strategy/Disseminate strategies to reduce backlogs and improve case processing times in courts throughout Serbia  SPP SC; MoJ   Non-PC Presidents, Dept Heads, Members of BR working groups & Court Administrators SC,MoJ, HCC Attorneys, (local bar assn.); Prosecutors; legal aid CSOs, International organizations BR working group members; Attorneys/Pros ecutors Backlog Prevention (case management): 2.2.2 2.2.2.A. Case flow management assistance to individual courts  SPP,   PC Presidents & SC, MoJ, JCC 49 Court Administrators , HCC 2.2.2.B. Dissemination of results  SPP   PC and non-PC Court Presidents SC, MoJ, HCC International orgs 50 ANNEX H: Findings, Conclusions, Recommendations (FCR) FCR Table Findings Conclusions Recommendations EQ1: What were the results and impact of SPP’s interventions for case backlog reduction and prevention in partner courts, compared to courts that were not part of the program, from 2010 to 2013? Note: At the in-brief, the Mission added a related question—what is the reason of variation in results between in partner courts? • All partner courts utilized a variety of the techniques for backlog reduction and prevention proposed by SPP. Since the 2013 requirement for preparation of backlog reduction plans by all courts, non-partner courts have implemented the basic measures required for backlog reduction (identifying, labeling and monitoring old cases) but adoption of efficiency/backlog prevention measures less common. • Trend was positive in partner courts from 2010-2013 (decrease in aggregate backlog and improvement in average clearance rate), but results in individual courts varied due to • Impact of SPP assistance evident in the level and sophistication of backlog reduction and prevention initiatives in partner vs. non-partner courts. • Degree of improvement varied from court to court and year to year not only as a result of the extent to which individual courts implemented the measures proposed by SPP, but also to circumstances outside the court’s 51 variety of factors: 1) predominant case type and large inflow of cases; 2) size, location, and case load; 3) cooperation of external partners; 4) attitude of court presidents and judges to reform. • Limited statistics based on most common types of cases in basic and higher courts indicate progress in backlog reduction and prevention at partner and non-partner courts. • Trends that can affect backlog reduction/prevention statistics in all courts: (i) declining number of cases; (ii) increasing number of judges. • MoJ institutionalized SPP backlog reduction and prevention methodology and required all Serbian courts to prepare backlog reduction plans for 2013. However, some courts had initiated backlog reduction activities earlier. • Statistics of basic and higher courts control. • In many cases, statistics are affected by factors that courts cannot control so statistics do not accurately reflect SPP interventions. • Comparative statistics do not indicate that SPP interventions did not have impact. Not possible to make reliable statistical comparisons based on data quality. Accurate statistical data necessary for implementation and monitoring of NBRP. • Beginning in 2013, statistics for all courts reflect results of initial national roll-out of backlog reduction and prevention efforts. • As discussed under EQ 3, priority must be given to developing accurate statistics and improved reporting. 52 widely considered to be unreliable. EQ2: What were the key factors and lessons learned in generating these results, if any? • UBRP Working Group very reliant on SPP • Five themes evident in SPP approach to partner courts: 1) select partner courts that had problems and would provide good examples of how interventions could work; 2) introduce practical techniques for backlog reduction and prevention and provide tools for successful implementation; 3) allow partner courts to select and own techniques; 4) work closely with partner courts and provide hands-on assistance; and 5) closely monitor progress and follow up on any apparent obstacles to progress. • SPP developed close working relationship with MOJ and SCC, which was a key factor in extending the SPP model of backlog reduction and prevention to all Serbian courts. SPP documented and leveraged its work • SPP methodology based on five factors was key to its success in introducing new backlog reduction and prevention techniques in partner courts. Foregoing “one size fits all” approach allowed partner courts to select measures best suited to problems in their courts and resulted in partner courts investing in the success of the measures. • SPP effectively used a top-down and bottom-up approach to test and refine backlog reduction and efficiency interventions, disseminate successful measures to all courts, as well as achieve a measure of • As discussed under EQ 3, Working Group should monitor and guide courts that are not familiar with or properly implementing backlog reduction and prevention techniques in the near term. Continued support to Working Group needed to sustain SPP activities. 53 with the partner courts through the Best Practices Guide, assisted the MoJ and SCC in drafting new Court Rules and procedural codes to reflect those practices, and finally the National Backlog Reduction Plan. • Court-to-court cooperation was one of the most common methods for court’s to share experience and lessons learned. SPP employed a variety of techniques to share experiences between partner courts and non-partner courts ranging from the formal (Presentation of Best Practice Guide) to less formal (roundtables, meetings of courts in the same region). • According to the project design, SPP initially followed dual tracks: focused backlog reduction with one set of partner courts and case management/backlog prevention activities that involved the full scope sustainability through rules mandating that all courts devise backlog reduction and efficiency plans, including targets, and report upon them. • Court-to-court cooperation is a key to successful sustainability of court efficiency and backlog reduction measures tested in partner courts through the SPP program. A regular exchange of information between partner courts that are proven champions of efficiency and backlog reduction and courts that are struggling to reduce backlogs is an effective and inexpensive method to disseminate best practices. • SPP correctly identified that backlogs are a symptom of court inefficiency and instructed partner courts to work on backlog reduction and prevention. It may have been better to focus on backlog reduction at all courts at the beginning and then extend activities • As discussed under EQ 3, Working Group and sub-groups should support court-to-court cooperation. 54 of court operations with a second set. In the fourth year of the project, all courts began to follow both tracks. • Serbian courts struggling to keep pace with the ongoing reforms in the Serbian judicial system. Lack of harmonization of Serbian jurisprudence is a large problem that can lead to backlog. SPP only started to support harmonization efforts in the last year. At request of JA, SPP revised training modules on case management to reflect changes in procedural law and conducted ToT, but training not continuing. to include backlog prevention. • Continued focus on harmonization at the appellate level and SCC is necessary. Need to resume training on procedural codes, which will require support to JA. External support may be necessary. • As discussed under EQ 3, support for court practice harmonization and training should be provided. EQ3: Building upon key lessons learned, how can Serbian and international stakeholders expand and scale up these results nationally, in order to meet the implementation objectives outlined in Serbia’s National Judicial Reform Strategy? • Data from courts considered unreliable and lack of reliable statistics to measure progress is obstacle to progress on EU accession. • UNBP Working Group is delaying further work pending correction of • Correction of data in AVP system must be a priority. Development of unified national system of court data needed as discussed in CMS assessment report. • Court presidents and statisticians • MOJ needs to correct data immediately. • UBRP Working Group needs to develop plan for training Working Group members, court presidents and statisticians based on the 55 data for 2014 court network reorganization. • Quality of reports submitted by courts is poor. • Considerable concern about capacity of Backlog Reduction Working Group to maintain the progress to date in implementing the UBRP, including from Working Group members. All activity to date has been facilitated by SPP and the Working Group has not met since the SPP organized retreat in April 2014. • Common reason for non-partner court presidents to focus on efficiency and backlog reduction is that it is required. • EU IPA project that was intended to follow-up on SPP initiatives delayed for at least six more months. • Enforcement cases make up over 90% of backlog in basic courts. need training on preparation of statistical reports, data analysis, etc. • Working Group will suffer without SPP support, particularly due to the delay in the IPA project. If Working Group fails to closely monitor and provide individual guidance to courts in the near term, it is possible that further progress will not be made. • While enforcement cases do not require much work on the part of Statistics Manual developed by SPP in 2013. • Pending initiation of the new EU IPA project, USAID should consider engaging local rule of law-oriented organization to support continuation of the initiatives started by SPP during its final year. Could be accomplished through a grant under the USAID Serbia Civil Society Partners program. • Working Group should continue to focus on reducing number of backlogged enforcement cases in 56 Minimal enforcement backlog in higher courts. Reduction of backlog is important to progress in EU Accession. • SCC issued order for courts to resolve cases more than 10 years old in 6 months. Under criminal and civil procedure codes, SCC can reassign cases to different judges. This technique used to transfer cases from Belgrade and Nis Higher Courts to Subotica and Pancevo. judges, closing them will allow a clearer picture to emerge of the true extent of the backlog problem and have a positive impact in terms of how outside stakeholders assess the problem of case backlogs. • SCC could use ability to reassign cases to transfer backlog cases. Could also form special teams to handle these cases. basic courts. There are several possible interventions that the Working Group should examine, such as a national plan modeled on successful collaboration with Infostan at Belgrade First Basic Court or reassignment of cases by President of SCC. 57 ANNEX I: Final Success Story Separation of Powers Program: Applying Local Lessons Learned Throughout Serbia In Order to Increase Court Efficiency and Reduce Case Backlogs As in most Serbian courts, the Basic Court in Uzice suffered from a large number of backlogged cases in 2010, when its current court president was appointed. The new court president had some ideas regarding how to address the backlog problem, but she was unsure how to apply them. In February 2010, she saw a call for applications from USAID’s Separation of Powers Program (SPP), asking for partner courts that were willing to work on backlog reduction. She applied, and SPP accepted Uzice’s application in May 2010. Thus began a partnership between SPP and the court that would result in the caseload per judge in the criminal department has decreasing from 400 cases per judge in 2010 to 46 cases per judge today, and now the Uzice Basic Court does not have any criminal cases over five years old. The court also increased its efficiency rate from an 89% case completion rate in 2010 to a 102% completion rate by the end of 2013. Because of the SPP program, Uzice now completes more cases than it receives in a year, which will not only gradually lead to a reduction in current backlogs, but will prevent backlogs in the future. The first hurdle the partnership had to overcome was the reluctance of judges in the Uzice Basic Court to accept any reform measures. Judges in the court were cautious and risk averse, partly as a result of the recent non-election of several hundred judges nationwide. To overcome this reluctance, the court president appointed a reform-minded deputy and formed a backlog reduction team in her court, made up of judges and court staff who were motivated to increase efficiency in the court. SPP and the team then devised a backlog reduction plan for the court, emphasizing measures that were practical, based upon the law and upon local circumstances. The team decided to focus on backlogged criminal cases, since the court president was a criminal judge and had experience in those cases. Recognizing that case backlogs result from court inefficiency, and that increasing court efficiency depended not just upon the court but also upon other institutions, the president started a dialogue with external partners who have a stake in court efficiency. With prosecutors, she emphasized the negative budgetary impact of their practice of appealing almost every case, and she persuaded them that if they spent less time on frivolous appeals, they could spend more time on serious cases. As a result, prosecutors began only appealing cases where the verdict in the basic court was an acquittal. The court president next contacted the 58 police department and secured their agreement to streamline the process required for police to bring to court accused who have failed to appear. The president selected a police liaison from among her judicial assistants, improving the process of on-going communication with the police department. Next, SPP and the president began working with the post office to address problems in the way court documents are delivered. As a result, the post office no longer requires numerous written documents from the court to act in most routine matters, speeding up service of process in civil and criminal cases. With SPP’s encouragement, the court president changed the working hours of the court to ensure the court is open during times that are convenient for citizens who use the court. to assist judges who are struggling with particularly difficult cases, the Uzice Basic Court formed teams of experienced civil and criminal judges, who make themselves available to peers who need help with legal issues in individual cases. The court president credits SPP for providing the ideas and encouragement that made the backlog reduction and prevention successes possible in her criminal department. She also acknowledged SPP’s assistance in refining the process by which the court monitors and manages their cases, allowing the court to mitigate the problem of unreliable data in the AVP system. This allows her to identify emerging problems and determine whether certain reforms are working. The president believes the reforms implemented through SPP’s program in her court are permanent. This experience in Uzice is representative of SPP’s approach in nine other partner courts in Serbia. In all the courts, SPP helped court presidents to form backlog reduction teams, to devise plans for increased efficiency and reduction in backlogged cases, and to implement them. The solutions in each court relied upon common sense, local, and cost-effective solutions to backlog reduction and inefficiency. This bottom-up approach also allowed SPP to greenhouse solutions that were applicable in all courts in Serbia. The following table illustrates the successes that have been achieved by SPP. Efficiency rates measure the number of disposed cases as a percentage of the number of new cases. Efficiency rates under 100% indicate that a court is in danger of creating a backlog or that existing backlogs will grow. All ten partner courts increased their efficiency rates between 2010 and 2013. Court Efficiency Rate 2010 Efficiency Rate 2013 High Court Belgrade 69.25% 120% High Court Subotica 89.17% 97.11% High Court Novi Pazar 100.85% 106.23% Basic Court in Subotica 96.90% 112.14% Basic Court in Vrsac 94.85% 112.67% 59 Basic Court in Sremska 78.18% 125.10% Basic Court in Cacak 81.74% 107.84% Basic Court Uzice 89.08% 102.06% Basic Court Nis 89.12% 97.79% Basic Court in Vranje 73.40% 114.34% to implement these successes nation-wide, SPP approached the President of the Supreme Court of Cassation (SCC) about devising a national plan to implement them that was based upon the National Judicial Reform Strategy 2013 – 2018. Using its experience in local courts, SPP assisted with a Draft Backlog Reduction Plan and submitted it to the SCC for consideration. This Draft formed the basis of the National Backlog Reduction Plan, issued by the President of the SCC in December 2013. The National Plan incorporated ideas for backlog reduction that were successfully used in SPP’s ten partner courts. They include the formation of backlog reduction teams in each court and the requirement that each court submit individual backlog reduction plans based upon local realities. implementation of the plan, the SCC established a Working Group made up of judges from the SCC and from other courts, including the court president in Uzice. Pursuant to the requirements of the National Backlog Reduction Plan, court presidents throughout Serbia have formed backlog reduction teams in their courts, prepared backlog reduction plans and are monitoring implementation of the plans and preparing reports on a monthly basis. Court-to-court roundtables have been held for Working Group members and court presidents in partner courts to assist court presidents in non-partner courts with the process of implementing their plans. This nationwide implementation of practical, effective solutions to the problem of case backlogs and inefficiencies in Serbian courts is a direct result of SPP’s work in partner courts such as Uzice. If courts in Serbia adhere to the National Backlog Reduction Plan inspired by SPP’s work, court efficiency will rise, backlogs will fall, and the citizens of Serbia will receive better service from their judiciary. 60 ANNEX J: Comparative Statistics Spreadsheet Refer to attached Excel file. 61 Social Impact, Inc. 2300 Clarendon Boulevard Suite 300 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com