source sensitive document USAID/KOSOVO MIDTERM PERFORMANCE EVALUATION OF THE JUSTICE SYSTEM STRENGTHENING PROGRAM (JSSP) FINAL REPORT This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), a Tetra Tech Company. source sensitive document USAID/KOSOVO MIDTERM PERFORMANCE EVALUATION OF THE JUSTICE SYSTEM STRENGTHENING PROGRAM (JSSP) FINAL REPORT July 2, 2018 IDIQ No. AID-167-I-17-00002 Order No: 72016718R00005 Prepared for United States Agency for International Development/Kosovo Ismail Qemali (Arberia), House 1 Pristina – Kosovo 1000 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV TABLE OF CONTENTS ACKNOWLEDGEMENTS............................................................................................................................ II ACRONYMS .......................................................................................................................................................III EXECUTIVE SUMMARY............................................................................................................................... 1 EVALUATION PURPOSE ..................................................................................................................... 1 PROJECT BACKGROUND.................................................................................................................. 1 EVALUATION QUESTIONS................................................................................................................ 1 EVALUATION METHODS AND LIMITATIONS........................................................................... 2 OVERALL ASSESSMENT....................................................................................................................... 2 CONCLUSIONS AND RECOMMENDATIONS............................................................................ 2 EVALUATION PURPOSE AND EVALUATION QUESTIONS ............................................... 8 EVALUATION METHODS................................................................................................................... 8 EVALUATION LIMITATIONS............................................................................................................. 9 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ...................................................10 EVALUATION QUESTION 1............................................................................................................10 EVALUATION QUESTION 2............................................................................................................24 EVALUATION QUESTION 3............................................................................................................31 EVALUATION QUESTION 4............................................................................................................35 EVALUATION QUESTION 5............................................................................................................38 source sensitive document ii | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ACKNOWLEDGEMENTS The evaluation team of Kristie Evenson, Marilyn Zelin, Dren Doli, and Fisnik Korenica would like to thank all those who assisted the team in carrying out their fieldwork and this evaluation. First, our thanks go to Melita Cacaj at USAID/Kosovo for her guidance during the evaluation process. We also would like to express our appreciation to the JSSP leadership and staff for their assistance in facilitating contacts with JSSP beneficiaries and making themselves available multiple times to answer questions and queries. We want to highlight the translation and logistics work that Murlan Jasiqi contributed as part of our team. We also want to thank all of our MSI colleagues for their technical and administrative support. Finally, we want to thank the informants who took time from their busy schedules to meet with the team. source sensitive document iii | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ACRONYMS ACDC Advocacy Center for Democratic Culture BIRN Balkan Investigative Reporting Network BRLO Backlog Reduction Officer CCK Constitutional Court of Kosovo CDCS Country Development Cooperation Strategy CDMS Classified Document Management System CLE Contract Law Enforcement Program (USAID) COE Council of Europe COP Chief of Party CMIS Case Management and Information System CPRU Court Performance Review Unit CSO Civil Society Organization EQ Evaluation Question EROL Effective Rule of Law Program (USAID) EU European Union EULEX European Rule-of-Law Mission in Kosovo FTA Free Trade Agreement FY Fiscal Year GIZ Gesellschaft für Internationale Zusammenarbeit ICT Information and Communication Technology IMG International Management Group JSA Justice Sector Agreement (Brussels Agreement) JSP Justice Support Program JSSP Justice Sector Strengthening Program KBA Kosovo Bar Association KI Key Informant KJA Kosovo Judicial Academy KJC Kosovo Judicial Council KLI Kosovo Law Institute KPC Kosovo Prosecutorial Council LAU Legal Advisory Unit MCLE Mandatory Continuing Legal Education MEL Monitoring, Evaluation and Learning source sensitive document iv | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV MOJ Ministry of Justice MSC Most Significant Change ODC Office of Disciplinary Counsel OPDAT Overseas Prosecutorial Department Assistance and Training (U.S. Department of Justice) OSCE Organization for Security and Cooperation in Europe PEC Performance Evaluation Committee PIO Public Information Officer PMP Performance Management Plan PRP Property Rights Program (USAID) PSA Public Service Announcement RFP Request for Proposal SAA Stabilization and Association Agreement SOW Statement of Work TOC Theory of Change TOT Training of Trainers UNDP United Nations Development Programme UNMIK United Nations Mission in Kosovo source sensitive document 1 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV EXECUTIVE SUMMARY EVALUATION PURPOSE During April 2018, Management Systems International (MSI) conducted a midterm performance evaluation of the Justice Sector Strengthening Program (JSSP), an activity funded by the United States Agency for International Development Mission in Kosovo (USAID/Kosovo). The purpose of this evaluation is to provide the Mission with an assessment of progress to date toward achieving the JSSP program objectives, and to identify potential areas of modification and emphasis for the remainder of the program. The findings also aim to inform future justice sector programming. PROJECT BACKGROUND JSSP is a four-year rule-of-law activity funded by USAID and implemented by Millennium DPI Partners during the period November 23, 2015, to November 22, 2019. Building upon USAID’s prior efforts to advance the rule of law in Kosovo, JSSP’s overall goal is to improve the quality of the justice system in Kosovo through three program components that: 1) strengthen the efficiency, effectiveness and service delivery of the Kosovo Judicial Council (KJC) and courts; 2) enhance the accountability and professionalism of justice system institutions and actors; and 3) support the functioning and integration of the judicial structures in the north of Kosovo. EVALUATION QUESTIONS The evaluation questions (EQs) are: EQ 1: To what extent have the independence, professionalism, efficiency and accountability of the Kosovo Judicial Council and Courts been advanced by JSSP’s assistance? ● Sub-question: To what extent has JSSP been successful in increasing the self-governance of the judiciary to better serve the public? EQ 2: To what extent have JSSP activities been beneficial to the implementation of the EU￾facilitated 2015 Justice Sector Agreement? ● Sub-question: Are there any challenges and/or opportunities that the activity can explore and address? EQ 3: How successful is JSSP in managing its civil society subcontractors to achieve the activity objectives? ● Sub-question: Were there any other alternative approaches that could have been taken to maximize subcontractor performance? EQ 4: How effective and efficient is JSSP in coordinating with USAID and other donors’ programs? EQ 5: How effective has JSSP been in conducting activities to promote equal opportunities for women in the justice sector? ● Sub-question: To what extent has JSSP been able to promote women as change agents in the judiciary? source sensitive document 2 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV EVALUATION METHODS AND LIMITATIONS The evaluation team applied mixed methods, including a Most Significant Change (MSC) approach, for answering the performance questions (EQ1, 2, and 5). Contribution analysis was added to the approach for answering EQ 5 on gender, while a process-based approach was used for EQs 3 and 4 on civil society organization (CSO) management and donor cooperation. The evaluation team reviewed secondary data from JSSP and conducted a total of 59 key informant (KI) interviews with 91 project beneficiaries, implementers, donors, representatives of CSOs and other experts across the sector. This included three group interviews with sets of interns and legal officers to gauge MSC effects in Mitrovica, the Pristina Basic Court and the KJC, respectively. The team was aware of the potential for recall bias, response bias and JSSP project reporting gaps, and undertook to mitigate any evaluation limitations due to these issues as much as possible. OVERALL ASSESSMENT A scorecard of JSSP efforts to date would suggest that JSSP has managed to get many things in place over the past two years to improve the KJC and functioning of the courts. However, most of these accomplishments are the result of highly labor-intensive efforts to embed experts and young workers in the relevant institutions. The level of ownership and institutionalization of these efforts remains in question, as does the ability of these institutions to continue to progress without such inputs. Most of these efforts have also been “opportunistic,” in the sense that the work took place where JSSP and KJC interests were aligned to address an issue. Further progress in the performance of the KJC and courts depends on clarification and resolution of a number of structural, legal and regulatory questions, some of which are more difficult than others. Without resolving some of these issues and shifting to a slightly more “tough love” approach, the probability of KJC achieving significant gains in performance, including vis-à-vis the courts and further integration of northern courts, is limited. Further progress also rests on connecting JSSP activities a bit more coherently — internally as a program, with other USAID programs, with other donors and with civil society groups. The integration of judicial structures in the north should parlay into functioning courts in the north of Kosovo; the performance of KJC committees and court presidents should be monitored and lessons should be learned; and gender empowerment objectives should be matched and tracked with issues of merit and transparency that have resonance with all stakeholders. JSSP’s preparatory efforts to date are extensive. Now is the time for the program to focus and strategize around key issue areas, and link these areas of the program to be more mutually reinforcing. CONCLUSIONS AND RECOMMENDATIONS EVALUATION QUESTION 1 JSSP assistance to date has further advanced the independence, professionalism, efficiency and accountability of the KJC and the courts, but with more significant gains in some areas than others. Strengthening the KJC has been a major focus of JSSP efforts; there are positive results in both procedure and leadership, such as improvements in policy and decision-making processes of the KJC and its committees, elevating the position of the KJC chair and revision of the law on the KJC to increase the number of full-time members. However, KJC institutional capacities are still relatively weak in strategic planning for the courts and the day-to day-management and monitoring of them. This is partly source sensitive document 3 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV due to the need to first resolve internal management issues, as evidenced by delays in approving the 2017/2018 Action Plan and restructuring the Secretariat. In relation to the courts, JSSP efforts have resulted in marked improvements in basic court transparency and a reduction in cases backlogs. Other court capacity and judicial quality improvements are still to be proven, however, as many of the regulations and tools (training programs, manuals, guides) developed with JSSP assistance are awaiting implementation, and the KJC leadership still needs to exhibit sufficient ownership and institutionalization of changes before knowing if the regulations/tools will be effective. Therefore, while JSSP has been successful in improving the self-governance of the judiciary as a whole, these gains are not sustainable without clearer ownership by the leadership in the KJC and courts and structures in place to support the changes — e.g., a restructured KJC Secretariat, a Cabinet for the chairman, a (de)centralized resolution process and performance criteria in both institutions that hold them accountable. Many of the achievements to date are based on the fact that key JSSP advisors are embedded in the KJC, and JSSP-supported Kosovar advisors, backlog reduction officers (BRLOs) and interns are performing routine activities at the KJC and courts. The KJC has started to make plans to engage more professional associates, but planning for sustaining overall JSSP gains is still in the early stages. Many of the achievements are also based on the fact that JSSP has seized available opportunities as they have presented themselves to make progress in specific areas. Whereas tackling some of the more difficult structural issues, such as genuine restructuring of the KJC Secretariat and fully addressing implications of the decentralization process and the state administration law require more time and effort. More significant gains during the remainder of the project appear to be linked to JSSP’s ability to plan for and focus more attention on these areas. RECOMMENDATIONS 1. Recommendations focus on encouraging JSSP to emphasize and zoom in on needed structural and legal efforts (however tough), while also putting in place plans that will allow sustainability and clearer monitoring of actual institutional performance gains. 2. Further strengthen critical members/aspects of the KJC, including the new full-time members and the chair, and the restructuring of the Secretariat. 3. Further support KJC and its Secretariat with development and implementation of a thorough restructuring plan. 4. Assist the KJC in developing a clear policy proposal for reform of the civil service to address judicial administration reform and development of a system to monitor recruitment. 5. Guide and assist the KJC in systematically assessing and reconsidering the decentralization process within the framework of the current restructuring. 6. Further develop case backlog efforts to ensure a system is in place and is sustainable. 7. Make the communication and transparency improvements more comprehensive and sustainable. 8. Further expand efforts to enhance the courts’ accountability to the KJC. 9. Review results of ethics training and modify commentary and other training materials as necessary. source sensitive document 4 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV 10. Continue to enhance to Mandatory Continuing Legal Education (MCLE) efforts. 11. Be ready to provide further support to the Constitutional Court of Kosovo (CCK) as needed. EVALUATION QUESTION 2 JSSP activities have been extremely beneficial to the initial implementation of the EU-facilitated 2015 Justice Sector Agreement (JSA, or Brussels Agreement). JSSP has played the role of international interlocutor and helped to put in place basic planning procedures to ensure that the many different actors would be ready when work on integration actively started after October 2017. JSSP has also played this role to help get the Mitrovica Basic Court, its branch courts and the Appellate Court Division to the basic level of functioning. In addition, JSSP has brought a needed youth perspective into the judicial process and system. However, JSSP results have not (yet) resulted in integration in the sense that the relevant institutions — the KJC and the Mitrovica Basic Court, its branch courts and the Appellate Court — directly, routinely and effectively cooperate. Rather, JSSP became the indispensable interlocutor, which has resulted in dependencies from both the basic courts in the north and the KJC; neither has much confidence or experience in dealing directly with the other and tend to use the JSSP as their primary point of contact. Integration has been “too important to fail,” and JSSP likely has prioritized this goal over the process in reaching the levels of integration to date. JSSP should continue to support the KJC to further integration and institutionalization of courts in the north. Balancing support for the KJC and encouraging the KJC to take charge is challenging, since KJC capacities are still low and the issues that need addressing — from case language assignments and proper translation to additional infrastructure upgrades — continue to increase. These technical issues already have political overtones, and can easily be further politicized in ways that could derail the integration process. JSSP still has a role to play in ensuring that issues receive attention in a proper way, particularly by continuing to encourage the KJC to include the courts in the north in overall reform efforts, but the time should be ending when a phone call to JSSP solves problems. No less important are the challenges and opportunities related to integrating young legal professionals from the north into the judicial system, building greater confidence within the general public for integration, and ensuring that judicial and law enforcement institutions are engaged and functioning. JSSP took steps to integrate youth through internship programs early on, but these need to be made more comprehensive to ensure that the path from practical experience to recognition of degrees, to passing bar exams and attaining employment is more straightforward. Public awareness helps, but public confidence is likely only after citizens see real functionality in the courts. To this end, JSSP has an opportunity to assist donors and the respective institutions to be more coordinated and focused on helping the judicial system promote the rule of law in the north in a way that functions and is efficient. This is not the case yet. RECOMMENDATIONS The recommendations below emphasize the need for JSSP to stay engaged in the north, just in different and more comprehensive ways. 1. Transform engagement from being interlocutor to being an assistant in relation to the KJC and the integration efforts of courts in the north. 2. Emphasize mainstreaming of judicial institutions in the north into countrywide reform processes. source sensitive document 5 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV 3. Amplify and design a comprehensive package of support for (youth) legal internships. 4. Strengthen the links between donor efforts and other justice sector institutions and programs for a more comprehensive approach. EVALUATION QUESTION 3 The evaluation team concluded that JSSP has been successful in managing civil society subcontractors to achieve specific activity objectives. Activities focused on providing technical training on communications, on conducting court satisfaction surveys, and on providing community awareness and facilitation services are complete or will soon finish, according to the subcontractor statements of work (SOWs). However, the management process has raised some concerns: the fixed-price contract mechanism is more suited to deliverables based on research, such as the court satisfaction survey, but is less suitable for capturing the process-focused work of facilitation and capacity building. Duplication of some efforts (e.g., a similar court survey conducted by another donor) and some of the facilitation work suggest that JSSP could have been more selective in its design of activities and that the subcontractors could have been more aware of related efforts by other stakeholders. In addition, the relatively little attention given to shared learning or integrating work among the three CSO subcontractors has reinforced their territorial (north vs. rest of country) competencies. Slight tweaks to subcontractor management and mechanisms could address these concerns, but the larger concern raised by informants related to this management process is what was missing and how it was conceptualized. While CSO activities have contributed to overall JSSP goals, the intended original focus on watchdog and monitoring work – which was later changed – remains a need. There are gaps where focused monitoring efforts on specific areas of KJC and court performance could more clearly complement JSSP’s work, and this alternative approach should be explored. RECOMMENDATIONS 1. Continue to support CSOs, but shift some of their work to watchdog efforts that could further complement the program’s goals and related advocacy efforts. 2. Consider using a subgrant mechanism with focused outcomes and accompanying monitoring processes. EVALUATION QUESTION 4 The evaluation team concluded that JSSP has been somewhat effective and efficient in its efforts to coordinate its work with other donor-assisted programs in the justice sector, with the KJC acting as a focal point of this coordination. However, even with the KJC, the coordination has tended to be informal. A similar dynamic has been at play with other USAID programs; where overlap or duplication of efforts exists, JSSP and other programs eventually found ways to (mostly) ensure complementarity. Sector-wide coordination is difficult; on a day-to-day basis, JSSP has displayed reactive cooperation more than forward-looking, proactive cooperation. In some situations, JSSP has been able to coordinate quite well, such as with early efforts in the north. Current coordination for the north of Kosovo is less apparent, even as the need is great, if not greater than before. These needs are technical but are also politically sensitive, and therefore it is strategically important to provide the right kind of support from a donor financing and coordination perspective. As the centerpiece USAID rule of law program, JSSP is playing the role of getting the necessary donor activity implementers around the table, but this role source sensitive document 6 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV could be further amplified and focused, particularly in assisting the KJC to effectively carry out this coordination role. RECOMMENDATIONS 1. Focus more on coordination with other donor projects in providing assistance to the KJC. 2. Emphasize donor coordination and prioritization planning with the KJC. 3. (Re-) Elevate JSSP’s role in donor coordination in the north temporarily, while ensuring that this role can be transitioned to the KJC. 4. For USAID: Review how current and planned justice programming overlaps with and/or complements JSSP efforts. EVALUATION QUESTION 5 JSSP has been somewhat effective in its activities to promote equal opportunities for women in the justice sector. It has put gender on the agenda in the KJC (again), and through this has created procedural guidelines for gender empowerment and opportunities to use gender as a criterion in the selection of new members. Beyond this top-level “setting the stage,” further improvements in gender empowerment require the will of KJC members to institutionalize gender equality considerations. Similarly, the JSSP has indirectly supported making gender a consideration for hiring in the courts. By encouraging adherence to the laws, JSSP has supported getting more women into leadership positions and has noted some successes during the program. At the other end, JSSP’s efforts to develop more young women’s skills – either through participation in training or through offering various intern program experiences – have laid the groundwork for providing entrance into the sector, but these need attention. In general, the training and internship experiences appear to have been useful and the skills and knowledge gained are being applied. However, JSSP’s efforts to promote women as change agents have had little traction. Program adjustments to focus efforts on youth and mainstreaming training and building skills for women appear to be correct and practical. But they have also not used framing that women might stand behind in clear numbers: one that focuses on merit in recruitment and advancement. This might be a way to galvanize women to support other women in the system and a way for JSSP’s top-down and bottom-up efforts to be mutually reinforcing. RECOMMENDATIONS 1. (Re-) Focus KJC attention on applying a gender empowerment strategy and track key implementation benchmarks of this as part of monitoring KJC performance. 2. Continue to use opportunities to encourage application of gender equality laws and policies and to champion women in leadership positions, but publicly and in coalition-building efforts stress that the key criteria for hiring should be merit-based credentials because this has a greater resonance with everyone. 3. Continue programming focused on supporting young people’s entry into the legal/judicial professions. Reinforce this by closer tracking of what interns learn and what skills/knowledge they utilize from trainings and practical work experience. source sensitive document 7 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV INTRODUCTION AND PROJECT BACKGROUND In February 2018, the Republic of Kosovo celebrated 10 years of existence as a country. While recognizing this milestone, Kosovo is still in early stages of developing and implementing democratic governance and economic reform policies, and the international community plays a significant role in Kosovo’s transition to a full market-oriented democracy. The international community is focused on providing support for institutional and policy reforms across sectors, building capacities of actors and institutions and supporting Kosovo’s Euro-Atlantic integration. The United States Agency for International Development (USAID) has been one of the largest supporters in Kosovo’s development since the end of the Kosovo-Serbia conflict in 1999 and continues to support development efforts as guided by the five-year (2014-2018) Country Development Cooperation Strategy (CDCS). Specifically, since 2001, USAID/Kosovo has focused substantial efforts on strengthening the justice sector through a series of sequential and complementary programs aimed at supporting Kosovo to put in place judicial institutions and practices that can meet the needs of its citizens. These include (but are not limited to) the Justice Support Program (JSP) from 2007-2011 and the Effective Rule of Law Program (EROL) from 2011-2015, Started in late 2015, the Justice System Strengthening Program (JSSP) is the centerpiece program of current justice sector efforts. The JSSP is a four-year rule-of-law effort implemented by Millennium DPI Partners during the period November 23, 2015, to November 22, 2019. Its overall goal is to improve the quality of the justice system in Kosovo through three program components as outlined below. 1. Strengthen the efficiency, effectiveness and service delivery of the Kosovo Judicial Council (KJC) and courts. Activities under this component have been implemented in thematic clusters designed to: (1) strengthen KJC operations, (2) improve KJC policy and regulatory processes, (3) functionalize budget planning, management and decentralization, (4) improve the capacity of court administrators and (5) strengthen court operations and case processing. 2. Enhance the accountability and professionalism of justice system institutions and actors. Activities under this component have been aimed at enhancing the accountability, transparency and professionalism of Kosovo’s judiciary by strengthening ethics and discipline (Activity 2.1), developing KJC communications and outreach (Activity 2.2), promoting Mandatory Continuing Legal Education (MCLE) (Activity 2.3), improving the quality of judicial decisions (Activity 2.4) and building Constitutional Court of Kosovo (CCK) capacity (Activity 2.5). 3. Support the functioning and integration of the judicial structures in the north of Kosovo. Judicial structures in north Kosovo have been inactive since the 2008 declaration of independence. The European Union (EU) brokered negotiations between Kosovo and Serbia, resulting in the 2015 Justice Sector Agreement (JSA) that provided broad parameters for integrating courts in the north into Kosovo’s system. Several subsequent agreements failed, however, to move integration forward. An August 2017 agreement for implementation of the JSA ultimately yielded the desired results. As agreed, new judges were appointed to Kosovo’s judiciary in October 2017, and court facilities opened in north Mitrovica, south Mitrovica, Leposavic and Zubin Potok. source sensitive document 8 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV As part of this process, JSSP has focused on activities aimed at: Activity 3.1 – Facilitating the Institutional Integration of Northern Courts; Activity 3.2 – Operationalizing Courts in the North; Activity 3.3 – Building Capacity of Human Resources; and Activity 3.4 – Engaging Civil Society and the Public. Throughout its programming, JSSP also has crosscutting themes, including gender empowerment. Per JSSP reports and program descriptions, gender sensitivity is factored into all program activities to ensure that women are empowered to participate, manage and lead. Specific areas of focus include development of gender-sensitive policies, laws and rules that promote equal participation of women. EVALUATION PURPOSE AND EVALUATION QUESTIONS The evaluation aims to provide USAID/Kosovo with a midterm assessment of JSSP performance to date. Specifically, the evaluation is meant to: 1) identify strengths and weaknesses in activity design and/or implementation; 2) identify gaps and/or opportunities where programmatic support might be warranted; 3) provide key findings, conclusions and recommendations for the JSSP activity; and 4) provide key recommendations for USAID’s new CDCS and potential future programming in the rule-of-law area. The Mission will use the evaluation report to make adjustments to the JSSP activity, share lessons learned and inform future programming in the sector. The Mission also anticipates that the evaluation report will inform planning for rule-of-law aspects of the new CDCS for fiscal years (FYs) 2019-2024. EVALUATION METHODS AND LIMITATIONS The evaluation team utilized a performance evaluation approach for EQs 1, 2 and 5 to examine relative progress and reasons for the performance. This included, as part of the interview process, a Most Significant Change (MSC) approach, which looks at both intended and unintended effects of change linked to the interventions. For EQ 5, which focuses on the impacts of crosscutting efforts on gender empowerment, the team also used a contribution analysis approach to gauge the extent to which JSSP efforts have contributed specifically to gender empowerment within the justice sector. This included designing an explicit theory of change (TOC), finalized with the JSSP leadership, outlining how specific JSSP interventions are intended to contribute to intended results. As part of designing the TOC, the amount of data gathered in relation to gender efforts was less than anticipated, but corresponds to the actual level of JSSP effort on gender in the program. Second, the team used a process-focused approach to assess EQs 3 and 4, in cooperation with CSOs and donors, respectively. The aim was to examine whether management systems and processes have been put into place, are relevant to the tasks at hand and have been effectively implemented. EVALUATION METHODS The team used mixed methods to answer the evaluation questions. This included a review of JSSP documents and secondary project data, key informant (KI) interviews and several group interviews. Prior to arriving in Kosovo, the team began their review of key JSSP documents, including work plans, Performance Management Plan (PMP) data and quarterly and narrative reports through the end of Year 2.1 Based on this review, the team sketched out descriptive overviews of program efforts to date for 1 The team did not receive the JSSP Year 3 Work Plan or Year 3 Quarter 1 Report until the first week of fieldwork. At that time, JSSP also provided copies of the first three years of project deliverables. source sensitive document 9 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV each question. The team members who were already in Kosovo also held five preliminary meetings with external informants to get general impressions of JSSP, contextual considerations and recommendations for additional KIs. Together, these efforts informed the design of semi-structured interview guides for six types of informants.2 After arriving in Kosovo, the team met with USAID/Kosovo to finalize selection of locations for site visits, and also met with JSSP leadership to agree on a TOC for EQ 5. The team also spent time together to go over initial evidence capture efforts and to refine questions for the interview guides. During the next two weeks the team conducted 59 semi-structured interviews with 91 KIs, including personnel from the four Basic Courts of Pristina, Gjakova, Gjilan and Mitrovica; and held meetings with KJC and JSSP personnel, CSO subgrantees, donors, external experts and civil society representatives. As part of these interviews, and particularly to gauge MSC effects in the KJC and courts, the team held three group meetings with mid- and entry-level legal professionals. This included interviews with 1) a set of backlog reduction officers (BRLOs) at the Pristina Basic Court; 2) a set of interns in Mitrovica participating in the Advocacy Center for Democratic Culture (ACDC) internship program; and 3) a set of legal officers in the KJC. Throughout the fieldwork, the team took an iterative approach to interpreting and synthesizing data. This included daily team meetings to share impressions and key points from interviews. It also included two longer weekend sessions focused on identifying initial findings and emerging trends across each activity area. For example, in relation to EQ 1, in examining the program’s efforts to strengthen the KJC, the team reviewed each JSSP input step by step, using evidence from key informants and program documentation, to understand what happened and any apparent effects. These efforts formed the basis of the team analysis workshop, where the same process was repeated to synthesize key findings, discuss what these suggested about larger trends to form initial conclusions, and to focus on areas of recommendations that address concerns noted in the findings and conclusions. EVALUATION LIMITATIONS The team would like to note several potential limitations. These include: RECALL BIAS: The team met with people from the KJC and courts, particularly those who have been cooperating with JSSP and other programs (funded by USAID or other donors) that in some cases have been very similar to JSSP. Accordingly, while the team focused specific questioning around engagement with JSSP, there is a chance that informants’ responses represented in some cases a mix of experiences with different USAID and/or other donor programs. RESPONSE BIAS: Since many of the informants were beneficiaries of JSSP in some way, it is possible that they presented a more favorable view of the program and its efforts. To try to address this, the team emphasized the confidentiality of the interviews, emphasized their interest in frank assessments and allowed informants to speak in their local language to express themselves better. JSSP REPORTING STYLE COMPLEXITIES AND TIMELY REVIEW OF KEY DOCUMENTS AND DELIVERABLES: In addition to the fact that the team did not receive the JSSP Year 3 Work Plan or Year 3 Quarter 1 Report and most JSSP deliverables until after arriving in the field, the JSSP narrative reports overall were less useful than anticipated for understanding the full range of project activities, specific inputs, and their effects. This might be partly due to the complexity of the program and its many 2 Please see Annex B for the set of draft guides. source sensitive document 10 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ongoing activities. While the team endeavored to cross-reference quarterly and annual reports, it found that the reporting style made it difficult to identify concrete inputs, effects and current status of certain activities. Therefore, the team spent significant time initially working with JSSP leadership and staff to understand the status of different project activities; this hindered more in-depth focus on some activities. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The findings, conclusions and recommendations follow for each evaluation question. EVALUATION QUESTION 1 To what extent have the independence, professionalism, efficiency and accountability of the Kosovo Judicial Council and Courts been advanced by JSSP’s assistance? ● Sub-question: To what extent has JSSP been successful in increasing the self-governance of the judiciary to better serve the public? INTRODUCTION At the beginning of the program, JSSP conducted a rapid stocktaking assessment of the KJC and courts. This formed the basis for much of the program’s work undertaken to date. As the findings and conclusions in this evaluation report indicate, progress on the priorities identified in the stocktaking assessment is at differing stages. To capture this progress in a systematic way, the findings are divided by thematic clusters, as designated by JSSP. Therefore, because the first evaluation question covers a substantial portion of the JSSP efforts to date, short conclusions follow for each thematic cluster under this question. The overall conclusions for the question are at the end. THEMATIC CLUSTER: STRENGTHENING THE OPERATIONS OF THE KJC Donor support to improve the efficiency and effectiveness of the KJC has been a multi-year effort and continues to top the donor agenda in the rule-of-law sector. The evaluation team found that JSSP has assisted the KJC with clearer strategic planning. JSSP guided the Council through a strategic planning exercise that established five midterm strategic priorities for 2017/2018. Based on discussions with the prior JSSP chief of party (COP), the evaluation team learned that JSSP did this to re-engage the KJC in the five-year strategic plan that was prepared under EROL, but was ignored during the gap in USAID programming support to the KJC. JSSP also helped the KJC draft an Action Plan for 2018. According to KIs in the KJC, with the midterm strategic priorities and action plan, the Council now has a clearer policy path and a new practice of setting midterm priorities that help it better discharge its constitutional mandate. However, the evaluation team learned that progress is still slow in this regard; the 2018 Strategic Plan had not yet been adopted by the KJC as of the end of April 2018. JSSP support to the KJC includes: • Legal drafting of 38 regulations, administration decisions and KJC instructions • Facilitation of countless meetings and discussions with the courts, donors, committees, etc. • Advisory services on management, integration of courts in the north, and policy issues • Mentoring on evidence based policy brief development and processes • Procedural guidance for functioning of the committees and Council • Technical guidance for the establishment of professional standards source sensitive document 11 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV The evaluation team found that JSSP efforts have resulted in better policy and decision-making processes. JSSP has supported the Council in re-establishing discipline and processes that had been lost during the gap between USAID projects. For example, the JSSP reports and interviews indicate that the Council developed a protocol for holding meetings and adopting policy. Now agendas are set in advance, and issues are well researched and developed into briefs so that Council and Committee members can make informed and considered policy decisions. During a group meeting with legal officers, the evaluation team learned that although they prepared agendas and memos before JSSP began, they did so for only the most important issues and the research was not as thorough. The evaluation team learned that once decisions are made, JSSP advisors guide the legal department through the drafting of regulations and supporting materials. According to JSSP advisors and KJC legal officers, both sides sit together to draft and discuss regulations. All members of the legal department remarked on the benefits of the collegial working group style introduced by JSSP. The evaluation team found that JSSP has elevated the leadership role of the chair, and this has led to better procedures for coordinating the Council’s external business. This is important, because the chair is currently the only full-time member of the KJC, and as such plays an essential role in coordinating KJC’s daily business with other institutions and acting as the leader and public face of the judiciary. JSSP has provided direct support to the chair of the Council by supplying two personal advisers. JSSP reports suggest, and the chair confirmed, that JSSP advisers help manage his daily agenda and prepare speeches and written statements that the chair delivers to other institutions, the media and the public.3 Both the chair and JSSP are advocating for the advisor positions to become part of a permanent Cabinet for the chair. In relation to KJC legislative and regulatory changes, the evaluation team found that JSSP seized the opportunity to improve the operational framework of the KJC when the Ministry of Justice (MOJ) drafted a new Law on the KJC. As described in JSSP reports and later by JSSP and KJC informants, JSSP worked with the KJC to develop and present a new model of membership for the Council that would increase the number of full-time members, thereby increasing the capacity of the KJC. The amendment provides that the chairs of the Normative Issues and Performance Evaluation committees would be full￾time members of the Council, and would suspend their judicial positions while serving on the Council. KJC leadership described their hopes for this model, particularly in elevating the leadership capacity of the Council and providing continuous, regular support for key Council committees. It is expected that the amended law will be adopted when the Assembly reconvenes. JSSP has provided advisory support to most operative units of the KJC, including (to name a few): The Department of Statistics, Department for Budget and Finance, Legal Department, Office of the Spokesperson and the Court Performance Review Unit (CPRU). Heads of all the departments that JSSP has worked with spoke highly of the technical assistance provided by JSSP experts and staff. For example, the head of the Statistics Department said the department could not have prepared the Kosovo Judicial Workload Assessment without the assistance of the two JSSP experts. He noted that even after five meetings, the 15-member working group had no idea how to proceed. Similarly, the head of the CPRU praised JSSP’s assistance in developing a 15-point template for court reporting that is being piloted in Year 3 of JSSP.4 Previously there was no set format, and the evaluation team was informed that court reports often exceeded 80 pages. The manager of the Case Management and Information System (CMIS) project5 noted that JSSP participates in the CMIS Steering Committee and attends all 3 This included assistance with the chair’s efforts vis-à-vis court integration in the north, which the section on EQ 2 covers more fully. 4 The Council of Europe (COE) is assisting the CPRU with development of the graphics for this template. 5 The Government of Norway finances CMIS, and the International Management Group (IMG) implements it. source sensitive document 12 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV meetings, and that JSSP’s case management expert provides strategic advice on CMIS. Evaluation team meetings with department heads of these units suggest that they are generally following the JSSP advice, but it is not clear what the actual levels of capacity and interest will be among department heads when they have to do this independently. The team found that JSSP also has provided some support for restructuring the Secretariat. Assistance in the restructuring began under EROL, but with limited results. The limitations, according to informants involved in the process, were due to changes in the Secretariat and Council leadership and competing understandings of the competencies of each position. JSSP and KJC informants describe the current effort as twofold: first, to establish smoother interactions between the Council and the Secretariat to make better use of administrative resources and ensure that decisions of the Council are continually implemented; and second, to ensure a better division of duties between administrative units so that their mutual work is more aligned according to a policy framework and courts are better connected to that framework. However, while both JSSP and KJC appear to have a shared goal, the KJC has been slow to move forward with the restructuring. This appears partly because the director of the Secretariat invests most of his current attention on efforts surrounding integration of courts in the north and decentralization (see the section on EQ 2). As a result, many in the KJC look to JSSP to suggest next steps and alternatives for moving some type of restructuring forward. Overall, the evaluation team found that JSSP works closely with the KJC using a labor-intensive advisory model; its legal advisors work directly with the KJC, and with other counterparts in the KJC. This has included embedding staff in the KJC, including JSSP’s judicial reform advisor and one of its legal advisors. The evaluation team found that the judicial reform advisor, who has worked on previous USAID projects in Kosovo, is the principal advisor to the KJC chair, committee chairs and the director of the Secretariat. The legal advisor’s focus is legal drafting, and she worked closely with KJC staff, particularly the legal officers, but also the Normative and Evaluation committees. These JSSP advisors are described by KJC informants as instrumental in introducing discipline into the policymaking process and in assisting staff to prepare issue documents for KJC consideration. This is a complete reversal from the KJC’s former modus operandi of dealing with all issues on an emergency basis without any planning or analysis. The evaluation team learned that the judicial reform advisor views his role as leading the KJC through the process, rather than doing things for them, and KJC officials, in their description of the work he does, appear to appreciate this “soft approach.” The legal advisor, working closely with the KJC, emphasized her role as a member of the team, but said it was important for the legal officers to view her as a support mechanism and recognize that they/KJC (not her and other JSSP staff) are the owners of their work products. The importance of JSSP’s role is evidenced by statements from KJC members such as, “JSSP helps us more than we help ourselves.” During a group discussion with KJC legal officers, it was noted that the difference between JSSP and other donor projects is that JSSP advisors are engaged in the daily business of the KJC and meet with KJC staff on a regular basis. Interviews with JSSP leadership and staff suggest that the JSSP leadership is keenly aware of the dependence of Kosovar institutions on the JSSP and the program is planning to slowly separate itself from these institutions. In relation to the role of JSSP and USAID more generally, several KIs (beneficiaries and external observers) at the KJC and other institutions remarked on the Kosovars’ respect for U.S. experts, saying that “if the advice is in English, we will listen.” They also remarked on donor-implementer overload and excess pressure from international advisors, which prohibits Kosovo people and institutions from growing. source sensitive document 13 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Conclusions JSSP has assisted the KJC to reach an early stage of improvement in efficiency and effectiveness. The JSSP also has assisted the KJC with important legislative and regulatory changes designed to further improve KJC efficiency and effectiveness, but approval and implementation of these changes is still pending. Overall, KJC performance gains can be linked to the close advisory model of JSSP with the KJC, specifically the embedding of staff in the KJC, but this has implications for sustainability. THEMATIC CLUSTER: FUNCTIONALIZING BUDGET PLANNING, MANAGEMENT AND DECENTRALIZATION The KJC decided to delegate the main operational responsibilities of the Council (human resources, procurement, and budgetary planning and implementation) to the courts approximately three years ago.6 As the decision was made before JSSP started, JSSP efforts have focused on support for decentralization. The evaluation found that the manuals JSSP developed were useful, but they were only partially successful in assisting in the complex decentralization process. JSSP developed manuals on human resource management, budgetary planning and management, and public procurement to support the work of court officials in carrying out decentralized tasks. While an informant at the KJC suggested that the budget manuals in particular have been beneficial for court officials to understand how the process should be conducted, and court informants described using the manuals,7 the process appears to still have been fraught with challenges. Most of the judicial officials across the four courts the team visited pointed out the problems in local procurement and hiring (particularly for smaller courts). Informants across the four courts visited also expressed frustration with the KJC’s communication and guidance on different budgetary and reporting tasks. For example, one court official explained that due to rigid budget guidelines,8 his role was officially budgeted as an information technology (IT) specialist position, when in fact he worked in the caseload management office. Overall, the evaluation team could not determine what part of the confusion was due to the newness of their tasks, the actual capacities of staff or the quality of KJC guidance. The level of court staff’s frustration with this process, however, was clear. The evaluation team found that JSSP’s later role in assisting the KJC to reconsider the decision to decentralize the main operational responsibilities of the Council has assisted the KJC to take a more systematic approach to the issue. Similar to its advisory role on other issues, JSSP staff described how they helped the KJC begin to systematically assess and review what needs to be done, and to work with court presidents on this effort.9 According to both KJC and JSSP staff, the JSSP has supported the KJC Secretariat and courts by setting the framework for discussing the effects of decentralization, and identifying the problems from the review that need be addressed (this process is currently being run by a working group of the Council). JSSP has offered alternative perspectives on aspects of the review, including issues that could be reconsidered as part of decentralization. However, this process is ongoing and a sustainable solution is yet to be found by the project to suggest to the Council. 6 The administration instruction was adopted in early 2015. 7 Evaluation team reviews suggested that the guides were of good quality. 8 It should be noted that the rigid budget guidelines are dependent on problems with job classification/job titles, and the KJC’s inability to resolve this with the Ministry of Public Administration. 9 While all court presidents that the evaluation team spoke with were critical of the process, some had more interest than others in preserving different parts of decentralization (e.g., the parts on staff hiring). Others had different positions on procurement depending on court size. Such issues are part of what informants describe as the planning work (and compromises) that need to be undertaken in the next phase. source sensitive document 14 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Conclusions JSSP’s development of how-to manuals to aid in the decentralization process appears to have had limited effect on easing confusion, but later efforts to assist the KJC to systematically reconsider the process and delegated responsibilities appear to be more useful. THEMATIC CLUSTER 3: IMPROVING THE INDEPENDENCE AND CAPACITY OF COURT STAFF The evaluation team found that JSSP has undertaken preparatory work on court administration training, but the training has not yet taken place. As described in JSSP reports and by JSSP informants, JSSP also worked with the Kosovo Judicial Academy (KJA) to create a training program10 on judicial administration based on the four manuals developed to support decentralization. The JSSP Year 3 Work Plan indicates that the training is expected to begin in Quarter 2 of Year 3. KJA feedback suggests that JSSP support to the KJA for this process has encouraged the KJC and KJA to introduce mandatory training on judicial administration. In parallel, the evaluation found that JSSP has assisted the KJC to identify alternatives for regulating the status of judicial staff, but this process is ongoing. One of the main problems in judicial administration is the largely unclear status of civil servants working in this sector, and the applicable law regulating their position in central administrative structures. To address this policy gap, JSSP reports describe how JSSP has reviewed several alternatives for regulating the status of civil servants in the judicial sector, and has supported this process within the purview of proposed amendments to the law on state administration. JSSP experts prepared a concept paper highlighting three options for the regulation of judicial administration staff.11 Although the concept paper appears to the evaluation team as a strong basis for drafting a new law on judicial administration, informants describe how other policy barriers have prevented the process from going further. Among other challenges, the financial impact of the new law is a major issue as the law needs to comply with the general process of public administration reform that the government is undertaking in cooperation with the EU. In addition, the new law on salaries that is expected to set harmonized salary levels for all public-sector employees. Efforts in that regard are ongoing, and a more comprehensive consideration of this policy is a precondition for embarking on the drafting of the law.12 Conclusions JSSP has undertaken preparatory work to put in place training on court administration and also has provided legal advice on regulating court administration staff, but both of these efforts are ongoing and dependent on clarification of the legal status of the staff of the court administration. THEMATIC CLUSTER: STRENGTHENING COURT OPERATIONS AND PROCESSES One of the greatest challenges faced by the Kosovo Courts is the reduction and prevention of case backlogs. The KJC adopted a Backlog Reduction Strategy in 2013 that allowed each court to develop individual case management plans, and there was some success in reducing backlogs under USAID’s 10 As described to the evaluation team, this includes some training materials from the other USAID programs, CLE and PRP. 11 Please see Year 1 Deliverables: Concept Paper on Judicial Administration. 12 As part of this process, the law on salary has already been drafted by the MPA and MoF. It has been supported by SIGMA, and it is now a precondition by the EU for direct budgetary support for PAR. source sensitive document 15 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV EROL and Contract Law Enforcement (CLE) programs. However, the backlog of cases remains substantial. The evaluation team found that JSSP developed an evidence-based and focused backlog reduction and prevention strategy. At the start of the JSSP program, staff conducted an inventory at its partner courts which showed a backlog of 39,000 cases as of December 31, 2015.13 Next, JSSP’s experts reviewed the backlog and developed a detailed, step-by-step approach to reduce the backlog and improve case flow management, with the goal of reducing backlog by 80 percent during the four-year JSSP program. JSSP began by piloting backlog reduction activities in four basic courts and subsequently expanded into branch courts. By the end of Year 2, the backlog in those courts had been reduced by 59 percent.14 The evaluation team found that the key to significant progress in backlog reduction was JSSP’s engagement of young legal professionals known as backlog reduction officers (BRLOs).15 Focusing on backlog, defined as cases over two years old, the BRLOs support judges in the whole cycle of case adjudication, starting with review of admissibility criteria and requests for missing information to the drafting of the decisions, including judgments. During interviews with the evaluation team, all judges and court administrators that worked with the BRLOs noted the high quality of their assistance, said that the BRLOs were essential to reducing the backlog, and indicated that they hoped to be supported by such teams in the future. Based on meetings with BRLOs, the evaluation team found that they had excellent qualifications, were enthusiastic about their work and were eager to secure full-time employment in the justice sector. JSSP court administration advisors informed the evaluation team that JSSP has taken great care in recruiting BRLOs, starting with an initial review of applications, followed by a written exam and in-person interview — a process that took over a week to complete.16 The team also learned that BRLOs received initial training and continuous monitoring by JSSP staff. JSSP is supporting the backlog prevention effort with additional interns that screen civil cases when they are filed at the court. These interns contact parties and lawyers to obtain all necessary documents and information, and in so doing are “training” parties and the courts to respect filing requirements. The screening process was successfully piloted at the Pristina Basic Court17 and then extended to the Gjakova Basic Court and Podujeva Branch of the Pristina Basic Court. However, the evaluation team found that there is currently no sustainability strategy in relation to BRLOs and interns. The evaluation team understands that JSSP will hire additional BRLOs/interns (in addition to retaining existing ones) to continue with backlog reduction and screening.18 Based on 13 See JSSP Year 1 Work Plan. 14 See JSSP Year 2 Annual Report, p. 12. 15 Backlog exists in all court departments, with the highest percentage in the general department and the lowest serious crimes. A large majority of backlogged cases have either passed their statutory limitations and/or are already prescribed. Though they stand in such a procedural status, they need be disposed of with a court decision. 16 JSSP received over 200 applications for 15 positions. 17 Approximately 60 percent of cases were returned to plaintiffs for correction or payment of fees. See JSSP Year 2 Annual Report, p. 13. 18 The evaluation team also learned that the KJC is in the process of securing more legal associates, however this is still not implemented. How to keep BRLOS and interns in the system? Court officials find value in the work of interns, but have not found a way to “keep them in the system.” When asked whether there could be funding for such interns in the future, one court president was clear that he would like to find funds, but the budgeting system did not allow for this. Other court professionals focused on hiring regulations, stressing that there was not a lot they could do to ensure that young talent could get into the system without new provisions in the law. source sensitive document 16 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV interviews at the KJC and courts, the team learned that there is currently no plan by the KJC or courts to make these jobs permanent, which risks a renewed build-up of backlogs. KIs from the CLE program informed the evaluation team that this is already the situation with BRLOs supporting enforcement actions; CLE will be closing and is no longer employing BRLOs. Furthermore, discussions with BRLOs and interns indicate that these highly qualified, young professionals face difficulties finding permanent employment in the justice sector. For example, several applied to announced vacancies, but none were hired. Conclusions JSSP’s focus on using BRLOs and interns to address court backlog reduction and prevention has resulted in a significant reduction in backlogs to date; however, the sustainability of this approach is still unclear. THEMATIC CLUSTER: STRENGTHENING ETHICS AND DISCIPLINE The evaluation team found that JSSP has assisted the KJC on drafting ethics codes and focused, practical training materials, but that testing to identify any effects from these efforts will only take place after the Year 3 trainings. During Year 1, JSSP assisted the KJC with the drafting and adoption of a new Code of Ethics for judges19 consistent with the Bangalore Principles, as well as with a regulation clearly defining misconduct, and KJC delivery of regional ethics training in cooperation with the Kosovo Academy (KJA).20 According to JSSP reports and KIs, despite the ethics training and regulation, judges were still confused about how to apply the Ethics Code. Accordingly, JSSP engaged a short-term expert to support drafting of a set of commentaries for the code that reflected the Kosovo reality, and the expert worked with the KJA to upgrade training materials.21 The evaluation team found that JSSP has done significant preparatory work for enhancing the judicial evaluation process in Kosovo, but that the quality of the process still needs strengthening. JSSP efforts have included assisting the KJC in drafting a Regulation on Performance Evaluation, and development of the standardized process required by the regulation that includes a structured scoring methodology, templates and a users’ guide. KJC members22noted that this methodology, which includes qualitative factors, was an improvement over the previous process that was based solely on the quantity of decisions produced. However, use of the new methodology was problematic. Of 70 judges evaluated in 2017, most received very good ratings, a small number were rated excellent and none were rated unsatisfactory. KJC and JSSP informants, reflecting on the process, noted that because there has been no training for committee members, the evaluation criteria were not applied as intended.23 According to JSSP KIs, the project is now working with the KJC to refine the evaluation criteria, which will include a review of judges’ decisions. The evaluation team found that JSSP has undertaken efforts to encourage proper and transparent disciplinary procedures within the KJC, but that progress is dependent on finalization of the Law on Disciplinary Liability for Judges and Prosecutors. Despite initial JSSP progress in improving the process used by the Office of Disciplinary Counsel (ODC) for handling complaints and reporting to the KJC, later reports describe how the work stalled when the MOJ started to draft a new law on disciplinary 19 According to JSSP reports, the project also prepared an Ethics Code for Court Administrators (non-judicial staff). 20 This was the predecessor to the KJA. 21 The expert held a refresher training of trainers (ToT), and a series of regional ethics trainings began while the evaluation team was in Kosovo. Given the timing of the training, the evaluation team was unable to meet with the trainers or trainees. 22 The KJC’s Performance Evaluation Committee (PEC) is composed of 13 members: the Chair and 12 judges (five from the Supreme Court; four from the Appellate Court; and three from the Basic Courts). To ensure that the PEC is truly independent, the 12 judge members are not on the KJC, and the Chair of the Committee cannot participate in the evaluations. 23 For example, committee members did not use the new scorecards included in the manual and how to measure qualitative criteria like communication skills was unclear. source sensitive document 17 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV liability for judges and prosecutors that might eliminate the ODC.24 As described in JSSP’s Annual Report,25 JSSP then switched its focus to working with the KJC’s Disciplinary Committee and supported the publication of disciplinary decisions and developed a set of “portable best practices” for the disciplinary process. The Law on Disciplinary Liability for Judges and Prosecutors is currently under review. According to JSSP, once the law is passed, JSSP intends to resume work on judicial discipline. Conclusions JSSP has contributed to the development of a framework for increased accountability and integrity of judges, but implementation of the new standards is only beginning. THEMATIC CLUSTER: DEVELOPING KJC COMMUNICATIONS AND OUTREACH As described in JSSP’s Year 1 Annual Report, the program had originally planned to focus its efforts on communications and outreach at the KJC by developing a strategic communications plan. However, when it became clear that many donors and projects were already working in this area, JSSP decided to concentrate its efforts at the court level. The evaluation team found that JSSP assistance with the websites and social media pages of the Constitutional Court of Kosovo (CCK) and basic courts improved the websites’ search functions, as well as visibility of the court online. JSSP supported an upgrade of the CCK website and will do the same for website of the KJC in Year 3 (Basic Courts each have their own page on the KJC portal). KIs at the CCK praised the design of the new website and the improved search function.26 JSSP also assisted the CCK, KJC and Basic Courts with setting up Facebook pages, developed a protocol for content development and management,27 and placed interns to support outreach through social media in the Basic Courts of Pristina and Prizren.28 In addition, JSSP contracted with the Balkan Investigative Reporting Network (BIRN) to facilitate live communications between the court and citizens and to organize live streaming sessions at the Pristina Basic Court; three of the nine planned were held by April and drew over 20,000 viewers.29 During meetings at the courts, the evaluation team found that judges, public information officers (PIOs), court staff and interns were all positive about the Facebook pages and live streaming events, and felt that these were important developments in improving the public’s understanding and perception of the courts.30 According to JSSP reports, the age of typical visitors to these Facebook pages is between 25 and 35.31 The evaluation team found that JSSP has continued to strengthen the role of PIOs, a position that was introduced under EROL. The support has consisted of trainings and provision of interns. In meetings with PIOs, the evaluation team learned that they perform a variety of functions, including posting decisions on the court’s website, updating social media pages, responding to requests from the media and public and organizing public outreach activities. These PIOs reported that JSSP has conducted trainings on general communication skills, publication of decisions and use of social media. The interviewed PIOs were particularly enthusiastic about the social media training conducted by BIRN. 24 See JSSP Year 1 Annual Report, p. 20 and Year 2 Annual Report, p.15. 25 JSSP’s original Statement of Work (SOW) was amended in Year 2 to eliminate work with the Office of Disciplinary Council. 26 As explained to the evaluation team, this function will make it easier for legal clerks to find cases based on keywords, therefore helping them to be better informed and to manage the opinion delivery process more smoothly. 27 The KJC and Basic Courts of Pristina, Prizren, Gjilane and Ferizai have Facebook pages. The typical Facebook page visitor is between the ages of 25 and 35. 28 The three-month term of the social media intern at the KJC has expired. 29 The remaining sessions were completed by the end of May 2018. 30 One specific point of feedback and interest among judges was ways to protect the integrity of judges’ personal webpages; JSSP provided them with guidance on what judges should personally share and what they should professionally share. 31 See JSSP Year 3, Quarter 1 Report, p. 17. source sensitive document 18 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Almost every KI (judges, external observers, representatives of CSOs) noted the importance of the PIOs in improving the public’s access to information about and perception of the courts. One PIO told the evaluation team that they had their first meeting with the KJC and director of Secretariat in February 2018 and that the KJC was supportive. However, the KJC was unable to respond to PIO training requests (e.g., on public speaking, press releases, etc.) due to a lack of funding. To partially address this, JSSP supplied an intern to help the Pristina Basic Court PIO to help post decisions to the KJC website. According to the PIO, that intern assisted with other tasks as well. The KJC spokesperson had similar praise for the JSSP intern that worked at the KJC for three months. The evaluation team notes that JSSP has continuously exceeded its target for Indicator 18, which measures the number of USG-supported activities that improve collaboration between the KJC and courts with civil society and the media.32 JSSP and BIRN KIs reported that they consider the Prizren Basic Court to be the leader in public outreach and communication. JSSP staff explained that the Prizren Basic Court referred to the guide developed under EROL and decided what parts of it they wanted to use, and now other basic courts are copying what Prizren is doing. Conclusions JSSP activities have increased transparency and improved communications between the KJC, Basic Courts, CCK and the public. However, there is still more that can be done in terms of support for PIOs and their capacity and with expanding public visibility of the courts among older demographics. In addition, the sustainability of these gains is unclear. THEMATIC CLUSTER: PROMOTING MANDATORY LEGAL EDUCATION The evaluation team found that JSSP support has been critical to progress on developing minimum Mandatory Continuing Legal Education (MCLE) standards for judges, an area in which judges have generally been resistant. Through discussion between the KJC and judges, facilitated by the KJC, an agreement was reached to form a MCLE working group in April 2016. JSSP then supported the MCLE working group by facilitating meetings and providing summaries of research on international standards on MCLE and best practices.33 JSSP KIs said that judges still could not agree on the scope of MCLE, so JSSP broadened the discussion to include leadership and management, which the judges were interested in. JSSP KIs further noted that to reach an agreement and adoption of the Regulation on MCLE, JSSP had to set a low bar of four hours of mandatory ethics training in the first year, and additional training for less experienced judges.34 KIs at the KJC and KJA noted JSSP’s continuous efforts to advocate for adoption of MCLE standards. More importantly, they noted that the KJC will have to overcome resistance among some judges to MCLE. According to JSSP’s Year 3 Work Plan, the project will monitor implementation, assess results and then determine whether to expand the requirements for 2019.35 Based on talks with KIs at the KJA, the evaluation team determined that training programs on topics other than ethics would need to be upgraded to obtain the interest and attention of judges and to have a meaningful impact on their performance.36 This is consistent with statements by external KIs, including 32 The title of the Indicator does not include the courts, but court-level activities are included in JSSP’s measurements. 33 See the JSSP MCLE Briefing Memo and Recommendations. 34 For now, this covers only judges who have been scored as unsatisfactory by the performance evaluation committee. 35 It is anticipated that additional training requirements will be added in the coming year, including a provision to require specific training to address deficiencies identified during performance reviews. Pursuant to the MCLE regulation, the KJC established a Training Committee that has drafted an action plan and draft training plan for 2018. 36 Experts noted that additional subject matter areas such as consumer protection, company law, etc., are needed and could pique the interest of judges. source sensitive document 19 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV people that have delivered training at the KJA. These informants noted that the training at the KJA has not been very effective and that the quality of training and trainers needs improvement.37 A new regulation on trainers has been adopted which will require an additional training of trainers (ToT), and KIs stated that the KJA will need assistance with this and would also benefit from a training impact assessment. Conclusions New professionalism standards have been prepared and adopted by the KJC with JSSP assistance, but are still in the early stages of implementation. THEMATIC CLUSTER: IMPROVING THE QUALITY OF JUDICIAL DECISIONS The evaluation team found that JSSP’s guide for publication of judicial decisions has had little effect to date. During Year 2, JSSP published a Legal Writing Handbook for Judges that was prepared in consultation with the KJA and delivered to all judges.38 JSSP started to conduct workshops in Year 3 to encourage use of the Legal Writing Handbook. Judges interviewed who have attended this training said that the handbook and training were well designed and engaging, and were well received. This positive reception of the handbook was confirmed in meetings at the KJA. Similarly, the JSSP and KJC developed standard operating procedures and a decision template that would significantly reduce the amount of private information included in decisions. These were approved during Year 2 but have had mixed effects. The BRLOs and interns interviewed reported that most judges do not want to use the templates, a standard format for publication does not exist, and that judges insist on using their personal format(s) and require the BRLOs and interns to comply. Still, external KIs and judges reported that the publication requirement has made judges more attentive to the quality of their opinions, which has potentially far-reaching effects on decision quality and transparency, according to external experts. The evaluation team found that a key reason for the increase in publishing of decisions was the efforts of JSSP to develop standard operating procedures for redaction and hiring interns to perform the redactions.39 From less than 100 published decisions in November 2016, the number reached over 3,700 as of February 2018.40 The evaluation team notes that despite the significant progress in publication of decisions, JSSP did not meet the targets set for Indicator 19 of its PMP ("percent of final court decisions posted on websites). JSSP reports and interviews indicate that the target has not been reached due to the failure of the courts to provide sufficient staff to collect and redact court decisions. However, the Pristina Basic Court, which has several JSSP interns, has exceeded the target. Conclusions JSSP has been instrumental in achieving higher publishing levels of judicial decisions, which will go some way to improving decisions. However, this process is still dependent on JSSP interns, is not standardized and judges appear to need clear incentives and engagement for further progress in this area. 37 The EU Support to Free Legal Professions project is preparing trainer profiles. 38 KJA informants who participated in preparation of the handbook reported that the three JSSP experts involved provided useful input on the structure and teaching methodology — for the handbook and trainings. The KJA also informed the evaluation team that it intends to incorporate the handbook into its revised legal writing training module. 39 JSSP BRLOS and interns are at the Pristina, Prizren and Ferizai Basic Courts. 40 See JSSP Year 3, Quarter 1 Report, p. 20. source sensitive document 20 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV THEMATIC CLUSTER: SUPPORTING THE CONSTITUTIONAL COURT OF KOSOVO (CCK) USAID programs have provided significant support to the CCK since its inception in 2008. This has continued under JSSP. CCK KIs unanimously praised USAID, and specifically JSSP’s assistance, stating that JSSP consulted with the Court on its needs and that this was reflected in the JSSP work plan. JSSP support has particularly been important, as the Court has prepared for the transition to a full complement of Kosovar judges later this year.41 The evaluation team found that JSSP has further strengthened rules and procedural standards of the CCK. All CCK KIs (judges, legal advisors and staff) commented on the excellent assistance provided by a JSSP expert consultant who was an Honorable Chief U.S. District Judge who had assisted the Court in drafting the rules of procedure under EROL, and facilitated a revision of the rules under JSSP.42 Judges remarked on the judge’s ability to facilitate discussion and assist the judges in reaching a consensus on key issues that the judges had been struggling with. As some issues remain unresolved, JSSP engaged the judge again in April 2018 to finalize the rules.43 The evaluation team found that JSSP support has increased the functionality of the CCK website. JSSP has supported improvements in transparency and outreach started under EROL. This has included development of a new website and improvements in the classified document management system (CDMS). KIs at the Constitutional Court noted that the design of the new website, with greatly improved search and internal communications functions, was excellent. KIs also noted that defects in CDMS have been eliminated. To further improve the Court’s outreach and communication efforts, the PIO has identified several areas where continued donor support is needed.44 The evaluation found that JSSP only partially developed an appropriate orientation package for new judges to the Court, but that this is being revised during Year 3. To support the addition of five Kosovar judges to the Court, during Year 2 JSSP engaged a local expert to assist with the preparation of the orientation package. Judges at the Court informed the evaluation team that because the Court was not satisfied with the section on comparative constitutional case law prepared by its expert, JSSP has engaged another local expert to revise this section and finalize the orientation package.45 Conclusions JSSP has furthered efforts of the CCK to prepare for its transition to a full complement of Kosovar judges, but additional targeted assistance may yet be needed. CONCLUSIONS JSSP assistance has advanced the independence, professionalism, efficiency and accountability of the KJC and the Courts, but with more significant gains in some areas than others. Strengthening the KJC has been a major focus of JSSP efforts to date, and there are positive results both in the procedure and 41 The evaluation team notes that JSSP has not met Target 22: “Management systems at CCK improved as measured by percentage increase in managements and capacity elements rated as ‘performing’ on CCK Institutional Scorecard.” Progress was made in Year 2, but further progress was delayed due to the late addition of new judges and other issues. 42The purpose of the revision was to resolve conflicts and questions raised by the Court’s judges in applying the rules during 10 years of practice. 43 See JSSP Annual Report, Year 3, Quarter 1. 44 These include: a public opinion survey that identified gaps and barriers in communication between the CCK and the public (to inform the Regulation on Communications that the Courts plans to finalize in 2019); a public information campaign; and a new server (as the Court has only one server and bears the risk of losing all data if that server fails). 45 In further support for the addition of the new judges and transition to an all-Kosovar panel, JSSP will conduct a study visit to the U.S. source sensitive document 21 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV leadership, such as improvements in policy and decision-making processes of the KJC and its committees, elevating the position of the KJC chair and revising the law on the KJC to increase the number of full-time members. However, KJC institutional capacities are still relatively weak in their strategic plan for and day-to-day management and monitoring of the courts. This is partly due to the need to first resolve internal management issues, as evidenced by delays in approving the 2017/2018 Action Plan and restructuring the Secretariat. In terms of the courts, JSSP efforts have resulted in marked improvements in basic court transparency and backlog reduction. Other court capacity and judicial quality improvements are still to be proven given that many of the regulations and tools (e.g., training programs, manuals, guides) developed with JSSP assistance are awaiting implementation. The leadership of the relevant institutions still need to exhibit sufficient ownership, and the changes need to be institutionalized before it will become clear if they will be effective. Therefore, while JSSP has been successful in improving the self-governance of the judiciary, the gains to date are not sustainable without both clearer ownership by the leadership in the KJC and courts and clear sustainability planning. Many of the achievements are based on the fact that key JSSP advisors are embedded at the KJC, and the JSSP-supported Kosovar advisors, BRLOs and interns are performing routine activities at the KJC and courts. However, the KJC does not have a plan for sustaining these gains and making further progress without assistance from JSSP or another USAID program. Many of the achievements are also based on the fact that JSSP has seized available opportunities as they presented themselves to make progress in specific areas. Whereas tackling some of the more difficult structural issues such as genuine restructuring of the KJC Secretariat, and fully addressing implications of the decentralization process and the state administration law require more time and effort. More significant gains during the remainder of the program appear to be linked to JSSP’s ability to plan for and focus more attention on these areas. RECOMMENDATIONS 1. Strengthen critical aspects of the KJC, including: ● Continue to support the KJC to implement its midterm strategic priorities and the action plan for 2018 with a focus on the practical implementation of regulatory frameworks and general legislation. Support for development of a longer, five-year strategic plan and annual plans for the years to come should also be considered. ● Support the incoming full-time members of the KJC46 with an orientation and guidance that covers both long-term and short-term issues such as the division of labor between them and the Chair, focused action plans for tackling issues currently facing their committees, and clear reporting lines between them and the Secretariat. ● Further support the KJC legal department’s work with committees on critical issues. For example, this could include continued support for the department’s work with the Performance Evaluation Committee, further enhance of rules and procedures for performance evaluations, improved procedures for collecting and reviewing data, and work on privacy issues. For the Normative Issues Committee, support completion of the inventory of 46 This assumes that the law will pass the Assembly of Kosovo. source sensitive document 22 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV applicable regulations, and timely and clear tracking of developments with regulations and legislation. For the Disciplinary Committee, continue to support improvements in review of disciplinary submissions coming from the ODC (or the institution that replaces it), better rules and procedures for evaluating evidence, and timely publication of disciplinary decisions. ● Continue to further support improvements in the Chair’s leadership role, including for daily business of the KJC. This includes supporting the Council to establish a cabinet, clearer regular communication with the courts, more dynamic engagement of relevant stakeholders in consultations with the KJC, more transparency and engagement with CSOs monitoring the KJC and courts, and a strategic positioning of the Chair vis-à-vis the government and Kosovo Prosecutorial Council (KPC). ● Focus on sustainability through localization of some of the JSSP advisory positions in the KJC. 2. Further support KJC and its Secretariat with development and implementation of a thorough restructuring plan. Three steps need be undertaken in this regard: ● Develop a thorough plan for restructuring that covers not only structural problems but also behavioral shortcomings within the Secretariat, court administration and among others working in the KJC/judiciary. ● Support the Secretariat and the KJC with implementation of this plan, and coordinate with other donors to ensure that aspects not covered by JSSP are covered by others. ● Conduct systematic monitoring of the progress with implementation of the plan, with a view to mitigating anticipated and observed risks. 3. Assist the KJC to develop a clear policy proposal for reform of the civil service in order to address judicial administration reform and creation of a system to monitor recruitment practices. Specifically: ● Continue to assist the KJC with development of a legal framework for civil servants that gives the KJC the ability to attract highly qualified professionals. ● Assist the KJC to design a systematic process for monitoring the recruitment of civil service staff in all courts to ensure respect for the principle of meritocracy and appeal procedures. 4. Guide and assist the KJC in systematically assessing and reconsidering the decentralization process within the framework of the current restructuring endeavor. Specifically: ● Further support the KJC working group on decentralization in finalizing their review and in developing an action plan and develop concrete recommendations to overcome the current problems. ● Once the working group has decided the way forward, support the KJC and possibly MOJ in drafting amendments to ensure that international standards and best practices are reflected and that the amendments are in full harmony with existing legislation (laws on procurement, public financial management, and civil service, etc.) source sensitive document 23 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ● Support additional training of court staff on procurement, budget planning and expenditures, auditing, hiring, etc. 5. Support additional efforts to reduce case backlog, and ensure that a sustainable system is in place for this. Specifically: ● Support the KJC to adopt a complete case workload policy that is able to more precisely estimate needs for court administrative staff, judges and other resources, and develop a budgetary plan to fulfill these needs for the next 10 years. ● Support the KJC to go from a limited application to a comprehensive application policy on 'case screening,' across all courts and proceedings including procedures for case clearance, communication with claimants and their defense councils, and judicial oversight of the screening process. ● Continue to provide BRLOs to the selected courts while simultaneously assisting KJC with development of a new backlog reduction strategy to ensure that i) the current backlog is addressed with existing resources (including around 50 new judges who will start taking new cases in September 2018), and ii) additional donor support is obtained for a limited time. Offer support to the KJC to translate this policy into an action plan. ● Support the KJC to develop a program for interns who would play a similar role to the BRLOs, and organize it so that the best students are recruited through their law faculty rather than directly via KJC recruitment. 6. Make the communication and transparency gains more comprehensive and sustainable. Specifically: ● Support the KJC to adopt a decision requiring courts to publish summaries of their quarterly reports on their websites. This should include data on case clearance and other performance￾based indicators that can be understood by an ordinary citizen. ● Continue supporting courts to publish their decisions, with the target that all courts publish their new decisions in the next two years. This includes supporting the KJC to require the courts to follow the KJC’s redaction policy and template, and to establish a sanctions policy for failure to publish. ● Support the KJC to include publication of decisions in performance evaluation criteria for judges, and develop a clear policy that defines whose task it is to do the actual publication. ● Offer systematic training to court PIOs and presidents/judges on legislation pertaining to access to public documents, laws on personal data, and access to case files. ● Further support the KJC and courts to fully manage their websites and to adopt a decision to assign the task of website maintenance to specific administrative staff in each court. At the same time, offer trainings to PIOs based on best standards for news development, media access and standards of judicial ethics. 7. Further expand support efforts to enhance courts’ accountability to the KJC. JSSP should support the development of a practice/system that increases the courts’ responsibility towards the KJC. This should include the following: source sensitive document 24 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ● Design and put in place a system that is result-based, objective, substantive, and leads to competition between courts on best performance. The system should require reporting on a quarterly basis, and cover aspects of case clearance (with special emphasis on the numbers and quality of judgments), complaints from the public, publication of decisions, international judicial cooperation, and general managerial aspects of court functioning. ● Support the Council to establish a system for reviewing court presidents’ reports and analyzing their progress/challenges on a quarterly basis. 8. Further focus on ethics training and link this with disciplinary options. Specifically, JSSP should: ● Review the results of the ethics training and ToT to be conducted during this quarter with the KJA and assist the KJA with any improvements that need to be made to the training or materials, including the JSSP developed ethics commentary. 9. Consider further enhancements of MCLE efforts. JSSP should support the KJA on development of an enhanced curriculum that addresses the needs of the judiciary more systematically and objectively. Specifically: ● Add new aspects of education covering areas of law such as trade agreements, with a special emphasis on the EU Stabilization and Association Agreement (SAA) and other free trade agreements (FTAs), consumer protection, company law, etc. ● Support the KJA and KJC to ensure that the MCLE regulation provides for a parallel system of progress between judicial promotion and continuing education. 10. Be ready to provide further support to the CCK as needed. This may include both advisory and technical services, such as: ● Support to the CCK for finalizing the orientation package for new judges and organizing specialized trainings on constitutional justice for them, to ensure that they are familiar with the system and case law. Consider organizing joint colloquia with the law faculty on issues where they need most support. EVALUATION QUESTION 2 To what extent have JSSP activities been beneficial to the implementation of the EU-facilitated 2015 Justice Sector Agreement? ● Sub-question: Are there any challenges and/or opportunities that the activity can explore and address? source sensitive document 25 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV INTRODUCTION JSSP efforts in this area to prepare and assist in integrating courts in the north of Kosovo operating within the Serbian court system into the Kosovo Judiciary system only really gained traction in the last months of the second year of the project when the EU-facilitated 2015 Justice Sector Agreement (JSA, or “Brussels Agreement”) began to be implemented in practice (October 2017). Prior to that point the project undertook a number of preparatory steps, which positioned the respective actors to be ready for implementation. Accordingly, the findings for this question are described in relation to these preparatory steps, post-October 2017 efforts of the project, and current situation. PREPARATORY AND INITIAL PHASES First, the evaluation team found that JSSP served as a key facilitator during the preparatory stage. In July 2016, JSSP opened an office in North Mitrovica and assigned two permanent staff to this office. JSSP documents suggest that JSSP views itself as the only actor able to facilitate informal introductory meetings between the KJC and the representatives in the north, although some actors within the KJC tend to consider the meetings as a joint achievement. Regardless of the exact facilitation roles, in due course the communication between the judicial authorities in the north of Kosovo and the KJC intensified at the initiative of the KJC (especially the Chair of the KJC and the Director of the Secretariat). The evaluation team also found that JSSP was instrumental in facilitating integration planning. JSSP helped the KJC put together a plan for the integration process and reach early milestones. This included establishing an interim KJC committee tasked with managing the entire transition to full integration, and developing a “transition matrix” to serve as a roadmap. Information shared by informants in the north and other donors gives the impression that at that stage, JSSP was the only actor pushing the integration process. Later in the second year, JSSP combined the matrix and guiding documents addressing technical issues into a single blueprint. After being approved by the KJC, this blueprint remains the only document guiding the KJC on issues related to integration. JSSP’s assistance with implementation of the Blueprint has yielded positive results. JSSP reporting describes, and informants in the north confirm, that JSSP was quick to react and assess court facilities — including the Basic Court Mitrovica, facilities in the north and south, the Appellate Court Division, and the Basic Court Mitrovica branches of Leposavic and Zubin Potok — based on a two-tier checklist47 agreed upon by both sides. Infrastructure repairs in particular provided both substantive and symbolic value for staff working in those courts. Both Kosovo Serb and Albanian court personnel shared their appreciation for working in courts that had basic amenities.48 The evaluation team found that JSSP assistance to the court to prepare to begin their work was adequate. KIs reported on the positive results achieved with JSSP assistance, including an inventory of all pre-2008 cases in Mitrovica, Zubin Potok and Leposavic.49 Notably, the KJC Secretariat and the Mitrovica Basic Court staff jointly managed this process. JSSP further assisted the Basic Court Mitrovica 47 The checklist established basic criteria for making all courts functional. 48 Some court personnel also noted additional basic infrastructure needs for full functionality, such as a minimum number of chairs. 49 By the end of the Year 1, a total of 5,012 cases were inventoried. JSSP integration efforts have included: • Facilitating actors in the north and in Pristina to talk and plan together • Getting donors focused on needed inputs • Getting court personnel and infrastructure in place • Socializing youth and the general public about the prospect of integration source sensitive document 26 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV and its branches to operationalize their intake/registry offices, providing relevant parties with the opportunity to file new cases. The evaluation team found that a particularly important component of JSSP support was for translation of legislation and harmonization of the official versions of translated laws. As described in reports and confirmed by informants in Mitrovica, JSSP helped to resolve inconsistencies or doubts about the actual meaning of the laws.50 Judges said that JSSP’s support in this regard was substantive, given that it offered a solid basis for the judges to adjudicate cases. However, the harmonized versions of the laws have not been adopted by the Assembly of Kosovo, which makes usage of the harmonized text problematic. Despite this progress, the evaluation team found that JSSP encountered challenges in its preparatory efforts with the KJC. JSSP supported the KJC to develop an Oversight Committee51 to plan and guide integration, and although the idea of a committee was adopted, it was never operationalized. KJC informants suggest this was due to general political sensitivities among the members. Still, as discussed above, working groups did help to prepare the courts to open, and this marked the first joint exercise of the KJC representing Kosovo institutions with the parallel courts in the north. Also, the evaluation team found that some JSSP guidance in preparing draft regulations and protocols to facilitate integration of the Basic and Appellate Court Division appears to have become victim to the larger confusion over conflicts between the Brussels Agreement (JSA) and Kosovo law. JSSP assisted the KJC to develop rules and policies for the organization of the Appellate Court Division and the Basic Court in line with the Brussels Agreement. However, informants in the north criticized this approach, suggesting that this is further evidence that the KJC does not permit the Mitrovica Basic Court and the Appellate Division to ‘manage their own affairs.’ Also, the evaluation team found that JSSP efforts to strengthen human capacities are still in the early stage. JSSP has done initial training and orientation, and begun linking peer learning with other courts in Kosovo. JSSP initially planned to conduct training for judges in the north. However, since the U.S. Department of Justice’s Overseas Prosecutorial Department Assistance and Training (OPDAT) and GIZ were already assisting the KJA to develop training materials on civil and criminal matters, the focus of the JSSP shifted to training for the administrative staff of the court.52 The Training Needs Assessment report for the north identified the training priorities for the court administration staff, including case management, archiving, registry books and the use of information and communication technology (ICT) systems. The initial socialization of court administrative staff with countrywide systems appears to be on the right track. Mitrovica Basic Court staff and the manager involved in the training from Gjakova suggested that, at least in relation to caseload management efforts, the process was both useful and something that should be a model for further peer-to-peer learning. The evaluation team also found that JSSP efforts to engage youth and legal professionals starting their careers have been useful but not adequate in terms of the needs. JSSP efforts to engage with these groups in the north started before integration. Given the delays in integration, JSSP cooperated with ACDC to offer a three-month internship program for 19 young lawyers, initially in law offices in 50 Per JSSP reporting, it prioritized several laws for inclusion in this list, as follows: 1) Criminal Code; 2) Criminal Procedure Code; 3) Juvenile Justice Code; 4) Law on Contested Procedure; 5) Law on Non-Contested Procedure; 6) Law on Labor; 7) Law on Obligations; 8) Law on Enforcement; 9) Family Law; 10) Law on Inheritance; 11) Law on Property and Other Real Rights; 12) Law on the Administrative Procedure; 13) Law on Administrative Conflicts; and 14) the Law on Execution of Criminal Sanctions. 51 The committee would have comprised seven members from the Kosovo judiciary, parallel courts of the north and the international community. 52 JSSP also plans to include judges in the north in ethics training, to be organized in the second quarter of Year 3. To date, communication between the Mitrovica Court and KJC has been individual-based, not institution-based. source sensitive document 27 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Mitrovica. The program focused on providing legal training and information on the Kosovo system. JSSP staff also described how they began working with the Ministry of Education on recognition of Mitrovica￾based School of Law degrees, and how this process helped put in place a functioning committee to recognize these degrees. A group meeting with several interns from the internship program reinforced the value of having such an initiative. Interns described (finally) getting the chance to gain practical work experience and to begin to be part of the system initially with JSSP, and later for some this included being part of the additional internship efforts provided through the United Nations Mission in Kosovo (UNMIK), KJC, and Organization for Security and Cooperation in Europe (OSCE), and the stipend that these programs provide. At the same time, the interns described frustration at not having clearer paths to working in the court system. Of the four interns interviewed, none had managed yet to have their degrees recognized. They believe this is partly due to the larger issue of Kosovo and Serbia principles of degree recognition reciprocity, and it appeared the interns were not clear about how to go about the process of degree recognition. None of the four had found gainful employment in the court system, and they had the impression that there was little chance this would happen in the near future. They felt this was due to the lack of vacancies as well as the lack of transparency around hiring processes. One continued to ‘volunteer’ time in the courts while working at night at a gas station in order to earn some income. In agreeing to meet with the evaluation team, the young people demonstrated interest in being part of the system, and they reinforced this impression through their specific stories. Yet, as they described and as others involved in the internship programs also alluded to, the problems are systematic. The JSSP and other similar internship programs help, but the problem will not be resolved until other system-level issues are addressed. CURRENT SITUATION The evaluation team found that the KJC’s role in the north of Kosovo has been less visible following October 2017 integration, and that JSSP is in some (both large and small) ways playing the role that the KJC should play. At the point of integration, the KJC had visibility (according to JSSP staff). Through efforts such as ensuring that the Mitrovica Basic Court President, would not resign, the KJC demonstrated political resolve to work with the north. This impression, however, has since dissipated. KJC informants describe their commitment to the north, yet several months after integration formally was achieved, the same set of actors shared concerns that meetings and regular communication between the KJC and the north are becoming sporadic and are not formalized. Descriptions of current communication processes suggest little regular communication. In general, informants in the north spoke of their frustration that the KJC is not present or particularly effective. They expressed the need to get things done quickly and clearly, however, and because of this the JSSP appears to them as a continued necessary interlocutor. Mitrovica informants give the impression that JSSP has been the “KJC” for the north, coordinating every detail, and acting as the “neutral player” much needed until now. Although the JSSP has tried to play the role of technical advisor to the KJC so that it would take on some of these efforts,53 the effect until now has been that JSSP is the key player. 53 For example, according to JSSP reports, JSSP supported KJC efforts to recruit lay judges and certify court translators in the north. The program also organized several meetings with the Mitrovica Court president to get agreement on transfer of former parallel court cases into the new system, as well as on issues related to the inventory of cases, and other topics. JSSP also tried to arrange support for the Chief Judge of the Appellate Court Division for the North and the President of the Court of Appeals on issues such as assignment of cases, translation, etc. source sensitive document 28 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV As JSSP, KJC, and court personnel describe, information flows from the president of the Mitrovica Basic Court to the JSSP office in Mitrovica, then to the JSSP Judicial Reform Advisor and eventually to the KJC and vice versa. Informants from the court describe sharing information with the JSSP team on a daily basis and asking their advice/support on additional important issues for the functioning of the court. For example, JSSP supported the KJC and other donors to renovate the heating system for the branch court in Zubin Potok. Informants also describe JSSP’s role in working on sensitive translation issues. For this, JSSP is described as undertaking shuttle diplomacy between the courts, KJC and international community to broker support from the international community for translation services for the Mitrovica Basic Court. They also note JSSP’s continuous efforts to address the issue of case assignment protocol. The evaluation team cannot confirm the specific role KJC has had in this process, but interviews with informants in the north strongly suggest that only the Mitrovica Basic Court president deals directly with JSSP when presenting his case for resolving the language-based assignment issue. The president describes a consultation process with JSSP that he believes will “assist the KJC to find common ground with him.” In fact, leaders interviewed in the north made no effort to hide that fact that they trust “Americans” more that the KJC, partly due to the fact that the Americans have a direct communication style and link with the JSSP and because there is no KJC representative resident in the north. As noted above, issues such as language case assignment54 are technical in nature, but as interviews revealed, they have the tendency to be politically understood. Some of these issues have been caused by lack of resources (translators), but others are tied into what informants describe as the competing components and interpretations of the Brussels Agreement and its relation to Kosovo law. Specifically, judicial professionals point out that several aspects of the Brussels Agreement are not compliant with Kosovo law. For example, the Kosovo Serb President of the Mitrovica Basic Court insists that the court serves as an ethnic component of assurance that Kosovo Serb citizens will not fall under the jurisdiction of a court proceeding led by a non-Serb judge. KJC informants shared a different interpretation. Overall, the interviews suggest that this situation is making the implementation and management with the KJC integration process of even more difficult since interpretations of how to address some issues inevitably comes down to political “positioning” rather than focusing on technical solutions. Also, whether by design or default, the evaluation team found that the KJC integration process in the north is still concentrated on the functioning and operations of the basic courts more than on mainstreaming and integration the courts into countrywide reform processes. To some extent this seems reasonable given the need to begin court functions. JSSP has encouraged some cross-learning and sharing among court officials and anticipates increasing this in Year 3, but at least initially the program also worked with the courts in the north in isolation on such things as communications and outreach. This included developing communication materials such as notices, brochures, and a public service announcement (PSA). Through ACDC, JSSP also hosted numerous focus group discussions, TV and radio shows, and lectures, and facilitated meetings between judges, citizens, 54 Additional “technical” points noted to the evaluators include issues such as quality and recruitment of translators, performance reviews of Serbian judges (who have yet to start with quantitative targets), usage of the case management information system (CMIS), etc. Translation of even basic documents like court appearances can be points of contention. The team saw one example where the Albanian language version confirmed the court appearance time while the Serbian version indicated the appearance was postponed. source sensitive document 29 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV students and civil society organizations. What is not yet present is a PIO for the court who could inform the public about the work of the court and its process.55 The extent to which the JSSP has managed to focus KJC attention on these activities is unclear. According to interviews with those involved, KJC has not included the President of the Basic Court and the Chair of the Appellate Court Division in the north in wider discussions relating to judicial reforms within the KJC and other MOJ efforts on functional review of the justice system. KJC officials themselves describe integration in one way and sector-wide reform in another, suggesting that they still do not see all courts as contained within one framework. While not specifically part of its current scope of work, the evaluation team found that JSSP could have focused more attention on considering the larger situation of encouraging a more functioning law enforcement system in the north, which would assist them in their specific integration support efforts. Donor interviews suggest that there has been little meaningful coordination around improving the general rule-of-law environment in the north. While specific donor work with prosecutors and support to free legal professions is ongoing, and JSSP has had some role in this,56 informants suggest that neither JSSP as an implementer nor other implementers and donors have focused sufficient attention on a more comprehensive plan for encouraging law enforcement inter-institutional functionality. Specifically, informants described the poor state of relations between the now-integrated courts with prosecutors, the police and mayors in the north.57 The fragility of these relationships impacts the ability of the institutions to work together. It also, according to the informants, impacts the public’s support for the integration process. Informants suggest that the high proportion of people expressing concern about the state of the rule of law in the north in the United Nations Development Programme (UNDP) March 2017 survey58 has changed little, despite efforts by ACDC and others to inform and improve confidence in the system. CONCLUSIONS JSSP activities have been extremely beneficial to initial work on implementing the 2015 Justice Sector Agreement. JSSP has played the role of international interlocutor and helped put in place basic planning procedures to ensure that the many different actors would be ready when integration could more actively start after October 2017. This assistance and basic planning has helped to get the courts in the north of Kosovo to a basic level of functioning. JSSP has also brought a needed youth perspective into the judicial process and the court system. However, JSSP efforts have not (yet) resulted in integration in the sense that the relevant institutions — the KJC and the courts in the north of Kosovo — directly, routinely, and effectively cooperate. Rather, JSSP has become an indispensable interlocutor, which has resulted in dependency from both the basic courts in the north and the KJC; neither has much confidence or experience in dealing directly with the 55 A PIO was hired in May 2018 after fieldwork for this evaluation was conducted. 56 Per the SOW, JSSP was to advise the KJC and MOJ in promoting the work of free legal professions in the north. While JSSP reported that it supported the EU-funded project “Support to Free Legal Professions and the Bar Association” to organize meetings between the project and legal professionals in the north to discuss opportunities for graduates from the north to join legal professions, informants involved in the process suggest that there was little attention paid to such efforts. 57 For example, the evaluation team learned during its interviews that the president of the Basic Court Mitrovica did not receive a single phone call to congratulate him (on his appointment to the position) from the head of the prosecutor’s office, head of police or any mayors in the north. 58 The results show that 33 percent of the respondents define the situation with the rule of law in the north of Kosovo as not effective. In addition, 27.5 percent of respondents consider the north of Kosovo as “a disorganized society, defined by contradicting laws that people largely ignore” and only “12 percent of the respondents were of the opinion that the north of Kosovo is a well-functioning and well-organized society defined by the rule of law.” NDP Kosovo, The Brussels Agreement on Justice: Expectations and Concerns of the Citizens in North Kosovo, March 2017. source sensitive document 30 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV other and tend to use the JSSP as their primary point of contact. Integration has been “too important to fail,” and to date JSSP likely has prioritized this goal over the process. JSSP should continue to support the KJC to further integration and institutionalization of courts in the north. Further integration requires institutionalization, and for this the KJC needs to be more clearly engaged. Balancing support for the KJC and encouraging the KJC to take charge is challenging since KJC capacities are still low, and the issues it needs to address — from case language assignments and proper translation to additional infrastructure upgrades — continue to increase. These technical issues already have political overtones, and can easily be further politicized in ways that could derail the integration process. JSSP still has a role to play in ensuring that integration issues receive proper attention, particularly in encouraging the KJC to include the Mitrovica Basic Court, its branches and the Appellate Division in overall reform efforts. However, the period should be ending when a phone call to JSSP solves problems. No less important are the challenges (and opportunities) of how to integrate young legal professionals from the north into the judicial system, how to continue to build confidence among the general public for integration, and how to ensure that judicial and law enforcement institutions are engaged and functioning. JSSP has taken steps to involve youth through its internship programs, but this work need to be made more comprehensive to ensure that the path from practical experience to help support the recognition of degrees, passing of bar exams and employment options are more straightforward. In terms of public support for integration, public awareness helps but public confidence is likely only after the public sees real functionality in the courts. To this end, JSSP has an opportunity to assist donors and the relevant institutions to be more coordinated and focused on achieving rule of law that functions and is efficient in practice. This is not yet the case. RECOMMENDATIONS 1. Transform engagement from being interlocutor to being an assistant. The need for more sustainability in the north necessarily means that the role of the JSSP should both mature and evolve. As part of this process, JSSP should: ● Encourage the reformation of the KJC working group on integration and assist in the design of a flexible, proactive decision-making process to support the Mitrovica Basic Court and the Appellate Court Division. ● Support the KJC, Basic Court of Mitrovica and Chair of the Appellate Court Division to proactively discuss and decide on priorities, address their needs and strategize. This might include assisting with the design of a plan for administrative operations of the courts in the north that addresses/resolves the questions related to staff recruitment, procurement and infrastructural needs. This would also include coordination with other donors to support implementation. The purpose should be that all administrative gaps are addressed/resolved in a period of two years, and that the court’s clearance rate is significantly improved. 2. Emphasize mainstreaming of judicial institutions in the north of Kosovo into countrywide reform processes. This should include efforts to: ● Encourage KJC and MOJ planning that ensures the north is part of the KJC countrywide reform processes such as on court reporting, accountability, etc. This includes bringing the judges from the north of Kosovo into discussions and getting them involved in the processes, especially with regards to drafting of regulations, MCLE, decentralization and Secretariat reform. This might also include assisting the KJC and representatives in the source sensitive document 31 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV north to design a concrete plan that offers a transitory clause/policy for case targets for Kosovo Serb judges in the north, and design of a tailored performance evaluation review that fits that purpose and gradually includes them in the regular performance evaluation process, along with other courts in Kosovo. ● Continue to support the Mitrovica Basic Court in applying court accountability tools, such as publication of decisions, maintaining court performance statistics, posting news of landmark cases on the court’s website and more engaged cooperation with the KJC Secretariat. ● Assist the KJC and the Mitrovica Basic Court, its branch courts and the Appellate Court Division in the north to improve and strengthen different aspects of coordination with central institutions, with an emphasis on the MOJ, KJA, etc. 3. Design an expanded and comprehensive package for (youth) legal internships. Youth internship programs have helped young people become familiar with the judicial system. The current strategy to continue the programs in the courts should include formalizing the programs and making them more comprehensive. As part of this process, JSSP should: ● Take the lead on strategizing and designing efforts to create a full-cycle training program (at least for a transition period, and regardless of the general policy tools that KJC has for other parts of Kosovo) that covers internship opportunities and how to finance them, as well as how interns can prepare for the bar exam and for employment opportunities in the legal profession. This could include specific cooperation with other actors who are already undertaking or planning parts of training and support in these areas. It could also include greater involvement of the Kosovo Bar Association and the chambers of Notaries and Private Enforcement Agents for their roles in the training related to the bar exam and access to free legal professions. ● Support the Mitrovica Basic Court to develop communication tools that inform youth about career opportunities in the legal/judicial systems, and provide information about the recognition of degrees, bar exams, and scholarship and professional development programs. 4. Improve linking of donor efforts to other justice sector institutions and programs for a more comprehensive approach. JSSP should consider taking the lead on working with relevant donor programs to strategize how to make policy to encourage rule of law in the north more complementary and cohesive. This could start with working with the Justice Sector Functional Review Working Group and with various donor projects to address gaps and increase public awareness of how citizens can engage with and use the justice system. EVALUATION QUESTION 3 How successful is JSSP in managing its civil society subcontractors to achieve the activity objectives? ● Sub-question: Were there any other alternative approaches that could have been taken to maximize subcontractor performance? INTRODUCTION JSSP issued a request for proposal (RFP) in the second quarter of Year 2 to CSOs interested in working with JSSP on assisting the courts to increase judicial transparency and accountability, with a specific focus source sensitive document 32 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV on court outreach activities and ways to enhance public trust in the courts. The three CSOs selected were the Balkan Investigative Reporting Network (BIRN) to provide technical assistance to courts in improving their outreach [as well as facilitating engagement between the public and court]; Democracy Plus to conduct a court user satisfaction survey; and the Advocacy Center for Democratic Culture (ACDC) to further facilitate information sharing and public awareness around integration.59 FINDINGS The evaluation team found that JSSP has applied good partnership processes in its management of civil society partners. The program engaged formally with CSOs via two different tracks during the first years of the project.60 During year one, JSSP subcontracted with ACDC to increase public awareness around integration issues in the north of Kosovo through town hall meetings. ACDC also held training sessions with 19 law students in the north of Kosovo to provide information on the Kosovo judicial system. Despite the fixed-price contract mechanism used by JSSP, the CSOs described their partnership with JSSP as consultative and iterative, with the flexibility to shift and/or develop efforts as the situation required. As one partner explained, “I like that they see themselves as partners both with institutions and with us, and I think that this is how JSSP is seen by others in the system, and I think 18 years after the war this is the way they should do it.” At the same time, in reviewing the SOWs for CSOs and in speaking with CSO representatives, the evaluation team found that the contract mechanism used by JSSP was more appropriate for some activities than others. Namely, the fixed-priced contract mechanism was more suited to Democracy Plus’ survey work than for some of the more process-focused work undertaken by BIRN and ACDC (e.g., facilitation, awareness raising, advocacy and confidence building efforts). The Democracy Plus survey and focus group activities allow for a clear set of outputs linked with results. Using the same mechanism with BIRN and ACDC for their efforts created less clear and realistic linkages between outputs and results given the scopes of the initiatives. For example, the efforts in the BIRN SOW anticipated as one result “a citizenry well informed about the KJC and the target courts’ operating activities and competencies” and another result was the “effective cooperation between existing judges, future lawyers, prosecutors and judges through open discussions.” The ACDC SOW identifies ambitious results such as to “increase public confidence in the judiciary in the north” and to “improve the judiciary’s transparency in the north.” Other wording in the SOW suggests that the CSOs identify ways to measure such results for the purposes of the contracts. For example, transparency is linked to better cooperation between the institutions and CSOs, and this was to be formalized by signing MOUs with them.61 What is not clear from the SOWs or interviews with CSO informants is the extent to which such measurements provide meaningful information and guidance for future CSOs and JSSP efforts. For example, ACDC staff stressed that public awareness is improving, while acknowledging that it is a process. But for the purposes of the contract, public awareness is measured as an activity not as a process, and is it not clear that either ACDC or JSSP has developed meaningful monitoring and evaluation techniques to capture steps in such processes. The evaluation team also found that in some cases the scopes of engagement for the CSO subcontractors included tasks that JSSP or others could or were already doing. For example, BIRN’s provision of technical assistance to PIOs and the courts is not markedly different from other JSSP technical assistance efforts undertaken directly by staff or short-term consultants engaged with different 59 Please see the specific CSOs’ SOWs for more details. 60 JSSP also cooperated with CSOs through its public integrity initiative by including them in the roundtable discussions with judges, prosecutors and attorneys that took place around Kosovo. See JSSP Annual Report for Year 1 for more details. 61 The evaluation team did not see CSO reports to confirm how the CSOs reported on these results (and some are still in process). source sensitive document 33 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV court institutions. The decision to work with a CSO on this was described by JSSP as partly due to BIRN’s expertise and partly due to the hope that this interaction would encourage/strengthen CSO￾judicial interactions and transparency. Yet BIRN’s specific technical support role under JSSP is distinct from its other reporting and watchdog activities, which raises the question of whether this approach did indeed have this secondary effect of strengthening CSO-judicial relations. The evaluation team learned that BIRN itself sees its role vis-à-vis the courts in multiple ways, and while it continues its monitoring efforts, BIRN did not specifically link its technical assistance funded through JSSP with enhancing its own organizational engagement with the courts. The evaluation team found that the role of ACDC in the north (in a different way) mimics the work JSSP has been doing in-house with the JSSP Judicial Reform Advisor in Pristina and the JSSP staff in Mitrovica. The impression from speaking with ACDC staff working closely with the JSSP office in the north is that this ACDC team acts in close coordination with JSSP and almost as “an arm of JSSP.” As ACDC staff said, “If anyone wants to do something in the north, they come to us.” Statements from Mitrovica Basic Court and ACDC staff describe very close collaboration, with sharing of information and together “getting things done,” often through a combination of JSSP and ACDC efforts which blurs distinctions between the roles of JSSP and the CSO. The evaluation team also found that the activities JSSP envisioned for the CSOs duplicated or overlapped with other donor efforts. BIRN meetings between students and judges are similar to initiatives that staff of the Kosovo Bar Association (KBA) described to the evaluation team. And perhaps more clearly, the evaluation team found that the court satisfaction survey undertaken by Democracy Plus is very similar, and was undertaken without initial coordination with a Council of Europe (COE) survey.62 While both Democracy Plus and the COE are now planning to cooperate on sharing findings, both also acknowledged that it would have been good to know about each other’s efforts beforehand. The evaluation team also found that the JSSP management of CSOs did not encourage collaboration or integrative approaches. The terms of the CSO contracts meant that there was little overlap between the three partners and little need or opportunity for them to collaborate, even if some work was similar. For example, ACDC’s work with students and judges was somewhat similar to work BIRN was doing, but they did not share experiences or lessons learned. The CSO respondents describe JSSP calling them together for one meeting, but primarily so to present on their projects, not necessarily so they could talk with each other about their projects in more detail. The delineation of duties also reinforced the CSOs’ divided work between the “north and rest of the country.” ACDC undertook various facilitation and public awareness efforts as part of their JSSP cooperation, but only focused on the north. BIRN and Democracy Plus implemented their efforts everywhere else. ACDC staff described communicating with other CSOs to see if they were interested in the situation in the north, but not to discuss other ongoing justice sector-related efforts around the country. Similarly, BIRN and Democracy Plus described their efforts with specific courts around Kosovo, but not with courts in the north. Finally, the evaluation found that while the CSOs conducted activities that contributed to the JSSP goals, they did not carry out the monitoring and analysis envisioned in JSSP’s SOW that might have furthered program accountability objectives. In JSSP’s initial SOW of work, anticipated cooperation with CSOs was to be focused on monitoring and watchdog activities. Later this focus was changed due to what JSSP described as sufficient donor support for CSO watchdog activities.63 JSSP informants confirmed this 62 The COE survey was initiated in December 2017. JSSP’s survey was contracted for in August 2017, and had a launch event held in September 2017. 63 See JSSP Year 1 Annual Report, Section 2.2. source sensitive document 34 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV view, even while noting that the program also has engaged and cooperated with CSOs through efforts like the Integrity Initiative, which provided a venue and forum for CSOs monitoring justice sector efforts. JSSP staff expressed concern that if it supported monitoring and watchdog activities the program might receive less cooperation from the relevant institutions. At the same time, the evaluation team observed that ACDC monitors court efforts in the north yet continues to retain access and cooperation from the court institutions there. This might be due to specific dynamics in the north, but it suggests that CSO monitoring efforts do not necessarily preclude other types of cooperation. The evaluation team confirmed that there are a number of ongoing monitoring activities conducted by different CSOs including the Kosovo Law Institute (KLI), BIRN, and OSCE, among others. These activities target different aspects of the courts and their functioning, but only sometimes cover issues that JSSP is focusing on. For example, the evaluation team found little evidence of active monitoring on appointments of court presidents, KJC disciplinary proceedings, the decentralization process, etc. The team also noted little data on or monitoring of the actual integration of the courts in the north. Some of the informants specifically engaged in monitoring emphasized the need for more focused monitoring. For example, KLI described its monitoring of some KJC efforts, but also acknowledged that it would be useful to track areas like performance evaluation of judges more closely. Other CSO respondents and external experts pointed out the link between public confidence in the courts and high￾level corruption cases, suggesting that the key to public confidence is not only better court transparency (as addressed by JSSP), but also more monitoring of these high-level cases. A final point made by external experts is the inherent difficulty of using monitoring and advocacy to push for reforms. According to both donors and external experts, there is no magic formula for being more effective in advocacy. However, they suggest that encouraging CSO coalition building and collaboration around key advocacy efforts would provide a better chance of success. CONCLUSIONS JSSP has been successful in managing civil society subcontractors to achieve the specific activity objectives. Activities focused on providing technical training on communications, conducting court satisfaction surveys, and providing community awareness and facilitation services are all completed or will soon end, according to the SOWs. However, there are some concerns: the fixed-price contract mechanism is more suited for deliverables related to research such as the survey and is less suited for capturing the process-focused work of facilitating and capacity building done in a short period of time. Duplication of efforts with the court surveys and with some facilitation efforts suggest that JSSP could have been more selective in its design of activities and that the subcontractors could have been more aware of related efforts. And the little attention paid to shared learning or integrating work among the three CSOs reinforced territorial (north vs. the rest) competencies. Slight changes to the management and mechanisms could address these concerns, but the larger concern raised related to the CSO management process is what was missing and how it was conceptualized. While CSO activities contributed to the overall JSSP goals, the intended original focus on watchdog and monitoring work, which was later changed, remains a need. There are gaps where focused monitoring efforts on specific areas of KJC and court performance could more clearly complement JSSP’s work, and it this alternative approach that should be explored. source sensitive document 35 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV RECOMMENDATIONS 1. Continue to support CSOs, but shift some of their focus to watchdog efforts that could complement the project and its advocacy efforts. Consider key areas of KJC and court functioning that are not covered by other watchdog efforts and where JSSP would like to see more scrutiny. As part of this support, consider working with a variety of CSOs that are active in the north and in the rest of Kosovo to undertake specific monitoring and advocacy work, and to underpin the integration efforts encouraged by JSSP. 2. Consider using a subgrant mechanism with focused outcomes and accompanying monitoring processes. Irrespective of the contracting mechanism or focus, develop meaningful monitoring, evaluation and learning (MEL) indicators that will be able to capture short-term effects and provide data for ongoing JSSP strategic and tactical program planning. EVALUATION QUESTION 4 How effective and efficient is JSSP in coordinating with USAID and other donors’ programs? FINDINGS In general, JSSP’s cooperation with other donor-supported programs on strengthening the KJC has been good (if informal), though some efforts have been duplicative due to a lack of formal planning. Multiple donors64 have been focused on strengthening the KJC, and JSSP is one of several donor-supported programs with experts embedded in the KJC. All sides describe their coordination as generally good on a day-to-day basis. For example, the European Rule of Law Mission in Kosovo (EULEX) and JSSP will provide comments to the KJC on a regulation, or staff from both programs will sit in on a committee meeting to provide advice on the process of developing MCLE standards (for example) and preferred content. In addition, one donor described efforts by the “principal” donors and implementers working with the KJC to have a monthly lunch or coffee chat to discuss current and upcoming issues. These discussions, according to this donor, tend to focus on both project and policy issues. This has allowed a mostly joint approach among programs to different policy issues, but differences in European and U.S. approaches have occurred; for example, JSSP reports note how full implementation of the 2017 KJC regulatory plan was delayed due to multiple revisions to the regulation on court president appointments and multiple inputs from the international community. In terms of duplication of efforts, part of the challenge appears to be related to planning cycles and changes in personnel for the respective programs; the EU Twinning project, for example, has a plan that extends through May 2019, but it is not clear whether they developed this plan based on consultation with other principal donor-supported programs. While JSSP describes incorporating suggestions and recommendations from the Twinning team into the policy and legal drafting work at the KJC, syncing the actual activities of the two programs has been more difficult. For example, staff from both JSSP and the Twinning program describe plans for similar ethics training for judges and have had similar training and manual development initiatives in the past. Similarly, it is not clear the extent to which JSSP consulted with other donor-supported programs when developing its Year 3 activity plans for the KJC. Interviews with JSSP staff suggest that they have an awareness of other program plans, but it is unclear if there were formal consultation efforts. 64 The “principals’ in this cooperation are identified as JSSP, EU (EU Office, EUSR, EULEX, EU Twinning), UNDP, the U.S. Embassy (including USAID) and the British Embassy. source sensitive document 36 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV A factor in such duplication is the KJC’s low capacity to coordinate its work with various donor￾supported programs and to be discerning in its acceptance of donor assistance. While JSSP reporting suggests that the KJC established formal donor coordination meetings in June 2017, later reports provide little information on these meetings, and it was not clear how useful these meetings might have been. KIs described a recent example in which the KJC requested COE assistance for designing graphics for court reporting templates prepared by JSSP (presumably a situation where JSSP could also have helped with the graphics). This suggests that coordination is still more ad hoc than formal. In addition, KJC officials describe accepting most donor efforts and programs as welcome, and specifically highlighted their wish lists to the evaluation team. While they also identified which of the donor program inputs appeared most useful – suggesting that the Twinning program structures were less useful – they were quick to say that donor support in general is still needed, giving the impression that they are not ready to turn away donor funding even if it is duplicative. The evaluation team found that JSSP cooperation with other USAID programs has been good, and programs have found ways to resolve overlap and be complementary. JSSP started after two other rule of law programs, the Property Rights Program (PRP)65 and Contract Law Enforcement (CLE) Program, had already been in place for several years. Because case backlog reduction and prevention is a major component of all three projects, the PRP and CLE programs shared best practices and lessons learned on topics such as caseflow management and use of BRLOs. However, there were also inevitable overlaps. Specifically, PRP is tasked with working on property cases, which are a subset of civil cases. Therefore, when PRP staff were reviewing case files, it did not seem efficient to put non-property cases aside, which resulted in PRP working on some cases that would be in the JSSP portfolio.66 According to KIs involved in these programs, to avoid overlap, JSSP works on criminal cases in courts where PRP also works. The evaluation team also learned about other points of coordination. For example, JSSP is including training modules developed by CLE and PRP for its management-training program being developed for the KJC.67 JSSP also cooperated with CLE to make sure that a mediation clerk would be appointed in each Basic Court. The evaluation team also notes that cooperation among the programs appears to be developing further. For example, JSSP describes in its Year 3 first quarter report plans to work with PRP on a practical guide for property rights cases and a Civil Bench Book. It is also notable that JSSP narrative reports as of Year 3 began to devote a specific section to coordination with other USAID activities, which suggests a heightened awareness (if still not institutionalized practice) of this coordination. The results of JSSP coordination with non-USAID donor-funded programs sector-wide has been mixed. Per JSSP reporting and donor interviews, the evaluation team delineated two general ways that JSSP has cooperated with other donor-supported programs: joint project funding/planning coordination and policy advocacy coordination. Each of these areas of cooperation has both positive and negative examples. JSSP narrative reports provide a running commentary on efforts, which sometimes worked (such as the eventual united policy approach to independence of the ODC and where JSSP and EULEX encouraged the KJC to include Kosovo Serb translators in certification and licensing procedures). But in areas with many donor inputs, cooperation has been less stellar. One case where cooperation did not happen, as noted under EQ 3, was with the COE on court user satisfaction surveys. The fact that the COE and JSSP are housed in the same building and regularly talk with each other (according to both 65 JSSP started Nov. 2015, PRP May 2014 and CLE May 2013 66 For example, per JSSP reports, at least one set of training materials related to court administration curriculum completed during EROL (2011-2015) could not be found early in Year 2, and as a result the program delayed its plans to use this curriculum to develop a court executive curriculum. (JSSP Year 2 Quarter 1 Report). 67 JSSP Year 3 Quarter 1 Report, p. 10. source sensitive document 37 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV COE and JSSP staff) just reinforces how easy it is to duplicate efforts. Similarly, donor interviews reveal that where there have been different approaches to issues by different donors — such as with the definition of disposition of cases based on a U.S. or a EU understanding — the process of coordinating programming has been more difficult. However, within these general trends, the evaluation team found that JSSP coordination in the north (as also noted under EQ 2) stands out as being primarily positive to date. Examples include JSSP-facilitated coordination among donors in the north for refurbishment of court facilities as well as JSSP stepping aside from its plans for training of judges after seeing that GIZ and others were engaged on this issue already. These examples suggest that JSSP has been aware and proactive in its coordination efforts. Furthermore, meetings with other donors engaged in rule of law efforts highlight the coordination role that JSSP has played to date. At the same time, current JSSP efforts to cooperate with other donor-assisted programs in the north appear less proactive. Through interviews with different donors, the evaluation team found that key European donors are less aware of JSSP plans for the north than they were previously. Specifically, the interviews provided the impression that JSSP and these donor programs do not have a shared understanding of key priorities for translators, translation needs, and further facilities refurbishment, nor is there shared understanding on the general policy emphasis that needs to be put forward by the donor community. The evaluation team could not confirm whether or not this is due to how JSSP is sharing information (directly or through the KJC). Feedback during donor meetings and from those in the north receiving donor assistance indicates that what donor assistance and policy stances were previously more planned has become more ad hoc. KIs describe countless hours taken by donor visits, without many specific results. CONCLUSIONS JSSP has been somewhat effective and efficient in its efforts to coordinate its work with other donor￾assisted programming in the justice sector, with the KJC a focal point of this coordination. Yet even with the KJC, the coordination has tended to be informal rather than structured. A similar dynamic has been at play with other USAID programs; where there was overlap and duplication, JSSP and the others eventually found ways to mostly ensure complementarity. Sector-wide coordination is difficult, and JSSP has displayed day-to-day reactive cooperation more than forward-looking, proactive cooperation. In some situations, JSSP has been able to coordinate quite well, such as with early efforts in the north. Current coordination in the north, however, is less apparent even as the need is great if not greater than before. These needs are technical, but as noted in earlier sections, they are politically sensitive and therefore strategically important to get right from a donor financing and coordination perspective. As the centerpiece USAID ROL program, JSSP is playing the role of getting the necessary donors and implementers around the table, but this role could be expanded and more focused, particularly in assisting the KJC to effectively carry out this coordination role. RECOMMENDATIONS 1. Focus more clearly on coordination with other donor implemented projects in providing assistance to the KJC. This includes sharing work plans and strategies to coordinate and avoid overlap. It also includes considering which other projects might be able to take over specific tasks, or more closely cooperating with other donors on some planned JSSP inputs (i.e., some trainings offered by the Twinning Project). source sensitive document 38 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV 2. Emphasize donor coordination and prioritization planning with the KJC. As part of efforts to work with the KJC on prioritized areas for 2018 and 2019, JSSP should put further emphasis on building up KJC capacities and mechanisms to identify needs, deliberate about donor offerings, and organize practical ways to coordinate this. 3. (Re) Elevate JSSP’s role in donor coordination in the north temporarily while ensuring that this role can be transitioned to the KJC. JSSP should put additional effort into focusing and facilitating donor projects around key input needs identified in the north. JSSP should ensure that KJC personnel are engaged and part of this process to ensure that they can eventually take over this coordination role. 4. For USAID: Review how current and planned justice programming overlaps with and/or complements JSSP efforts. With an eye to planning the next CDCS, USAID should conduct an exercise to identify perceived needs, the programming vehicles for these that are already present and/or planned, and how to create mechanisms for coordination between the different generations of USAID rule of law programming and concurrent rule of law programming. EVALUATION QUESTION 5 How effective has JSSP been in conducting activities to promote equal opportunities for women in the justice sector? ● Sub-question: To what extent has JSSP been able to promote women as change agents in the judiciary? INTRODUCTION The evaluation team, together with JSSP staff, drafted a Theory of Change (TOC)68 that would represent JSSP’s explicit approach to gender empowerment in the justice sector. This included inputs that provide procedural guidance to the KJC and courts, technical assistance on KJC decisions related to member appointments, and JSSP programming to provide training and professional development opportunities. Based on these inputs, JSSP’s specific contributions to gender empowerment suggest several findings, as summarized below. FINDINGS JSSP legal advice on selection of new KJC members has provided some basis for improving gender equality in the council member selection processes. Per JSSP reporting, JSSP has worked closely with the KJC, and specifically with the Normative Committee, to interpret existing procedures for filling KJC vacancies.69 During a series of meetings (which also included, at a minimum, representatives from EULEX and the EU Twinning project), JSSP provided guidance to the KJC on procedures for filling its positions in a way that would achieve gender proportionality across the courts. Based on these efforts, the KJC first decided to ensure that one of the two vacant positions would be filled by a woman, resulting in the promotion of a woman from the Supreme Court. Secondly, JSSP staff provided advice related to Constitutional Amendment No. 25 which increased the number of KJC members elected by 68 If the program can provide specific intervention guidance to the KJC and other institutions for gender equality plans and procedures; ensure % of women participation/inclusion/mentoring in its technical assistance and training programming; and actively ensure that gender issues receive attention across its different activities areas and through specific events, then attention to the role of women in the judiciary will be heightened AND opportunities for women and more equal gender representation will begin to shift to reflect this. 69 Per JSSP reports, this is the Law No. 05/L-033 on Amending and Supplementing the Law No. 03/L-223 on the Kosovo Judicial Council. source sensitive document 39 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV the judiciary from five to seven and provided a basis for greater gender equality in the judiciary, from the Basic Courts in the KJC. JSSP reporting at the end of Year 2 also suggests that its advice contributed to, not directly linked to, the fact that two women were elected to leadership positions – one as president of the Basic Court of Pristina and the other as the Supervisory Judge of the Kacanik branch of the Basic Court of Ferizaj.70 KJC leadership and JSSP staff highlighted these achievements in their meetings with the evaluation team. The evaluation team found less evidence of success with JSSP efforts to infuse gender empowerment practices into broader KJC hiring and selection processes. JSSP assisted the KJC to develop a Gender Equality and Empowerment Plan in Year 1, and this provided the KJC with a set of procedural standards on gender to use in their management decisions. However, there is little clear evidence that these standards have been taken up and institutionalized. JSSP reports after Year 1 provide little specific mention of this plan, or examples of the KJC applying this plan in their work. Rather, the JSSP SOW revisions in Year 2 that removed the requirement for JSSP to work with the KJC on gender-based budgeting suggest little traction within the Council for such a gender focus in general. The revision states as much, suggesting that there is “little institutional support for gender-based initiatives.”71 While JSSP has also recommended specific provisions in draft laws on the KJC and the Courts that would require the KJC to have equal male and female representation in management and leadership positions, the draft laws are still under review. The evaluation team found that JSSP’s decision to reframe efforts around youth entry opportunities provided the chance for a number of young women to get practical experience in the justice system. As per JSSP reporting and interviews, the JSSP plans to work with the Forum for Women Judges and Prosecutors to establish a court internship and mentoring program were changed during Year 1 after the group expressed little interest to work with JSSP on the initiative.72 JSSP’s decision to reframe efforts around youth entry opportunities provided the chance for a number of young women to get practical experience in the justice system. This included four interns in the north of Kosovo through cooperation with the ACDC; 10 hired as BRLOs; and four working in the Basic Courts. However, the evaluation team found that such internships have not yet resulted in many follow-on sector-specific work opportunities. Interviews with a set of the interns and BRLOs at the Pristina Basic Court generally suggested that they were pleased with the internship opportunities, felt that they had developed new skills through their work, and were interested in continuing to work in the justice sector. They were very happy for the opportunity to work directly with judges and appreciated training and mentoring provided by JSSP staff. Interns understood that the JSSP support would eventually end, and they looked forward to working with the courts in a similar capacity. However, several who had already applying for positions noted that due to the current hiring process, the JSSP experiences were of little consequence. This situation is not the case only for BRLOs or women, but it is telling that of the 10 women hired as BRLOs from a total of 24, only two (according to JSSP data) have been hired by the courts.73 As described earlier under EQ 1 on case backlog dynamics, efforts to encourage hiring of these types of individuals are connected to amending the Law on State Administration. The evaluation team also found that JSSP training (and mentoring) programs that included a sizeable share of women likely improved some of the skills of these participants. JSSP trainings, from those for BRLOs and PIOs to those focused on court administration and budgeting, included women, and through 70 See JSSP Year 2 PMP Outcome and Indicator Chart, Expected Result 14. Also see JSSP Year 2 success story highlighting the appointment of Aferdita Bytyci as President of the Basic Court of Pristina. 71 Memorandum: Proposed Modifications to Contract, No. AID-OAA-I-13-00029/AID-167-TO-16-00001, point 3, p.2 72 While the Forum was formed in 2014 with the purpose of promoting women in the justice sector, their priority appears to have been focused on women in leadership positions rather than mentoring efforts. 73 This includes one as a legal officer in the basic courts and one as a professional associate at the court of appeals. source sensitive document 40 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV these trainings many women appear to have enhanced their skills. According to JSSP cumulative data at the end of Year 2, 264 women participated in one or more of its 40 training programs.74 JSSP training evaluations and surveys suggest that the trainings were of high utility and the lessons learned by participants were being applied.75 In the case of interns who had taken part in trainings, they specifically corroborated the report findings. However, for other participants it was not clear to what extent the training had enhanced their skills and day-to-day performance. As noted in the section on EQ 1, special mention should also be made of the JSSP efforts within the KJC to work closely with legal officers on enhancing their skills to be support staff on policy development for the committees. JSSP staff interviews suggest that these efforts have resulted in a cadre of women at the technical level that can draft policy memos and briefings for committee deliberation, even if their confidence is still low. The evaluation team’s group meeting with four of these legal officers supported this assessment. The group had significant enough experience – some with over 10 years of work experience at the KJC – and all demonstrated knowledge of their respective committee focus areas. Still, none of the four presented themselves as able (or willing) to do the policy memo work without JSSP assistance (at least now). Finally, the evaluation team found that JSSPs shift to integrating gender considerations into trainings and general programming has partially missed an opportunity to link gender and merit-based issues. Given the low interest in gender issues displayed during Year 1 by counterparts, JSSP staff described its efforts to mainstream gender issues into its programming, with an emphasis on ensuring women participation in skills training and enhancement.76 However, JSSP has not clearly linked such training, which deliberately includes women at all levels, with an emphasis on merit-based hiring and advancement. This, the evaluation team found, would have resonance with women in the judiciary. Interviews with women at all levels of the justice sector suggest an interest in improving gender equality, but indirectly; most express concern at the lack of women in leadership positions, but they have a different framing of this challenge. Namely, women – whether judges, legal officers or interns – speak of performance and merit, and of their frustration that current hiring, promotion and evaluation procedures lack an emphasis on merit criteria. For example, two high￾level women judges interviewed explained that it is much easier to support young female colleagues based on their credentials rather than on the fact that they are women. Young professionals expressed similar sentiments, asking for a “chance to prove themselves” rather than special treatment. CONCLUSIONS JSSP has been somewhat effective in its activities to promote equal opportunities for women in the justice sector. It has put gender on the agenda in the KJC (again), and has created procedural guidelines for gender empowerment as well as for gender to be used as a criterion in the selection of new members. Beyond this top-level “setting the stage,” further improvements in gender empowerment require will within the KJC to institutionalize gender considerations. 74 Per Year 3 Q1 JSSP reporting, an additional 21 women court staff were trained primarily in Mitrovica and branch courts during Year 3 Quarter 1, but PMP data is not updated to reflect these trainings. 75 See JSSP Year 2 PMP Outcome and Indicator Chart. Expected Result 11. 76 A modification in Year 2 to the JSSP SOW reflects this change. It eliminates specific leadership training for women judges and court personal to focus instead on general management training. source sensitive document 41 | JSSP MIDTERM PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Similarly, JSSP has indirectly supported gender as a consideration for hiring in the courts. By encouraging adherence to the laws, JSSP has been a supporter of getting more women into leadership positions and has noted some successes during the program. At the other end, JSSP efforts to create more skilled young women – either through participation in training and/or through offering various intern program experiences – have laid the groundwork for providing entrance into the sector, but these need attention. In general, the training and internship experiences appear to have been useful and the skills and knowledge gained are being applied. However, JSSP’s efforts to promote women as change agents have had little traction. Program adjustments to focus efforts on youth and mainstreaming training and skill building for women appear to be correct and practical. But they have also not used framing that women might stand behind in clear numbers, which is one that focuses on merit in recruitment and advancement. This might be a way to galvanize women to support other women in the system and a way for JSSP’s top-down and bottom-up efforts to be mutually reinforcing. RECOMMENDATIONS ● (Re-) Focus KJC attention on using a gender empowerment strategy and track key implementation benchmarks of this when monitoring KJC performance. Consider increasing the interest of KJC members in monitoring of its gender policy efforts through: ensuring key gender considerations in the finalization of the 2018 work plan; raising an awareness of these plans among KJC staff; and cooperating with CSOs to monitor specific aspects of gender inclusion/equality in relation to the plan and hiring efforts. ● Continue to use opportunities to encourage application of gender equality laws and policies and to champion the hiring/appointment of women in leadership positions, but publicly and in coalition-building efforts stress merit-based hiring (because this has a greater resonance with everyone). JSSP should continue to encourage implementation of key policies and regulations and provide advice on gender equality provisions to the specific institutions. At the same time, presentation of these issues, whether as success stories and or in collaboration with organizations like the Women’s Forum for Judges and Prosecutors, should focus on the merit-based qualifications that specific candidates bring to the positions. ● Continue programming focused on supporting young people’s entry into justice sector professions. Reinforce this by more closely tracking what interns learn and utilize from trainings in their practical work. This would provide a clearer matrix of capacity gains for the project. Also, consider tracking if and when these young people attempt to apply for positions in the justice system, and consider linking this to monitoring of merit-based hiring in the courts.