EVALUATION Mid-Term Performance Evaluation of the Implementation of the Effective Rule of Law (EROL) Program and the Kosovo Legal Profession (KLP) Program [April 2014] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates. USAID/KOSOVO MID-TERM PERFORMANCE EVALUATION OF THE IMPLEMENTATION OF THE EFFECTIVE RULE OF LAW (EROL) PROGRAM AND THE KOSOVO LEGAL PROFESSION (KLP) PROGRAM Final Report April 2014 Prepared under RFTOP No. SOL-167-14-000005 Submitted to: USAID/Kosovo Submitted by: Mary Noel Pepys, Team Leader Roland Gjoni Nancy Sharp Nti Asare Vigan Qorrolli Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS EXECUTIVE SUMMARY ................................................................................................................ 1 Evaluation Purpose .............................................................................................................................................................. 1 Project Background ............................................................................................................................................................. 1 Evaluation Design, Methods and Limitations ................................................................................................................. 1 EROL Findings, Conclusions and Recommendations .................................................................................................. 2 KLP Findings, Conclusions, and Recommendations .................................................................................................... 6 General Recommendations .............................................................................................................................................. 8 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS .................................................. 1 1.1 Evaluation Purpose ............................................................................................................................................... 1 1.2 Evaluation Questions ........................................................................................................................................... 1 2.0 PROJECT BACKGROUND ................................................................................................... 1 2.1 Effective Rule of Law (EROL) Program ........................................................................................................... 2 2.2 Kosovo Legal Profession (KLP) Program ........................................................................................................ 3 3.0 EVALUATION METHODS & LIMITATIONS ...................................................................... 3 3.1 Evaluation Methodology ........................................................................................................... 3 3.1.1 Qualitative Research and Analysis ............................................................................................... 3 3.1.2 Quantitative Data Collection ....................................................................................................... 4 3.2 Evaluation Limitations ............................................................................................................. 4 4.0 EROL OVERALL FINDINGS AND CONCLUSIONS ........................................................... 5 4.1 Question 1: How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of EROL’s and KLP’s program assistance components? .............................. 5 4.2 Question 2: How have the Justice Institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of EROL’s and KLP’s activities? ........................................ 15 4.3 Question 3: What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on KJI, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar)? The impact of the programs on secondary legislation will also be important. ......................... 16 4.4 Question 4: What results have EROL’s and KLP’s partner institutions achieved through capacity building activities of the subject programs? .................................................................................. 19 4.5 Question 5: How effective and efficient are EROL’s and KLP’s activities in coordinating with other USAID and other donors’ programs? ................................................................................................ 22 4.6 Question 6: Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? ...................... 23 4.7 Question 7: What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs? ............................................................................................................. 23 5.0 EROL SPECIFIC FINDINGS AND CONCLUSIONS .......................................................... 24 5.1 Question 1: What is the level of satisfaction of EROL’s court users’ survey compared vs. Public Pulse survey supported by USAID? ................................................................................................... 24 5.2 Question 2: Is the trainees’ knowledge retention level of the EROL trainings supported at the KJI measured and what are the results of it? ........................................................................................ 24 5.3 Question 3: To what extent is the court case data base system populated and used for court administration throughout Kosovo? .............................................................................................................. 25 6.0 EROL RECOMMENDATIONS............................................................................................ 25 7.0 KLP OVERALL FINDINGS AND CONCLUSIONS ............................................................ 26 7.1 Question 1: How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of KLP’s program assistance components? ................................................... 26 7.2 Question 2: How have the Justice Institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of KLP’s activities? ................................................. 28 7.3 Question 3: What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on Kosovo Judicial Council, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar. The impact of the programs on secondary legislation will also be important. ............................................................................................................................................................. 30 7.4 Question 4: What results have KLP’s partner institutions achieved through capacity building activities of the subject programs? ................................................................................................................. 31 7.5 Question 5: How effective and efficient are KLP’s activities in coordinating with other USAID and other donors’ programs? .......................................................................................................................... 31 7.6 Question 6: Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? .......................................... 32 7.7 Question 7: What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs? ............................................................................................................. 33 8.0 KLP SPECIFIC QUESTIONS .............................................................................................. 33 8.1 Question 1: To what extent is the management, quality and availability of the legal clinics at the Law Faculty of the University of Pristina and the Iliria College better as a result of KLP assistance? ............................................................................................................................................................. 33 8.2 Question 2: How has the academic research capacity at the Law Faculty of the University of Pristina improved as a result of the KLP assistance? ................................................................................. 34 8.3 Question 3: What have been the effects of KLP support for Iliria College? ........................................ 34 8.4 Question 4: How has KLP contributed to improving the functioning of the Mandatory Continuing Legal Education and the disciplinary system at the Kosovo Chamber of Advocates? . 35 8.5 Question 5: How has gender and ethnic diversity improved within the Kosovo Chamber of Advocates membership as a result of the KLP assistance? ....................................................................... 36 9.0 KLP RECOMMENDATIONS .............................................................................................. 36 10.0 GENERAL RECOMMENDATIONS ..................................................................................... 37 LIST OF FIGURES Figure 1: EROL Trainings and Courses Taken by Focus Group Discussion Participants Figure 2: Mini-Survey Respondents Applying Skills Learned through EROL Trainings in Daily Jobs Figure 3: Increase in Law Student Legal Clinic Enrollment and Law Students Obtaining Internships ANNEXES Annex A: Statement of Work Annex B: Evaluation Design Matrix Annex C: Schedule of Meetings and Interviews by Sub-Evaluation Team Annex D: List of Documents Reviewed Annex E: Mini Survey and Focus Group Discussion Questionnaires Annex F: List of Regulations to be Adopted by KJC and Their Status as of March 31, 2014 Annex G: Disclosure of Conflict of Interest ACRONYMS ABA/ROLI American Bar Association/Rule of Law Initiative ATRC Advocacy & Training Resource Center BIRN Balkan Investigative Reporting Network CDC Career Development Center CLEP Continuing Legal Education Program CMIS Case Management Information System CoP Chief of Party CSO Civil Society Organization DCA Department of Central Administration DEIPC Department for European Integration and Policy Coordination DILC Department of International Legal Coordination DLA Department of Legal Affairs DOJ/OPDAT Department of Justice/Office of Overseas Prosecutorial Development Assistance & Training ELCO European Commission Liaison Office EQ Evaluation Question EROL Effective Rule of Law Program ET Evaluation Team EU European Union EULEX European Rule of Law Mission in Kosovo FGDs Focus Group Discussions FJC Federal Judicial Center GIZ German Society for International Cooperation GMC Gender and Minority Committee GoK Government of Kosovo ICT Information Communication Technology ILEP Initial Legal Education Program ILZ International Legal Cooperation IT Information Technology JSP Justice Support Program KBA Kosovo Bar Association KCA Kosovo Chamber of Advocates KCC KI Kosovo Constitutional Court Key Informant KII Key Informant Interviews KJC Kosovo Judicial Council KJCS Kosovo Judicial Council Secretariat KJI Kosovo Judicial Institute KLP Kosovo Legal Profession Program KPC Kosovo Prosecutorial Council LKJC Law on Kosovo Judicial Council LKPC Law on Kosovo Prosecutorial Council LOC Law on Courts LPDI Legal Profession Development Initiative M&E Monitoring and Evaluation MCLE Mandatory Continuing Legal Education ME&A Mendez England & Associates MoJ Ministry of Justice MoU Memorandum of Understanding NCSC National Center for State Courts NGO Non-Governmental Organization OCS Office of Clinical Studies ODP Office of Disciplinary Prosecutor OP Office of the President OSCE Organization for Security and Co-operation in Europe PILPG The Public International Law & Policy Group PIO Public Information Officer PSA Public Service Announcement ROL Rule of Law SAO State Advocacy Office SCDO Swiss Cooperation and Development Office SOW Scope of Work SSRN Social Science Research Network STTA Short Term Technical Advisor ToT Training of Trainers UNDP United Nations Development Programme UNICEF United Nations Children’s Fund UPFL University of Pristina, Faculty of Law USAID United States Agency for International Development 1 EXECUTIVE SUMMARY EVALUATION PURPOSE This is a report on the mid-term performance evaluation of the implementation of the Effective Rule of Law (EROL) Program and the Kosovo Legal Profession (KLP) Program funded by the United States Agency for International Development (USAID) Mission in Kosovo. EROL is being implemented by Checchi and Company Consulting, Inc., and KLP is being implemented by the National Center for State Courts (NCSC). The evaluation of EROL and KLP was conducted during the period of March – April 2014 by a team assembled by Mendez England & Associates (ME&A) with headquarters in Bethesda, Maryland. The purpose of the evaluation is to “provide USAID/Kosovo with an objective external assessment of the management and performance of EROL and KLP activities.” The main goal of the evaluation is to assess the “efficiency, impact, sustainability, and relevance of the activities implemented through the EROL and KLP programs,” as well as to “examine the impact of the activities on the target institutions and validate/observe the progress made in achieving the results and objectives as specified in the EROL and KLP awards and the adjusted strategic plan.” The main thrust of the Evaluation Team’s (ET) mission was to: 1) review actual versus planned progress in attaining the anticipated results; 2) identify and analyze problems, delays and other issues related to project implementation; 3) document lessons learned; and 4) make recommendations for future USAID assistance in the justice sector. The evaluation covered the periods of March 2011 to the present for EROL and January 2012 to the present for KLP. The information uncovered by this evaluation will provide feedback to USAID/Kosovo for possible corrections for the remaining life of the EROL and KLP programs, which are expected to end by March 2015 and January 2015, respectively. PROJECT BACKGROUND Kosovo lags far behind other former Yugoslav states in terms of the rule of law (ROL). Serbian oppression in the 1990s, war, poverty, and inexperienced leadership have all contributed to the country’s systemic judicial sector weaknesses. Aware of the consequences that might be brought by these weaknesses, Kosovo's government has taken great steps to reform the judicial system with the package of laws, specifically the Law on Courts (LOC), Law on Kosovo Judicial Council (LKJC), Law on State Prosecutor, Law on Prosecutors Council, and Law on Bar. While much progress has been made in restructuring the court system and developing key justice institutions, such as the Kosovo Judicial Council (KJC), the Kosovo Prosecutorial Council (KPC), and the Ministry of Justice (MoJ), challenges remain: the level of transparency in the courts is still weak; the judiciary continues to suffer from political interference, inefficiency, and a lack of enforcement; and a system for establishing the ROL in North Kosovo has yet to be developed. All these factors have contributed to keeping investors out and preventing Kosovo’s growth and development. To address the above challenges and assist the Government of Kosovo (GoK) in building justice sector institutions and developing legislation, USAID/Kosovo funded the EROL and KLP programs. EROL aims to support the strengthening of Kosovo institutions, enhance the capacity of justice sector personnel, increase the efficiency of the courts, and improve public awareness and outreach throughout Kosovo. KLP seeks to improve the quality of legal education as well as the legal profession. EVALUATION DESIGN, METHODS AND LIMITATIONS To conduct the EROL and KLP evaluation, the ET collected quantitative and qualitative data from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process as well as accuracy and completeness of the subsequent conclusions, recommendations, and lessons learned. The team utilized techniques that balance each other such as: quantitative vs. qualitative data; individual vs. group responses; semi-structured interviews vs. analysis of existing surveys; and data sets. The following main sources of evidence were used: 2  Critical desktop review of materials related to EROL and KLP, such as quarterly reports, annual work plans, monitoring and evaluation (M&E) plans, EROL-commissioned baseline survey on court users satisfaction, justice institutions’ reports, and programmatic documents and reports from EROL grantees and international technical assistance projects in the ROL sector  Data sources from EROL and KLP project performance reports, outputs against objectives and performance indicators  Field visits to Pristina, Ferizaj, Gjilan, Mitrovica, Malisheve, Srbica, and Rahovec  Focus groups discussions (FGDs) with court staff, justice sector journalists, faculty members at the University of Pristina Faculty of Law (UPFL) and Iliria College  Interviews with USAID staff, EROL and KLP implementers, and project beneficiaries and stakeholders  Five mini-surveys with multiple choice and open-ended questions providing qualitative and quantitative data with court staff, court administrators and law students  Direct observation to cross-check information The ET encountered some limitations inherent to the design of this evaluation and during its fieldwork in Kosovo. Some of the more relevant limitations are listed below:  Time allotted for evaluation. The ET had just over 2 weeks for its fieldwork to assimilate data from key informant interviews (KIIs), FGDs, and project personnel meetings. Although the team worked in 2 sub-teams, given the broad scope of programs, required meetings with over 80 individuals, and field visits in several courts and universities, having more time for second round meetings with several key informants (KIs) would have been beneficial.  Difficulty in evaluating impact. At the time of the evaluation, a large number of project activities were recently implemented or still a “work in progress.” Due to the short implementation timeframe of many of the initiatives of EROL and KLP, the ET found it hard to conclude convincingly on their impact and longer￾term sustainability.  Possible attenuated link between project activities and results. Due to the presence of multiple technical assistance projects, it was difficult for recipient institutions to clearly distinguish the contribution of each donor, sometimes confusing one USAID program for another.  Recall bias. Since a number of questions during the interviews dealt with issues that took place in the past, recall bias cannot be excluded. As EROL activities were launched in March 2011 and KLP activities in January 2012, some respondents found it difficult to accurately compare access to services 2-3 years from prior to now. Furthermore, both EROL and KLP built upon the work of previous USAID projects, namely the Justice Support Program (JSP) and the American Bar Association/Rule of Law Initiative (ABA/ROLI); accordingly, a few respondents referred to the past and current activities interchangeably. In summary, while important, the above limitations did not prevent the ET from gathering sufficient information and data needed to draw conclusions and make recommendations in response to the specific questions it was tasked with answering. Below is a brief summary of the main findings, conclusions, and recommendations for each program. Further elaboration can be found in the relevant sections in the main body of the report. EROL FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Findings 1. EROL’s overall assistance to the MoJ, Office of the President (OP), Kosovo Constitutional Court (KCC), and KPC has had a positive impact in strengthening their operations and enhancing their overall institutional capacity. 2. EROL’s assistance to KJC has had a moderate impact on improving the internal organization and operation of KJC and its committees and enhancing coordination between KJC and other justice institutions. Several EROL activities have not met the stated objectives such as those aimed at increasing KJC’s capacity to develop and implement judicial policies, reducing case backlog, improving caseload management, filling judicial vacancies, evaluating judges, enforcing discipline, and strategic planning for judiciary. 3. EROL’s program assistance to KPC has been limited but beneficial in developing internal rules, an organizational structure, and an institutional strategic plan and creating a public relations strategy. 4. EROL’s assistance to MoJ has been limited but beneficial in improving the legal drafting skills and enhancing the capacity of the Department for International Legal Cooperation (DILC). Assistance in drafting primary 3 legislation and strengthening MoJ’s capacity to represent GoK in court proceedings has not occurred due to MoJ’s internal decisions that are beyond EROL’s control. 5. EROL’s assistance has enabled the OP to better fulfill its responsibilities in the justice sector through clarifying procedures on the appointment and re-appointment of judges and prosecutors, enhancing legal drafting skills of the OP’s legal staff, and improving its communication and outreach. 6. EROL’s program assistance to KCC has been limited but beneficial in improving its internal operations, increasing the legal reasoning and drafting skills of legal advisors, and enhancing the transparency of KCC’s activities. 7. EROL’s assistance to the Kosovo Judicial Institute (KJI) focused on the skills of the judiciary and prosecutorial sectors with technical assistance including training courses, training of trainers (ToT), judicial conferences, study tours, and website enhancement. EROL’s program assistance to KJI targets Objective 3 through improving the professionalism of justice system actors. 8. EROL has refurbished 11 courts in its Model Court Program and has engaged in activities that improve court operations, increase competency of court staff in court administration, and enhance citizens’ understanding of and respect for the courts. 9. EROL provided assistance through the strategic activities fund for civil society organizations [registered non-governmental organizations (NGO)], and through a small grants program to improve citizen engagement in justice sector reforms and increase citizens’ understanding of these reforms. EROL also supported a public outreach campaign in cooperation with various justice institutions such as KJC, KPC, court Public Information Officers (PIOs) for enhancing citizen understanding of the implementation of court re-structuring, and reaching out to minority communities. 10. EROL focused its assistance on meeting its objectives with MoJ, KJI, OP, KCC, KPC, as well as the Model Court Program. 11. The sustainability of EROL’s activities is dependent upon the commitment and capacity of the justice institutions and court leadership to maintain and advance EROL’s achievements. 12. While EROL did not specifically focus on strengthening gender diversity in its activities with the justice institutions, in the Model Court Program and with its citizen awareness campaign, it supported the creation of the Forum of Women Judges and Prosecutors. 13. Justice institutions received significant program assistance from EROL to meet important timelines for implementation of the judicial package laws related to court restructuring. 14. Although EROL’s assistance to KPC and KJC has improved their capacity to exercise responsibilities under the relevant laws, their overall capacities to fulfill their legally required responsibilities remain weak. EROL has already provided the required assistance to the KPC whereas the activities related to the KJC have not yet been accomplished therefore; KJC’s, capacity remains weak and needs to be strengthened significantly. 15. EROL’s capacity building activities focus on introducing to KJI the benefits of on-the-job training and coaching in comparison to classical training. 16. EROL’s capacity building activities for the justice institutions (MoJ, OP, KCC, KJC and KPC) staff utilized classical training through a combination of training sessions, roundtables, and workshops. 17. EROL coordinated with other USAID and other donors’ programs, yet the perception of EROL’s work by the international donor community is mixed. 18. EROL’s personnel changes with a concomitant lack of continuity in the provision of services and its reliance on numerous short-term advisors has been identified as a deficiency by the justice institutions. 19. There are considerable donors and international programs supporting ROL activities in Kosovo that represent different legal systems and have overlapping mandates; conflicting directives within and among donors; abundance of international experts, full-time and short-term; and various programmatic timelines. 20. Justice institutions’ officials interviewed by the ET stated that they are spending considerable time responding to the requests and needs of long- and short-term advisors provided by various international donors at the same time. This is preventing them from focusing on their own responsibilities, which they believe is one of the major reasons why international assistance provided to the justice system, including EROL’s, is not as effective and efficient as it could be. 21. Due to the abundance of donors, some justice institution officials in the KIIs were unable to definitively state which donor provided specific trainings (confusing JSP with EROL) and which ROL programs supported by USAID and the European Union (EU) provided advice and guidance on laws, regulations, and policies. 4 22. The level of satisfaction of EROL’s court user’s survey of July 2013 is approximately 70% whereas the level of public trust in the judiciary measured by the Public Pulse Report (Pulse) survey of August 2013, supported by USAID and the United Nations Development Programme (UNDP), is approximately 17%. 23. Trainees’ knowledge retention level is not measured. 24. The case registry database developed by EROL as part of the Model Court Program has been implemented and is utilized for court administration to varying degrees in all courts of Kosovo. There is a disparity among the statistics generated by EROL’s case registry database, the courts and the KJC; this is a result of various factors, none of which relate to the design of EROL’s case registry database. Conclusions 1. EROL’s assistance to justice institutions has been instrumental in ensuring a smooth and timely transition into the new court system in January 2013. 2. Although KJC’s management and operations have been moderately improved due to EROL’s assistance, KJC’s capacity in exercising its main responsibilities to develop and implement policies for a more efficient and effective administration of the courts, particularly in the areas of reducing case backlog, improving caseload management, filling judicial vacancies, evaluating judges, enforcing discipline, and strategic planning for judiciary remain weak. 3. EROL’s assistance has helped KPC achieve moderate progress in fulfilling its responsibilities to develop and implement policies for the prosecutorial system, particularly developing long-term strategic plans, enforcing prosecutorial discipline, and evaluating prosecutorial performance. 4. EROL’s program assistance to MoJ has been limited yet beneficial in enhancing DILC’s capacity to handle requests for international legal cooperation more effectively. The adoption of the KPC’s strategic plan and the public relations strategy are delayed due to the KPC’s leadership decisions which are beyond EROL’s control. 5. While limited, EROL’s assistance to the OP has been beneficial and strengthened the collaboration among OP, KJC and KPC, and enhanced the legal drafting and research skills of the legal staff. 6. EROL’s program assistance to KCC has been beneficial in improving its rules of procedure, increasing the transparency of its activities, and enhancing the skills of its legal advisors. 7. KJI is strengthened and has benefitted by the implementation of EROL’s assistance. By providing support to KJI, EROL has met Objective 3 and has advanced ROL objectives. Further, EROL’s assistance to KJI is sustainable through education, trainings, curriculum, ToT, as well as improved website development and portal. 8. The courts of Kosovo have been strengthened and benefited from EROL’s Model Court Program by the improvement of court operations, advancement of the technical skills of court staff, upgrading of judges’ and court staff’s service to the public, enhancement of citizens’ interaction with court staff, and increase in citizens’ knowledge of and respect for justice services. 9. EROL has contributed to implementing the advanced ROL objectives by improving the capacity of the justice institutions and enhancing the skills of justice sector professionals, particularly court personnel, through its Model Court Program, resulting in a more efficient, transparent, independent, and accountable justice sector. EROL has also helped to improve civil society’s ability to conduct more effective oversight and advocacy by increasing citizen awareness and its role in ensuring the delivery of justice. However, the impact of minority outreach is difficult to measure due to the current political situation in the north. 10. EROL’s interventions with the KJI, OP, KCC, and KPC, as well as its robust Model Court Program, have met its 4 stated objectives. EROL’s interventions with MoJ have not fully met its objective to enhance the MOJ’s capacity to better represent the GoK in court proceedings due to MOJ’s delays in appointing the State Advocate and establishing the SAO. The assistance to the KJC has not met the objective of transforming the JKC into an effective governing body of Kosovo’s judiciary due to a number of factors, including the KJC’s limited capacity to absorb international technical assistance, the overabundance of international donors with individual missions and the lack of a more proactive approach from EROL’s team. 11. The ET cannot conclude on the sustainability of EROL’s activities since this is dependent upon the commitment of GoK and justice institutions to maintain and advance EROL’s achievements. 12. The Forum for Women Judges and Prosecutors has the potential of being a powerful voice in increasing and strengthening gender diversity in the justice system. 5 13. EROL’s assistance to KJC, KPC, courts and prosecution offices, and OP was crucial and enabled these justice institutions to meet the timelines of their respective responsibilities during the restructuring of the court system in accordance with the new LOC (Law No. 03/L199) and the Law on Kosovo Judicial Council (Law No. 03/L223). 14. Despite the improvements achieved through EROL’s assistance, the capacity of KPC and KJC to become effective in exercising prescribed responsibilities under the judicial package laws remains weak. EROL’s impact in drafting of secondary legislation has not been significant. 15. KJI has benefitted from EROL’s introduction of on-the-job training and coaching. The efforts have been well-received and more on-the-job training has been requested. 16. EROL’s capacity building events to trainees of different justice institutions largely utilized classical training methods to the benefit of the trainees. 17. While EROL coordinates with other USAID and other donors’ programs, this coordination is not to the degree which generates universal respect among donors. 18. EROL’s staff changes and reliance on short-term advisors caused disruptions in the provision of services to the justice institutions, although the services that were provided were considered to be of high quality. 19. It is difficult to attribute impact and benefit specifically to EROL as there are numerous international donors who provide similar assistance to the justice institutions. Furthermore, many officials within the justice institutions are often unable to identify which donor provided trainings and workshops, technical advice and guidance. 20. The overabundance of donors and international programs supporting ROL activities has unintended consequences and has impeded progress. EROL’s programmatic effectiveness and efficiency is reduced in those areas in which there are many donors. Its success in the Model Court Program and its citizen awareness campaign are due to several factors, of which the most important is that it worked in an environment without substantial involvement by other donors. 21. While informal agreements exist among donors, there is no formal coordination mechanism among all ROL donors, which would offer a meaningful consultation and division of responsibilities among them. 22. While court users’ satisfaction is considerably higher in the EROL survey than in the Pulse survey, which measured the level of public trust in the judicial system, it is impossible to compare the level of satisfaction between the 2 surveys due to issues of incompatibility. 23. Knowledge retention is not being measured by EROL; therefore, results cannot be analyzed. 24. The case registry system is maintained as accurately as possible and is utilized in varying degrees by courts for court administration. It is an essential first step in the development of a fully-automated case management information system. Recommendations  EROL should significantly decrease assistance to MoJ, OP, KCC and KPC, and redirect it to KJC and the courts.  EROL’s assistance to the MoJ should focus on supporting the process of revising or drafting justice related laws and build the capacity of SAO.  EROL should continue to offer assistance to the legal staff of the OP and facilitate coordination with other justice related institutions.  At KJC, EROL’s assistance should focus on increasing KJC’s and Kosovo Judicial Council Secretariat’s (KJCS) capacity in developing and implementing policies for a more efficient and effective administration of the courts. Future institutional strengthening assistance is also needed to improve KJC’s capacities in developing judicial policies on caseload management, case backlog reduction, judicial discipline, judicial statistics and data based strategic planning.  EROL’s in-country team should engage more actively with KJC and KJCS as opposed to extensive short￾term technical assistance (STTA) engagement.  KJI needs the support of major donors to continue but the effort needs to be meaningfully organized.  The Model Court Program is not yet sustainable and should continue to receive USAID’s support. The continuation of the Model Court Program should focus on fully implementing court administration policies within all courts of Kosovo, and on continued training of all court personnel, including judges, on court administration and case management. As the new Case Management Information System (CMIS) is being implemented, EROL’s assistance will continue to be necessary as it adapts to the court administration policies associated with KJC and CMIS. 6  EROL’s assistance to the Forum for Women Judges and Prosecutors should focus on increasing the organization’s membership, strengthening its internal management and operations, and building alliances in international organizations of women judges and prosecutors.  EROL should expand outreach activities in non-majority communities and, should the situation allow, launch a sustained public outreach in Northern Kosovo.  While informal agreements exist among ROL assistance programs, the existence of a formal coordination mechanism among the major ROL donors would offer a meaningful consultation and division of responsibilities among the donors, thereby reducing overlapping and conflicting projects.  Based on ET interviews, given that Kosovar capacity – all of whom have benefited from JSP’s and EROL’s interventions – is sufficiently high to assume leadership positions in USAID ROL programs, serious consideration should be given to reducing the number of international senior staff positions and increasing the number of local senior staff positions in future USAID ROL programs. KLP FINDING, CONCLUSIONS AND RECOMMENDATIONS Findings 1. KLP’s program assistance to UPFL and the Faculty of Law of Iliria College focuses on broad institutional support, skills training, career development, and academic research. KLP’s program assistance to UPFL and Iliria College targets the strength and quality of legal education, along with the USAID objective of empowering Kosovars to consolidate a functioning democracy. 2. KLP has provided key support to the Kosovo Bar Association (KBA, formerly the Kosovo Chamber of Advocates or KCA) on the new institutional framework, designed and implemented a large number of the legislative reform assistance, created a new structure for the Mandatory Continuing Legal Education (MCLE) disciplinary system, and developed the new KBA strategy. 3. KLP’s activities in strengthening gender and minority participation strategically focus on attracting and sustaining female and minority students to the Masters Level Law programs. As a result, there is an increase in the number of females who enroll in the program, take the bar exams, attend professional panel discussions, and receive internships. 4. KLP participated in, facilitated, and enhanced KBA’s GMC. 5. KLP provided KBA with expertise on all internal regulations, which have been harmonized with the Law on the Bar and KBA Statute, and are adopted by the KBA Managing Board. 6. KLP’s capacity building activities target UPFL and Iliria College with on-the-job training and coaching with results. 7. After the new Law on the Bar came into force, KLP was instrumental in enforcing MCLE regulations, introducing a new and improved Disciplinary Regulation, and establishing the Office of the Disciplinary Prosecutor (ODP) and Reform of the KBA Disciplinary Committees. 8. KLP appears committed to increasing the effectiveness of its assistance and strengthening donor coordination. 9. Repeated KIIs indicate that KLP’s success can be directly attributable to its Chief of Party’s (CoP) long￾lasting reputation and length of time spent in Kosovo. 10. Programmatic deficiencies include that activities with Law Faculties must be framed within the constraints of the academic and funding cycles. 11. KLP focuses efforts on the management, quality, and availability of the legal clinics at the Law Faculties. 12. The 3 main ways in which KLP has assisted in improving the research capacity at UPFL include: 1) assisting in writing a draft statute to establish a research institute to assist in publishing academic papers; 2) distributing the Blue Book, which provides proper legal citations, to law students and faculty; and 3) introducing law students and professors to the Social Science Research Network (SSRN), which is an avenue for the publication of social science papers. 13. KLP’s strong focus on Iliria College support and the addition of the legal clinics, has led to a noticeable jump from a strictly literature (theory only) style learning to clinical education (practical experience) and greatly improved research capacities for both faculty and students. 14. KLP has engaged in many activities to improve MCLE and the disciplinary system of KBA. 7 Conclusions 1. Justice institutions in Kosovo are strengthened and have benefited from the implementation of KLP’s interventions. Through strengthened legal education and career enhancement, KLP has met ROL objectives of empowering Kosovars to consolidate a functioning democracy. KLP’s efforts become sustainable by education, mentoring programs, academic publications, and job placement activities. 2. KLP has contributed to strengthening KBA by strategically improving its legal framework. KLP has also assisted with the enforcement of regulations and development of MCLE policies and procedures; improved the reorganization of KBA membership to improve the effectiveness of mentoring and professional development of members; professionally strengthened MCLE and other training programs; and improved outreach to the legal community on MCLE requirements. 3. The Faculties of Law at University of Pristina and Iliria College are strengthened in their gender diversity. However, other major donors such as EU and the German Society for International Cooperation (GIZ) are involved with increasing gender initiatives, thus it is difficult to link all improvements directly to KLP alone; regardless, it is clear that KLP’s efforts are strategically focused and successful. 4. A direct impact of KLP’s interventions is the re-establishment of the Gender and Minority Committee (GMC) of KBA. 5. KLP’s activities effectively contributed to re-organization of KBA, with the main work on the legislative amendments to the Law on the Bar and to the secondary legislation. 6. KLP’s capacity building activities have resulted in Masters Level law students that have better practical skills. KLP established cooperation mechanisms between the Law Faculties and other relevant legal institutions, and strengthened career centers and academic research capabilities. 7. With the changes made as a result of the implementation of some of KLP’s initiatives, Kosovo’s justice system has made progress in improving cooperation between the justice institutions and promoting ROL. 8. KLP’s activities in coordinating with USAID and other donors programs are effective and efficient. 9. KLP’s CoP is credited for his knowledge of the country’s legal education environment and his ability to navigate himself in the international donor community. 10. Both identified program deficiencies have been remedied to the best of KLP’s abilities and on-the-ground realities. 11. KLP’s efforts in management, quality, and availability of the legal clinics at the universities are significant. As a result, the universities provide a better environment for the students. 12. KLP’s efforts to improve the academic research capacity at UPFL are off to a good start; however, much work remains to be done. Although KLP has set the stage for huge improvements in UPFL’s research capacity, including starting the process to establish a research institute and facilitating beneficial relationships for the publication of academic research, these require further assistance in order to achieve actual publication. 13. KLP’s support to Iliria College is visible and profound; however support is still needed. 14. As a result of regulations drafted by KLP and KBA, 90% of KBA’s membership now complies with MCLE requirements. Although the number of KBA members complying with MCLE’s requirements has increased significantly – 72% (2011), 92% (2012), and 93% (2013) – the general awareness of what constitutes a disciplinary violation amongst lawyers remains weak. In general, the disciplinary system has been improved through training workshops developed and delivered by KLP, and although the Disciplinary Committee and the Office of the Disciplinary Prosecutor are more active they lack the proper expertise and need further strengthening to enforce lawyer discipline. Recommendations  KLP’s focus needs to be on the sustainability of the clinics, Career Development Centers (CDCs), and research capabilities. Partner institution support should be strengthened.  Continued robust support needs to be given to Legal Education.  Focus should be on legal specializations from curriculum to research institutions.  Academic research institutions need further support. These institutions should support specializations as well; for example, a Fiscal Law Academic Research Institution.  Partnerships with US Universities/Law Schools should be actively promoted, for student as well as professor exchange. 8  KBA and MCLE should be maintained with USAID’s support as KBA lacks the professional ability to do so. In order to achieve sustainable results from its activities, KLP should continue to work with KBA on regulatory reform.  KLP should further assist KBA on the quality of the MCLE Training Center and on improving the quality of legal education.  Additional assistance to the KBA GMC is necessary for it to become more engaged in empowering women and underrepresented communities, and to help increase the number of KBA’s women interns.  Future KLP support to KBA is necessary to focus mainly on increasing the quality of MCLE and strengthening the Office of the Disciplinary Prosecutor. According to the disciplinary provisions of KBA, the new control system of MCLE trainers should be established. The quality of MCLE trainers should be improved, as they are the main weakness of the current format. Well-known experts from the various fields of law should be invited to deliver trainings at MCLE. MCLE needs further support to function at a higher level, serve the needs of the lawyers, and become sustainable.  Should the political circumstance allow, KLP and KBA need to focus their outreach efforts to integrate the Kosovo Serb lawyers in Kosovo’s legal profession through licensing and training delivery in their native language. This will facilitate the full integration of Kosovo Serbs in the legal community. GENERAL RECOMMENDATIONS  If the result of the Normalization Agreement between Kosovo and Serbia create the conditions for the restoration of court institutions in the North, USAID should utilize its convening power and EROL’s successful experience in implementing court re-structuring to facilitate the implementation of transitional arrangements for establishing effective and efficient court operations in the North.  USAID’s future assistance in the ROL area in North Kosovo (through EROL, KLP or otherwise) should dedicate a strong donor coordination component between US, EU, and bilateral donors in order to avoid the negative consequences of uncoordinated assistance.  USAID should promote the establishment of the formal donor coordination mechanism where donors share their plans and clearly delineate the areas of assistance in the justice sector in North Kosovo before embarking on implementation. This donor coordination body should have periodic coordination meetings during the implementation phase in order to avoid overlapping projects and conflicting advice.  In accordance with the standards of the Model Court Program, EROL should work closely with KJC to complete the ongoing physical intervention in the court building in Mitrovica and start refurbishing Branch Court buildings in Leposavic and Zubin Potok.  EROL should work closely with the leadership and Basic Court of Mitrovica and KJC to coordinate transitional arrangements for establishing courts in the North such as recruiting of judges and court personnel, training programs, file transfers, record management and court administration.  Upon resumption of basic court operations in the North, EROL should facilitate the participation of court leaders in Mitrovica, Leposavic, and Zubin Potok in the KJC workshops with court presidents, and prepare plans for establishing case management offices, installing tracking databases, and training judges and court administrators on court administration.  EROL and KLP should work with KJC, KPC and KBA to establish closer relationships with Kosovo Serb lawyers, judges and prosecutors from North Kosovo to facilitate their integration in the legal community of Kosovo.  USAID’s future assistance in ROL should include a robust minority outreach component targeting the Kosovo Serb and other non-majority communities of Kosovo through national and local media. 1 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the mid-term performance evaluation of the implementation of the Effective Rule of Law (EROL) Program and the Kosovo Legal Profession (KLP) Program funded by the United States Agency for International Development (USAID) Mission in Kosovo. EROL is being implemented by Checchi Consulting between March 2011 and March 2015, and KLP is being implemented by the National Center for State Courts (NCSC) between January 2012 and January 2015. The main goal of the evaluation is to assess the efficiency, impact, sustainability, and relevance of the activities implemented through the EROL and KLP programs, as well as to examine the impact of the activities on the target institutions and validate/observe the progress made in achieving the results and objectives as specified in the EROL and KLP awards and the adjusted strategic plan. The Evaluation Team (ET) was tasked with reviewing actual versus planned progress in attaining the anticipated results; identifying and analyzing problems, delays and other issues related to project implementation; documenting lessons learned; and making recommendations for future USAID assistance in the justice sector. The results of the evaluation will be used by USAID/Kosovo for possible corrections for the remaining life of the EROL and KLP programs. The target audience for this evaluation includes USAID/Kosovo staff, especially the Democracy and Governance Office, its implementing partners, local stakeholders, and local beneficiaries. Local stakeholders include the Kosovo Judicial Council (KJC), Kosovo Prosecutorial Council (KPC), Ministry of Justice (MoJ), Kosovo Judicial Institute (KJI), Office of the President (OP) of Kosovo, Constitutional Court, Presidents and Court Administrators of completed Model Courts, the European Union (EU) Office and their implementing partners’ representatives, Kosovo Bar Association (KBA, formerly Kosovo Chamber of Advocates or KCA), Faculty of Law of the University of Pristina (UPFL), and Iliria College. 1.2 EVALUATION QUESTIONS As per the scope of work (SOW), the ET was asked to answer a number of evaluation questions – overall questions for both EROL and KLP and project-specific questions, outlined below: 1. How have the justice institutions in Kosovo been strengthened and benefited from the implementation of EROL’s and KLP’s program assistance components? To what extents have the programs met their five stated objectives and how effective have the programs’ interventions been in achieving the programs’ stated objectives? Why or why not? To what extent have projects implemented the advanced ROL objectives? Are these activities beneficial to the institutions? Are they sustainable? If not, why not? 2. How have the justice institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of EROL’s and KLP’s activities? To what extent have the programs supported/assisted institutions to increase and strengthen gender diversity? Why or why not? 3. What is the current capacity and timeline for justice institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on Kosovo Judicial Council, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar. The impact of the programs on secondary legislation will also be important)? 4. What results have EROL’s and KLP’s partner institutions achieved through capacity building activities of the subject programs?1 How much has EROL and KLP utilized on the job training and coaching with partner institutions (in comparison to classical training) and what are the concrete results on the ground? 5. How effective and efficient are EROL’s and KLP’s activities in coordinating with other USAID and other donors’ programs? 1 As approved by USAID/Kosovo, the ET merged part of Question 3, dealing with on-the-job vs classical training, with Question 4. The questions listed here represent the new and approved questions. 2 6. Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? 7. What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs? EROL Specific Questions 1. What is the level of satisfaction of EROL’s court users’ survey compared vs. Public Pulse survey supported by USAID? 2. Is the trainees’ knowledge retention level of the EROL trainings supported at the KJI measured and what are the results of it? 3. To what extent is the court case database system populated and used for court administration throughout Kosovo? KLP Specific Questions 1. To what extent is the management, quality, and availability of the legal clinics at the Law Faculty of the University of Pristina and the Iliria College better as a result of KLP assistance? 2. How has the academic research capacity at the Law Faculty of the University of Pristina improved as a result of the KLP assistance? 3. What have been the effects of the KLP support for the Iliria College? 4. How has KLP contributed to improving the functioning of the mandatory continuing legal education and the disciplinary system at the Kosovo Chamber of Advocates? 5. How has gender and ethnic diversity improved within the Kosovo Chamber of Advocates membership as a result of the KLP assistance? 2.0PROJECT BACKGROUND Kosovo lags far behind other former Yugoslav states in terms of the ROL. Serbian oppression in the 1990s, war, poverty, and inexperienced leadership have all contributed to the country’s systemic judicial sector weaknesses. While much progress has been made in the recent years in restructuring the court system and developing key justice institutions such as KJC, KPC, and MoJ, challenges remain: the level of transparency in the courts is still weak; the judiciary continues to suffer from political interference, inefficiency, and a lack of enforcement; and a system for establishing the ROL in North Kosovo has yet to be developed. All of these factors have contributed to keeping investors out and preventing Kosovo’s growth and development. To address the above challenges and assist the Government of Kosovo (GoK) in building justice sector institutions and developing legislation, USAID/Kosovo funded 2 ROL programs, EROL and KLP. EROL was initiated in 2011 with the purpose of supporting the strengthening of Kosovo institutions, enhancing the capacity of justice sector personnel, increasing the efficiency of the courts, and improving public awareness and outreach throughout Kosovo. KLP was initiated in 2012 and sought to improve the quality of legal education as well as the legal profession. Since launching EROL and KLP, Kosovo’s justice institutions underwent a number of important re￾organizations and reforms including, but not limited to, adoption of new laws on KJC, KPC and courts; reorganization of the judicial and prosecutorial systems; appointment and re-appointment of judges and prosecutors; introduction of new criminal procedure codes; and large scale court refurbishment and renovations. The scope and pace of such interrelated transitions occurring simultaneously and within a very short time poses a problem to any technical assistance project. 2.1 EROL EROL is a 4-year program that focuses on making the justice system more independent, accountable, efficient, and effective. The centerpiece of this program is bolstering the justice sector’s operational capacity, focusing on the courts, KJC, MoJ, and KJI. EROL’s purpose is to contribute to USAID’s strategic vision of “Kosovo as an effective state, with a viable economy and an inclusive democracy on the path to European Integration” through USAID/Kosovo Assistance Objective 3: Empowering Kosovo’s Citizens to Consolidate a Functioning Democracy. 3 EROL seeks to: 1) strengthen the independence, accountability, efficiency, and effectiveness of the justice system; and 2) bolster public confidence in the ROL by increasing public knowledge of and participation in the justice system through technical assistance and training for justice-related institutions, refurbishing court facilities, and targeting public awareness activities. EROL’s assistance is broken down into 4 components, each with a unique objective:  Objective 1: Improve the effectiveness of justice sector institutions  Objective 2: Increase the efficiency of court operations  Objective 3: Improve the professionalism of justice system actors  Objective 4: Increase citizen awareness and role in ensuring the delivery of justice 2.2 KOSOVO LEGAL PROFESSION PROGRAM KLP is a 3-year program with the purpose of supporting a capable and active legal education and legal profession, as well as contributing to an increased understanding by citizens of their rights. KLP has 2 components: 1) supporting a better quality of legal education; and 2) supporting an active and robust legal profession. KLP works with the UPLF and Iliria College to support clinical programs and practical skills development for law students. It supports the Faculties to strengthen their academic research activities and establish linkages between them and public and private organizations through Career Development Centers (CDCs) with the aim of improving students’ opportunities for internships and possible employment. In addition, KLP supports the establishment of functional cooperation among the Faculties and other organizations related to the legal profession. The program works with the KBA and its regional offices as a means to reach more lawyers. It facilitates better cooperation among lawyers, judges and prosecutors in order to identify weaknesses in the system and provide recommendations for improvement. Women and minority communities are paid particular attention in all focus areas, especially in those related to developing initiatives/activities to encourage their participation in broader areas of the law. 3.0 EVALUATION METHODS & LIMITATIONS 3.1 EVALUATION METHODOLOGY The evaluation methodology was carefully designed to respond to the evaluation questions outlined in the SOW, as well as to assess EROL’s and KLP’s current performance including their efficiency, impact, sustainability, and relevance. Given the type of EROL and KLP project activities, both qualitative and quantitative methods were used during the evaluation. Data was collected from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process, as well as accuracy and completeness of the subsequent conclusions, recommendations, and lessons learned. To collect data, the ET used a number of techniques that balance each other, including: quantitative vs. qualitative data; individual vs. group responses; semi-structured interviews vs. analysis of existing surveys; and data sets. 3.1.1 Qualitative Research and Analysis Qualitative data was collected from the following main sources of evidence:  Critical desktop review of materials related to EROL and KLP such as quarterly reports, annual work plans, monitoring and evaluation (M&E) plans, EROL commissioned baseline survey on court users satisfaction, justice institutions reports, programmatic documents and reports from EROL grantees and international technical assistance projects in the ROL sector.  Review of programs’ outputs against objectives and performance indicators  Interviews with key informants in the US  Interviews with USAID/Kosovo’s M&E Specialist, as well as with staff from the project implementers, Checchi Consulting and NSCS 4  In depth, semi-structured interviews with selected program beneficiaries and stakeholders such as KJC, KPC, Kosovo Constitutional Court (KCC), MoJ, KJI, KBA, UPLF, Iliria College, court presidents and supervisory judges, court administrators and staff, lawyers, law professors, law students, minorities, etc.  Meetings with court presidents and court administrators in 9 courts of all levels of Kosovo judiciary  Site visits to Supreme Court of Kosovo and Court of Appeals, as well as basic courts and branches in Pristina, Ferizaj, Gjilan, Mitrovica, Malisheve, Srbica, and Rahovec  Focus groups discussions (FGDs) with court staff, justice sector journalists, and faculty members at UPLF and Iliria College  Direct observation to cross-check information (e.g. comparing statements to observed practice) and identification of factors not previously recognized 3.1.2 Quantitative Data Collection Quantitative data collection consisted of data sourced from objective performance reports, project and progress reports, and performance indicators; previous evaluations and assessments conducted by the project or USAID; and mini-surveys and questionnaires. The ET conducted 5 mini-surveys: 3 with EROL court staff, most of whom had been EROL trainees, and 2 with KLP beneficiaries. The EROL surveys asked 9 multiple choice questions and 3 open-ended questions to court personnel, including court administrators, court registrars, public information officers, and legal associates covering courts of all levels that have received training and/or technical assistance from EROL’s Model Court Program. The survey aimed at gauging the effect of EROL’s capacity building and technical assistance in improving court operations and services, the extent to which court personnel is using the knowledge attained through on-the￾job coaching in carrying out their daily duties, the extent to which the case tracking database is populated in different courts, and their observations of shortcomings and improvements needed. Two mini-surveys were conducted with the law students at UPLF and Iliria College regarding the management, quality, and availability of the legal clinics. The surveys aimed at gauging the effect of law students’ participation in the legal clinics, their observations for improvement, and whether the skills-based training they received in the legal clinics had a positive impact on their educational experience. The mini-surveys included questions regarding the efficiency, impact, sustainability, and relevance of the activities implemented through EROL and KLP which the recipients engaged in. Mini-surveys were developed in English and translated into Albanian language. Responses to mini-surveys were translated into English. To assure respondents of the confidentiality of their input, the questionnaires were kept anonymous. Raw data were recorded on the individual questionnaires used for the interviews. To facilitate the analysis, the responses to the questions were compiled and tabulated on a spreadsheet that facilitated the comparison of the responses. This helped define response patterns and determine the similarity of the different responses. 3.2 EVALUATION LIMITATIONS The ET encountered some limitations inherent to the design of this evaluation and during its fieldwork in the capital and other cities. Some of the more relevant limitations are listed below:  Time allotted for evaluation. The ET had just over 2 weeks for its fieldwork to assimilate data from key informant interviews (KIIs), FGDs, and project personnel meetings. Although the team was divided in 2 sub-teams, given the broad scope of programs, required meetings with over 80 individuals, and field visits in several courts and universities, having more time for a second round of meetings with several key informants (KIs) would have been beneficial.  Difficulty in evaluating impact. At the time of the evaluation, a large number of project activities had been recently implemented or were still a “work in progress.” Due to the short implementation timeframe of many EROL and KLP initiatives, the ET found it hard to conclude convincingly on their impact and longer term sustainability. The recently adopted National Backlog Reduction Strategy, new office of Disciplinary Prosecutor at KBA, a recently re-vitalized Gender and Minority Committee (GMC) at KBA, and ongoing development of strategic plans at KJC and KPC have yet to produce concrete results that would allow the ET to conduct a thorough assessment of their tangible impact. 5  Possible attenuated link between project activities and results. Due to the presence of multiple technical assistance projects, it was difficult for recipient institutions to clearly distinguish the contribution of each donor. This suggests that there are many factors or variables that may contribute to the results described in this report.  Recall bias. As EROL activities were launched in March 2011, and KLP activities in January 2012, recall bias cannot be excluded. Indeed, some respondents found it difficult to accurately compare access to services 2-3 years from prior to now. Furthermore, both EROL and KLP built upon the work of previous USAID projects, namely Justice Support Program (JSP) and the American Bar Association/Rule of Law Initiative (ABA/ROLI), and a few respondents referred to the activities of past and current activities interchangeably. However, the ET responded to this bias by clarifying the scope of the evaluation and differentiating between EROL’s and KLP’s assistance and previous USAID programs in the area. In summary, while important, the above limitations did not prevent the Evaluation Team from gathering sufficient information and data needed to draw conclusions and make recommendations in response to the specific questions it was tasked with answering. 4.0 EROL OVERALL FINDINGS AND CONCLUSIONS 4.1 QUESTION 1: How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of EROL’s and KLP’s program assistance components? To what extent have the programs’ met their five stated objectives and how effective have the programs’ interventions been in achieving the program’s stated objectives? Why or why not? To what extent have projects implemented the advanced ROL objectives? Are these activities beneficial to the institutions? Are they sustainable? If not, why? 4.1.1 Findings EROL’s assistance to KJC has had a moderate impact on improving the internal organization and operation of KJC and its committees and enhancing coordination between KJC and other justice institutions. Several EROL activities aimed at increasing KJC’s capacity to develop and implement judicial policies; reducing case backlog, improving caseload management, filling judicial vacancies, evaluating judges, enforcing discipline, and strategic planning for judiciary have not met the stated objectives. KJC is the constitutional body for overall administration of courts and for recruiting, advancing, transferring, training, disciplining and dismissing judges. Under the new law on KJC, effective since 2011, KJC has 4 permanent committees, including the Committee for Normative Issues; the Committee for Budget, Finances and Personnel; the Committee for Court Administration; and the Disciplinary Committee. In addition, KJC can establish other temporary or standing committees as it deems necessary. KJC’s operations and decision-making processes are supported by the Kosovo Judicial Council Secretariat (KJCS), which currently consists of around 120 staff members organized into several departments – administration, budget and finance, judicial statistics, and logistics – each consisting of various offices. Assistance to KJC and KJCS constitutes an important part of EROL’s Objective 1, which is designed to achieve more effective operations of Kosovo’s justice sector institutions. The following information draws on the ET’s analysis of KIIs with officials from justice institutions and EROL staff, and of EROL’s programmatic documents. The ET found that EROL implemented activities with KJC through a two-pronged approach: 1) assisting KJC and courts in the court restructuring process; and 2) providing institution strengthening assistance to KJC as the governing body of Kosovo’s judiciary. EROL’s initial assistance to KJC on implementing the new Law on Courts (LOC), which entered into force in 2011 and called for a re-organization of Kosovo’s court system starting from January 1, 2013, was both intense and of high quality. The LOC introduced a new three-tiered court system with 7 Basic Courts and 20 Branch Courts, which serve as courts of first instance, a Court of Appeals, and a Supreme Court as the highest judicial body in Kosovo. The creation of the Court of Appeals and the concomitant changes of jurisdiction necessitated appointments and reappointments of judges and court 6 personnel, as well as transfer of cases to the new competent courts in order to ensure uninterrupted court services throughout Kosovo. As detailed further in response to the evaluation question (EQ) 3 in section 4.3.1, EROL’s expertise and hands-on support to KJC and courts in planning and implementing the court￾restructuring process was instrumental and received overwhelming praise from representatives of all justice institutions. In addition to assisting KJC in coordinating the court restructuring process and adopting necessary regulations and instructions to manage the transition, EROL was required to provide assistance to improve KJC’s capacity to effectively govern the court system. EROL has offered continuous assistance through its local and international long-term advisors and several short-term experts to KJC and its committees and to KJCS. EROL provided a number of resource materials designed to improve KJC’s public communication and outreach, including media relations guidelines, a crisis communication guidebook, a communication strategy plan, and a communication strategy implementation plan. The ET found that 2 years after their finalization, the media and communication documents have not been formally adopted by KJC. KIIs with justice institutions and the EROL team identified 2 conflicting reasons for KJC’s failure to adopt the documents: 1) the Normative Issues Committee believes these documents are internal operational guidelines and not normative sub-legal acts that would require codification by KJC; and 2) the Normative Issues Committee has been occupied with other, more immediate, regulations. EROL’s expertise in the process of drafting a new 5-year strategic document called KJC’s Strategic Plan for Judiciary (2014-2019) is considered of good quality despite the fact that the Strategic Plan development process has been very slow. EROL’s facilitation of a working group was properly designed to enhance the local buy-in through a comprehensive consultation process between KJC members, KJCS, court administration, court employees, and civil society representatives. However, EROL’s work has not met with commensurate interest from KJC. As of March 2014, while the document was finalized by the working group, it had yet to be formally adopted by KJC. In September 2013, KJC adopted a new National Backlog Reduction Strategy, which was developed with EROL’s assistance to replace the previous strategy of 2007. EROL is now providing assistance to courts in preparing an implementation action plan for the backlog reduction at each court and individualized action plans for each judge. According to its contract, EROL was required to work closely with KJC to create and enforce policies and procedures for case management, records management, and retention and information communication technology (ICT) management. EROL’s interventions in this regard include the design of a case-tracking database that was initially developed as a way to inventory all pending cases in Kosovo courts for the purpose of transferring case files to competent courts in accordance with the Implementation Plan for the LOC. The case-tracking database is now used to a varying degree in Kosovo courts and is currently estimated to include 500,000 cases. However, its future usefulness and sustainability are in question due to KJC plans to implement a new case management information system (CMIS) in accordance with KJC’s ICT Strategy (2012-2017) which EROL also helped develop. KJC has received financial support from the Government of Norway and an international technical assistance project, launched in January 2014, is expected to design and introduce the new CMIS in all Kosovo courts within 2 or 3 years. In addition, EROL is currently working with KJC on finalizing a web portal for all Kosovo courts, including the Court of Appeals and the Supreme Court. At the moment, KJC and EROL are still working on the rules and procedures on the web-content development but the timing for launching the web-portal for all Kosovo courts is not yet clear. Despite EROL’s technical assistance to strengthen KJC’s capacities, the ET collected sufficient evidence through KIIs, on-site observations, and FGDs to conclude that KJC’s capacities to administer Kosovo judiciary remain weak. The ET found that beyond the court restructuring process, EROL’s assistance to improve KJC’s and its committees’ operations, as well as strengthen KJC’s capacity to govern the court, has been as effective and successful as anticipated. Due in part to the limited capacity of KJCS’s staff to carry out their assigned responsibilities in a professional manner, EROL’s assistance has not resulted in significant re-invigoration of KJC’s committee structures and KJC’s decision-making process. Many important initiatives on revising case load standards, filling judicial vacancies, and adopting key regulations are awaiting deliberation and/or approval by KJC. EROL’s interventions aimed at enhancing KJC’s capacity to administer Kosovo courts more effectively including, but not limited to, improved data-based court administration, case load management, case backlog reduction, long-term strategic 7 planning, and information technology management, are still ongoing and the likelihood of such initiatives to deliver the expected impact in the remaining life of the project is questionable. KIIs disclosed several reasons for EROL’s limited success with KJC. First, the KJC is the target of international assistance from numerous organizations with different interests and concepts of court administration, which makes the coordination and division of responsibilities a daunting task for Kosovo justice institutions and leads to projects having an adverse impact on planning and decision-making. Secondly, the KJC is predominantly comprised of fulltime judges who continue their judicial tasks while serving on the KJC and its committees. Fulltime service in multiple judicial roles makes it humanly impossible for KJC judge members to devote time and energy to KJC’s activities and seriously hampers KJC’s ability to perform its role as the judiciary governing body. As a result, KJC continues to operate in a reactive manner with a piecemeal approach to the issues facing court administration, often relying on technical assistance providers to guide its work. Furthermore, the KJCS does not have sufficient in-house capacity to provide the KJC with necessary expertise and technical advice in developing policies for the efficient and effective administration of the judiciary. Lastly, the ET learned that KJCS staff members were transitioned from the Kosovo Judicial and Prosecutorial Council into the newly-created KJCS without due regard to their skills. KJCS’s broad responsibilities to assist the KJC and manage important aspects of court administration are not commensurate with its institutional capabilities. Several judicial leaders expressed concerns that the KJCS staff does not possess the desired level of expertise to discharge its broad responsibilities in court administration. The revision of KJCS’s organizational structure, foreseen under the draft Strategic Plan for Judiciary (Strategic Objective 2.5.1), is of critical importance in this regard and EROL must revisit this issue with the KJC as a matter of priority. The KJC’s membership with fulltime judges and the KJCS’s lack of skilled personnel undermines the absorptive capacity and ability of the KJC to devote time to EROL’s proposed activities aimed at strengthening the KJC. Moreover, EROL’s interventions could have achieved greater success if EROL were more engaged in pursuing its activities and establishing timelines for its delivery to the KJC, which by the nature of its composition admittedly is a passive institution. Despite EROL’s strong presence in all basic courts and its good working relationships with the KJCS, the lack of a more proactive approach by EROL, combined with EROL’s frequent staff changes, resulted in limited progress in effectively implementing the institutional strengthening assistance to the KJC. EROL’s assistance has been limited but beneficial in improving KPC’s organizational structure; developing an institutional strategic plan; and creating a public relations strategy. The adoption of the KPC’s strategic plan and the public relations strategy have been delayed due to the KPC’s leadership decisions which are beyond EROL’s control. The KPC was established in January 2011, in accordance with the Law on KPC, as an independent body responsible for recruiting and proposing for appointment, training, evaluating, disciplining, transferring, dismissing, and promoting prosecutors and for administrating the prosecution offices throughout Kosovo. Unlike the KJC, the KPC does not have a separate formal secretariat because its administrative support is provided by the staff of the Office of the Chief State Prosecutor. The following information draws on the ET’s content analysis of KIIs, EROL documents, and PMP results. As a nascent institution of the justice system, KPC had only 3 staff members when it was created in 2011. Pending the entry into force on the new law on KPC and State Prosecutor, EROL offered considerable technical assistance in assignment and transfer of prosecutors, transfer of cases to new prosecution offices, and the drafting of internal regulations on the operation of the KPC. EROL provided hands-on technical assistance in the drafting process, and facilitated deliberations of regulations by the relevant working groups and the Normative Acts Committee prior to the approval to the KPC. Through several KIIs, the ET found that EROL’s assistance was crucial in preparing the regulations on prosecutorial transfers, and prosecutorial appointments and regulations on transfer of cases required by the introduction of the new prosecutorial system effective as of January 1, 2013. The fact that the transfer of over 120 prosecutors within the justice system was managed smoothly and without appeals is also attributed to the clarity and quality of the regulations drafted through EROL’s advice. EROL’s assistance in drafting the 3 Codes of Conduct and Professional Conduct (Codes of Ethics) – one for the members of the KPC, the second for prosecutors, and the third for prosecutorial support staff – and the accompanying trainings of prosecutors on rules of ethics were positively assessed by KIs. As a result of the 8 EROL-supported Train the Trainers (ToT) workshop on the new Codes of Ethics, 7 prosecutors are now qualified to deliver trainings based on the curriculum of the KJI in professional ethics. Prior to the Code of Ethics, 20 prosecutors were subject to the disciplinary proceedings, yet after its adoption and accompanying training programs, only 4 prosecutors have been subjected to disciplinary proceedings. While it is difficult to link the improved regulatory framework on ethics with decreasing numbers of disciplinary proceedings against prosecutors, EROL’s drafting assistance and the training program were positively assessed. More recently, EROL cooperated with the EU-funded project supporting KPC in drafting amendments to the KPC‘s regulation on disciplinary committee and the regulation on performance evaluations of prosecutors. EROL’s technical assistance to KPC in developing a 2-year institutional strategic plan and improving its internal communication, public communication, and outreach are considered by the KIs to be of good quality. However, the ET found that as of March 2014 the strategic plan had yet to be formally approved, allegedly due to the Chief Prosecutor’s decision not to move forward with a strategic plan pending the outcome of the dialogue between Serbia and Kosovo on the justice system. The ET also found that a number of resource materials, which were all developed through EROL’s technical assistance, including the Media Relations Guidelines, a Crisis Communication Guidebook, a Communication Strategy Plan, a Communication Strategy Implementation Plan, and an Administrative Instruction on Internal Communication have not been formally reviewed by the Normative Acts Committee before its consideration and adoption by the KPC. Although concerns were expressed about EROL’s frequent leadership and staff changes, KIIs revealed that EROL has been responsive and the expertise it offered to the KPC was of good quality. The ET found that despite the fact that the 2 key expected results, namely the strategic plan and the public relations strategy are not yet in place, this delay is associated with internal KPC leadership decisions, which are beyond EROL’s control. Based on the evidence, the ET was unable to establish convincingly the extent of EROL’s impact and benefit to the KPC since there are numerous international donors who provide similar assistance to the KPC including the Department of Justice/Office of Overseas Prosecutorial Development Assistance and Training (DOJ/OPDAT), European Rule of Law Mission in Kosovo (EULEX), EU, and German Society for International Cooperation (GIZ). In general terms, however, the ET found that EROL’s help has been both effective and beneficial and has led to improved KPC’s operations and an improved regulatory framework on prosecutorial ethics. As a result of EROL’s and other donors’ assistance, KPC’s capacities as a new institution have improved. For example, in 2013, KPC had 26 meetings during which it adopted approximately 20 regulations and 150 decisions. EROL’s assistance to MoJ has been moderate but beneficial in improving legal drafting skills and enhancing the capacity of the Department for International Legal Cooperation (DILC). The assistance in drafting primary legislation and strengthening the capacity of MoJ to represent the GoK in court proceedings has not been delivered due to internal decisions of MoJ which are beyond EROL’s control. Drawing on content analysis of KIIs with officials from the justice institutions, content analysis of EROL’s programmatic documents, and KIIs with EROL staff, the ET found the following: EROL supported a number of legal drafting initiatives at the MoJ through participation in the working group on drafting amendments to the judicial package laws, a new draft Law on Minor Offenses, draft law on State Advocacy Office, and a concept paper for the Justice Academy. With the exception of the Law on State Advocacy Office, other EROL-supported drafts of primary legislation are not yet approved. Amendments to the judicial packages law are stopped, pending the outcome of the Normalization Agreement between Serbia and Kosovo. Completion of the Draft Law on Minor Offenses and Draft Law on Justice Academy within MoJ are still ongoing. EROL’s further contribution in revising and amending judicial package laws is dependent on the MoJ’s decision to resume the work of pertinent GoK working groups. In assisting DILC, EROL worked collaboratively with the EU Twinning Project launched in 2012. EROL focused on the DILC's capacity building assistance in the area of international legal assistance in civil matters, whereas the EU focused on supporting DILC in criminal and child abduction matters. EROL’s assistance to the DILC’s operations, training DILC lawyers, and developing training manuals is now successfully completed. DILC’s database, launched in October 2013 with EROL’s support and containing approximately 15,000 cases of international legal assistance, received high praise from local and international stakeholders. The ET also found that the quality of training on international legal assistance that EROL delivered to key staff members, as well 9 as the respective manual, were qualitative and beneficial. In overall terms, several KIIs revealed that the database, associated training, and the training manual have significantly improved the work flow and productivity at all levels and have been very beneficial to the DILC. Due to the absence of a State Advocate, EROL’s program was limited to the creation of a case tracking database to manage the case records of the Litigation Unit, which will transform into the State Advocacy Office (SAO) and be elevated to the status of an independent agency of the MoJ. Given the MoJ’s lack of interest to pursue capacity building assistance of the Litigation Unit pending the appointment of a State Advocate, EROL has not been able to deliver additional assistance. EROL’s assistance has been beneficial and enabled the OP to better fulfill its responsibilities in the justice sector through clarifying the procedures on the appointment and re-appointment of judges and prosecutors; enhancing legal drafting skills of the OP’s legal staff; and improving its communication and outreach. Drawing on content analysis of program documents, KIIs, and PMP results, the ET found the following: Although OP responsibilities were not significantly changed through the judicial package laws, EROL was the only international project providing assistance aimed at clarifying the OP’s responsibilities in relation to the justice institutions, training OP legal advisors, and improving its communication and outreach. EROL’s international and local advisors offered assistance in drafting 2 important regulations for the OP on the appointment and re-appointment of judges, and on appointment and re-appointment of prosecutors. EROL’s staff assisted with the preparation of the first drafts of such regulations which were circulated for input of the presidential administration staff and finalized and entered in force in August 2011. More recently, EROL provided short-term technical assistance in drafting a regulation on the examination of laws prior to the presidential signing. However, the ET found that the plan to adopt this EROL-assisted draft regulation was abandoned as the legal advisors to the OP deemed that such regulation would limit their discretion in assessing the constitutionality of laws and providing advice to the President. Instead, other resource materials and guidelines drafted with EROL’s support are being used as a Guidebook on Promulgation of Laws by the OP legal staff. The ET learned that legal advisors found the overall content of a 6-month course on legal English, designed by EROL to enhance legal advisors skills for conducting legal research, to be beneficial. It was aimed at helping legal advisors improve their English skills and identify databases for conducting legal research. EROL also conducted legal drafting courses which included a ToT program and, as result, 4 members of the OP’s legal staff are prepared to train on the topic. EROL’s technical assistance on upgrading the website of the OP is ongoing. Through KIIs with justice institutions personnel, the ET learned that the regulations on judicial and prosecutorial appointments were drafted after extensive discussions that EROL facilitated between legal advisors of OP, KJC, and representatives of other GoK’s institutions. The regulations codified the de facto practices adopted by the OP in appointing judges and prosecutors, and streamlined the relevant procedures between the involved institutions. As a result, written rules have formalized the relationship between the KJC, KPC, and OP, and have generally improved the coordination and cooperation between justice institutions. Various capacity building events, such as workshops, trainings and conferences on legal drafting and legal analysis attended by OP legal advisors were commended by the legal advisors as a useful way of enhancing coordination and understanding between lawyers and legal advisors from key justice institutions such as the KPC, KJC, and MoJ. EROL’s program assistance to the KCC has been limited but beneficial in improving its internal operations, increasing the legal reasoning and judgment drafting skills of the legal advisors, and enhancing the transparency of court activity. EROL’s assistance in revising its Rules of Procedures for the KCC through the engagement of a US Federal Judge in 2 workshops was deemed as an effective way of addressing procedural issues identified during the activity of the KCC and led to improvement of the Rules of Procedures, which constitute a fundamental act for KCC’s operation. Furthermore, EROL’s trainings and workshops on the role of legal advisors in conducting legal research, legal writing, and reasoning offered through short-term technical advisors were highly appreciated by the court leadership and legal advisors due to the interactive working group sessions based on 10 case study methods. To enhance the transparency of KCC’s activities, EROL engaged a short-term technical advisor who worked with the legal advisors and the IT staff to upgrade the KCC’s website by enhancing its functionality and usability by the legal professionals. The upgraded web-based database will have basic and advanced search capabilities and will allow legal researchers to identify and download KCC case law. At the request of the KCC, EROL helped KCC organize 2 conferences: one on human rights and the other on self￾executing nature of KCC decisions. Through several KIIs and on-site visits, the ET found that KCC has received significant technical assistance by international donors including other USAID-funded projects. KCC is widely considered to be one of the most effective and respected institutions in Kosovo. It processes between 200 and 350 cases a year, with an impressive annual case resolution rate of about 98%, and no pending cases older than 6 months. Although, KCC is currently heavily supported by international organizations which makes it difficult to establish correlation between EROL’s programmatic assistance and KCC’s success during its first 5 years, evidence suggests that EROL’ assistance to KCC, albeit moderate, was beneficial. EROL’s assistance to KJI focused on the skills of the Judiciary and Prosecutorial sectors with technical assistance including training courses, ToT, judicial conferences, study tours, and website enhancement. EROL’s program assistance to KJI targets Objective 3 through improving the professionalism of justice system actors. According to desk review of information and KIIs, KJI’s faculty assumed sole responsibility for delivering a number of courses for the Initial Legal Education Program (ILEP) and the Continuing Legal Education Program (CLEP) that were co-developed and previously delivered with EROL’s assistance. The courses included Judicial Ethics, Case Management for Judges, Legal Reasoning and Writing for Judges, Communication Skills for Judges, Judicial Trial Practice, and Legal Research for Judges In cooperation with EROL, KJI designed and implemented curriculum and instructional materials in building and sustaining an independent and transparent judiciary in Kosovo. Two programs anchor KJI’s core curriculum – ILEP for judicial and prosecutorial candidates training and CLEP which provides continuing legal education for judges, prosecutors, and other judicial professionals. In addition, EROL joined instructional forces with 2 judges who are KJI adjunct faculty. Together with a judge from Mitrovica and a judge from the Pristina Basic Court at the KJI, EROL delivered a civil law Judicial Trial Practice program for judges. According to the evaluations tabulated by KJI, the interactive educational methodology was well-received by the judge participants. KIIs indicated that more interactive education was requested and is needed. Further, EROL facilitated the transition of teaching responsibilities to KJI’s instructors by meeting with the judges to discuss the curriculum and teaching methodologies. In response to weaknesses in civil proceedings identified by the Organization for Security and Co-Operation in Europe (OSCE), EROL created additional case studies that have been added to the curriculum. The case studies provide KJI with a model case-study curriculum designed to foster critical thinking skills in judges. In addition to course development and training, EROL advanced KJI’s library of secondary legal materials, with support to KJI’s Bench Book Initiative. KJI, in collaboration with the KJC and EROL, assembled 2 working groups of experienced judges who will assume primary drafting responsibilities. EROL conducted initial workshops for the judges who were largely unfamiliar with the bench book as a procedural guide assisting judges with disposition of cases. The judges/authors learned how to conduct substantive legal research, and draft, edit, and schedule a production plan, reviewing bench book models from other countries. EROL engaged a US judge with experience teaching for the Federal Judicial Center (FJC) in Washington, DC; the advisor began working with KJI to oversee the drafting effort, establish an editorial review process, and provide technical assistance to ensure that the judges follow best practices in bench book composition. Reviewed data indicate that KJI, with support from EROL, officially introduced its database/web portal. The database application and web portal are an advanced combination of internal repositories for human resources, finance and procurement records and reports, with a public-facing website featuring training course materials and registration capabilities. Resources such as the bench books and all KJI courses and course materials are accessible to Kosovo’s legal community via the new KJI web site as well. 11 EROL supported KJI’s management and IT staff by coordinating the vendor’s work who met the database/web portal specifications. According to KIIs and reviewed data, the database/web portal is advancement for KJI in the use of technology and information management. EROL works with KJI on its website communication strategies, supporting the sustainability and capacity of KJI as a judicial education organization. EROL also provided assistance to KJI in the development of the e-Learning Initiative. Further, data indicates that EROL works with KJI in examining how current and future modules may be adapted to a distance learning format. According to the ET’s interviews with KIs, adaptation of the EROL/KJI judicial reasoning and writing training module into intermediate and advanced programs will be the centerpiece of the e-Learning platform rollout, taking place through 2014. Gathered data indicates that the professional development of KJI’s director and program staff were promoted with EROL’s support for a visit to FJC in Washington DC. The director’s trip was primarily funded by the US Department of Commerce to conduct other business; EROL coordinated an orientation to the FJC’s programming and resources as a result. FGDs and KIIs indicate that KJI’s training abilities have strengthened through the implementation of EROL’s initiatives. Through its continuing legal education program, KJI provided specific training courses to sitting judges and prosecutors. During the past year, it held 44 criminal law and 25 civil law trainings. It launched ILEP, which is a 15-month program geared to prepare incoming judges and prosecutors for their new professions. EROL’s efforts sustain the goal to empower KJI to operate in a professional and effective manner so that it will be able to provide quality trainings and educational materials to justice system professionals. Interviews, desk review, and program documents indicate that EROL’s activities to support KJI include successful drafting and delivery of training courses, developing ToT programs taught by KJI instructors, organizing judicial conferences held on a regular basis, and sending KJI upper management to international conferences. According to KIIs and reviewed data, coordination strategy with KJI was hindered as a result of the MoJ’s proposed but undecided transformation of KJI into an Academy of Justice. At present, a presumption is being made that the decision on transformation will be made in 2014. As a result, EROL will proceed to work with KJI and other justice institutions (KJC and KPC), to develop the National Legal Education Strategy in 2014. KIIs indicate that neither KBA nor KJI actually support the plan for transformation. The thought is that it will destroy KJI as an institution and that the change will hinder, if not destroy, needed stability. EROL has refurbished 11 courts in its Model Court Program, and engaged in activities that improve court operations, increase competency of court staff in court administration, and enhance citizens’ understanding of and respect for the courts. As a result of site visits, observations of 9 courts, KIIs with officials from the justice institutions, EROL staff, judges and court administrators, 3 FGDs with 35 court staff representing the Court of Appeals and various Basic and Branch courts, 3 mini-surveys with the same participants as those in the 3 FGDs, and EROL’s programmatic documents, the ET has found the following: Following JSP, which initiated the Model Court Program and refurbished 10 courts, EROL continued to refurbish, in varying degrees, 11 courts, and is currently refurbishing 5 others. With the addition of 9 courts refurbished or currently under construction by KJC, all courts in Kosovo will be refurbished by the end of 2014, including: all 7 Basic Courts, 20 Branch Courts, and the Court of Appeals as the first Model Court in November 2012, and the Supreme Court which was refurbished in 2013.2 2 EROL will refurbish by the end of 2014: Supreme Court; Court of Appeals; Rahovec Branch Court; Suharekë Branch Court; Vitia Branch Court; Istog Branch Court; Lipjan Branch Court; Gjilan Basic Court; Dragash Branch Court; Gllogovc Branch Court; Kacanik Branch court; Decan Branch Court; Podujeva Branch Court; Klina Branch Court; Malishevë Branch Court; and Shterpce Branch Court. JSP refurbished: Pristina Municipal Court; Pristina Economic Court; Peja District Court; Prizren District Court; Gjakova District Court; Mitrovica District Court; Ferizaj District Court; Gjilan Municipal Court (different building from the Basic Court); Kamenica Municipal Court; and Skenderaj Municipal Court. KJC built or refurbished: Gjilan Basic Court; Podujeva Municipal Court; Malishevë Municipal Court; Klina Municipal Court; Kacanik Municipal Court; Decan Municipal Court; Shterpce Municipal Court; Vushtrri Branch Court (still under construction); and Ferizaj Basic Court (still under construction). 12 The courts have been refurbished within the footprint of the existing court buildings and, to some degree, in a consistent fashion as they have adhered to the Model Court Standards formulated by EROL, in consultation with KJC, which have focused on: increasing public access to court staff; enhancing the transparency of court operations by locating the central records management offices immediately inside the court and visible to the public; improving the public’s knowledge of the courts by providing public information counters and brochures at the court’s entrance; and securing judges from unnecessary public interruptions by relocating their offices. Many judges pointed out that the Model Court Program is successful because “it is the type of change that is hard to be against and transforms the court operations for the better.” Furthermore, the Model Court Program “changed the culture of working in the justice system,” particularly since the refurbishment “removed justice from the dark offices of judges and staff.” One key informant (KI) emphatically stated that “no Kosovar believed the courts could be so modernized.” During the 3 FGDs with court staff, and from the responses to the 3 mini-surveys, the ET learned that the primary beneficiaries of the Model Court Program, i.e. judges, court staff, and the public, were greatly pleased with the quality of the court refurbishments, noting that it represented the value that the GoK now places on justice services. While the ET did not interview the public, court staff mentioned during the 3 FGDs that the public was more satisfied with court services than previously, which was evident from the citizens’ demeanor at court offices. FGDs revealed that citizens were also pleased with their increased understanding of court operations, explained by court staff and enhanced by the public brochures at the courts’ entrances. Some participants in the FGDs stated that the refurbishment demonstrated to the public the seriousness of the court staff’s work while increasing the transparency of court procedures. One participant in the FGDs demonstrated the importance of public service by commenting “I learned to keep a smile on my face even though I was angry with the citizen.” Another common complaint that the ET heard, which was not lodged against EROL but against the KJC, was the lack of audio recording equipment in most of the refurbished courts, although a room under the Model Court Standards has been specifically designated for audio equipment. Many court staff who participated in the 3 FGDs believed there must be at least one court in each district that has audio equipment allowing for simultaneous translations as “judges often get annoyed with consecutive translations.” In all 3 FGDs, the ET heard enthusiastic responses from court personnel who genuinely expressed their gratitude for the Model Court Program as it inspired them to enhance their work ethic and productivity by virtue of the physical reconstruction of their offices. “Having court staff work in glass-enclosed rooms causes us to work much harder and with a much greater team effort,” stated a participant in a FGD. His statement was echoed by many others. Although the ET was informed during a KII at the KJC that EROL’s Model Court Standards are de rigueur for all KJC court reconstruction projects, the ET learned of a notable exception to KJC’s strict adherence to the Model Court Standards during one of its site visits. The KJC’s refurbishment of the Malisheva Branch Court building was completed in 2011 without adherence to the Model Court Standards; EROL will be renovating the building this year so that the Model Court Standards can be met. KJC also refurbished Shterpce Branch Court in 2013 – which EROL was not informed of until one week prior to its completion. While it is unclear whether these examples are exceptions, they, nevertheless, question the effectiveness of KJC’s collaboration with EROL and the sustainability of EROL’s Model Court Program, if, after several years of working with JSP and EROL, KJC does not inform EROL of its work and neglects Model Court Standards in even one court refurbishment project. As a complement to court refurbishment, EROL supported court operations by developing a court registry database; training court staff in its use; training judges, court administrators, and staff on relevant court administration, management, and public outreach topics; and providing on-the-job technical assistance in the form of EROL liaisons assigned to each court. According to EROL’s documents and numerous KIIs, EROL created a case registry database on a Web platform as a means of facilitating the transfer of cases from courts operating under the former court structure to the new court structure mandated by the Law on the Courts. The case registry database tracks fundamental data generated through the manual court registry books, such as the case name, type of case, names of parties, date, number of hearings, and case status, and is operational in all courts of Kosovo. The database has been designed to ensure sustainability so that data can be ultimately migrated into the KJC’s new Information and 13 Communication Technology/CMIS Project to be developed with the financial support of the Government of Norway and implemented by the International Management Group in collaboration with KJC. The court registry database served as the mainstay of the approximately 240,000 court cases that were pending, and tracked the transfer of cases as a result of the court reorganization. Without a doubt, many of the court staff who participated in the 3 FGDs and 3 mini-surveys were pleased with the case registry database. Several court staff praised EROL’s creation of the database, stating in essence “we are now able to respond to the public by being able to quickly extract data.” It was not uncommon to hear comments such as “EROL’s database allows us to work more efficiently and diligently,” leading some participants to recommend that “the law should be changed to eliminate the need for hardcopy registries.” The court registry database also provides sufficient data to generate statistics for expediting case processing, streamlining workflow, managing judicial caseloads, and identifying, to a degree, causes of case backlog, allowing for workload analysis and backlog reduction strategies to be developed. Working closely with KJC and the courts, through liaison staff, EROL assisted in preparing a National Backlog Reduction Strategy and is developing individualized court action plans for each court, and, within each court, individualized action plans for each judge. From the information gathered from KIIs with KJC staff and judges, it appears that there has yet to be a reduction in case backlog of 240,000 cases, although the positive outcome of EROL’s backlog reduction activities is that the backlog has not been increased. This is particularly laudable as there are a significant number of judicial vacancies. EROL provided considerable training on computer skills and on the proper usage and application of the database, and training on New Case Registry to court staff at the Supreme Court, Court of Appeals, all 7 Basic Courts and numerous Branch Courts, which enabled court staff to assume the responsibility of the court registry database in January 2014. Prior to January 2014, EROL’s 26 temporary database entry assistants were primarily responsible for the inclusion of data in the court registry database. The responsibility of inputting data and maintaining the court registry database was transferred to court staff in January 2014. EROL has also provided training on other topics related to modern court administration and management practices for Presiding Judges and Supervisory Judges, court administrators, and specialized court staff, such as court statistic officers, IT officers, and case management office heads. An instrumental course of the Model Court Program was given on Change Management Leadership to, at the time, Presiding Judges and Court Administrators so they may understand and implement their new managerial responsibilities under the court re-organization. In addition to introducing the concept of change management, EROL also introduced the concepts of customer service, effective communication and team building in its training of court staff, which were essential, given the reconfiguration and transparency of court staff offices. Additionally, EROL provided training to statisticians on their role in generating reports and the importance of such reports to modern court administration, EROL’s training as an activity of the Model Court Program was the subject of the 3 FGDs and 3 mini-surveys with court staff as well as numerous KIIs, and is reported in response to EQ 4, which pertains to capacity building. EROL also supported public information officers by improving public outreach and uniting courts with the surrounding communities, as well as demonstrating not only to the public but to the court staff as well that the courts exist to serve the public. According to the 3 FGDs and 3 mini-surveys, which included public information officers as participants, EROL’s assistance “helped develop public relations for the courts which is a new concept for Kosovo.” Before EROL, “journalists received information about the courts in the cafeteria; today, they are receiving it directly from the courts and disseminating it to the public.” EROL’s court user survey conducted in 2013 demonstrates that 91% of respondents believe they were treated courteously by court staff, while 85% of respondents were provided information and answers to their questions by court staff in a timely manner. Interestingly, many participants in the 3 FGDs stated that while there has been an increased awareness of the court’s role to serve citizens, more needs to be done. “EROL’s engagement is not enough to eliminate citizens’ perceptions of the courts. KJCS must step in and take responsibility. EROL should transfer its knowledge to KJCS’ staff to ensure sustainability.” 14 Several of EROL’s trainings consisted of ToT workshops to ensure the continuation of training by experienced court staff and the sustainability of the training courses. Under Objective 4, EROL provided assistance through a small grants program to improve citizens’ engagement in justice sector reforms and increase citizens’ understanding of these reforms. Two NGOs were beneficiaries of the strategic activities fund, managed through the USAID Forward Program dedicated to Civil Society Organizations (CSOs). The Balkan Investigative Reporting Network (BIRN) implemented a 1-year nationwide public awareness campaign designed to increase citizens’ understanding of the justice system, whereas the Advocacy & Training Resource Center (ATRC) managed a sub-grant program supporting CSOs with small projects of court monitoring and public awareness aimed at improving the delivery of justice and increasing transparency and fairness among justice institutions in Kosovo. EROL’s assistance was limited to providing technical support to ensure that the CSOs managed the grants in compliance with USAID requirements. Through Objective 4, EROL supported a public outreach campaign in cooperation with various justice institutions such as KJC, KPC, and court Public Information Officers (PIO), to enhance citizen understanding of the implementation of court re-structuring and reaching out to minority communities. Drawing from on-site observations, KIIs, FGDs, and review of EROL’ programmatic documents, the ET found the following: EROL launched a multi-faceted communication and outreach campaign to improve relationships and understanding between courts and media, and increase citizen awareness of the justice system. EROL conducted a ToT workshop for the KPC, KCC, and KJC spokespersons and other public information specialists working in the justice system. As a result, they are now certified trainers and may deliver and train junior or less experienced EROL personnel, and provide workshops with journalists who report on the justice sector in Peja, Prizren, Gjilan, Pristina and Cagllavice (with Serbian journalists). The ET found the workshops and roundtables with journalists had a positive impact in improving the knowledge of judicial/prosecutorial processes, as well as legal terminology, and enhanced skills for investigative reporting on the justice system, ethics for journalists, and new media best practices. EROL launched a national public awareness campaign on court restructuring in 5 local languages, including sign language. The campaign, co-organized with KJC, lasted for 27 days in TV Public Services Announcements (PSAs), 3 radio PSAs, 3 website banners, and 3 newspapers banners. EROL’s court and public outreach in Year 3 included distribution of “This is Your Court” print brochures explaining the new court re-structuring and prosecutorial system in Kosovo. In addition, EROL supported several roundtables with participation of PIOs of Basic Courts and journalists reporting on the justice system. Through the KIIs and FGDs with journalists from Pristina, Peja, and Prizren, the ET found that the workshops and roundtables increased journalists’ awareness of the court system, and helped them better understand, document, and report court proceedings. The ET also found in its FGDs with journalists that, despite their trainings on developing professional relations with judges and prosecutors, most journalists believe that “judges and prosecutors have not changed their mindset and continue to be close to journalists.” EROL’s minority outreach activities were targeted to under-represented communities such as Turkish, Serbian, and Roma, to communicate information about careers in law in the courts and legal practice, examination requirements for attorney licensing, and reciprocity of law degrees from countries outside of Kosovo. In cooperation with KJC, EROL accomplished this with a wide distribution of a set of brochures, addressing each of these topics in easy-to-understand vernacular. 4.1.2 Conclusions  EROL’s assistance to justice institutions has been instrumental in ensuring a smooth and timely transition into the new court system in January 2013.  Although the management and operations of the KJC have achieved moderate progress due to EROL’s assistance, several EROL interventions have achieved limited success to strengthen KJC’s capacity in 15 exercising its main responsibilities to develop and implement policies for a more efficient and effective administration of the courts, particularly in reducing case backlog, improving caseload management, evaluating judges’ performance, enforcing judicial discipline, and institutional strategic planning.  EROL’s assistance has helped KPC achieve moderate progress in fulfilling its responsibilities to develop and implement policies for the prosecutorial system, particularly in developing an institutional strategic plan and public relations strategy.  EROL’s program assistance to the MoJ has been limited yet beneficial in enhancing the capacity of the DILC to handle requests for international legal cooperation more effectively. The assistance related to revision of judicial package laws and establishing the SAO has produced no results yet due to internal decisions of the MoJ which are beyond EROL’s control.  While limited, EROL’s program assistance to the OP has strengthened the collaboration among the OP, KJC, and KPC, and enhanced the legal drafting and research skills of the legal staff.  KJI is strengthened and has benefitted by the implementation of EROL’s assistance. By providing support to KJI, EROL has met Objective 3 and has advanced ROL objectives. Further, EROL’s assistance to KJI is sustainable through education, trainings, curriculum, ToT, as well as improved website development and portal.  The courts of Kosovo have been strengthened and benefited from EROL’s Model Court Program by the improvement of court operations, the advancement of the technical skills of court staff, the upgrading of judges’ and court staff’s service to the public, the enhancement of citizens’ interaction with court staff, and the increase in citizens’ knowledge of and respect for justice services.  EROL has contributed to implementing the advanced ROL objectives by improving the capacity of the justice institutions, and enhancing the skills of justice sector professionals, particularly court personnel, through its Model Court Program, resulting in a more efficient, transparent, independent, and accountable justice sector. EROL has also helped improve civil society’s ability to conduct more effective oversight and advocacy by increasing citizen awareness and its role in ensuring the delivery of justice. However, the impact of minority outreach is difficult to measure due to the current political situation in the north.  EROL’s interventions with the KJI, OP, KCC, and KPC, as well as its robust Model Court Program, have met its 4 stated objectives. EROL’s interventions with the KJC, and the MoJ, have not fully met its 4 stated objectives and have achieved uneven success due to their limited capacity to absorb international technical assistance, the delay in establishing the SAO, and the overabundance of international donors with individual missions.  The ET cannot conclude on the sustainability of EROL’s activities since this is dependent upon the commitment of the GoK and Kosovar justice institutions to maintain and advance EROL’s achievements. 4.2 QUESTION 2: How have the Justice Institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of EROL’s and KLP’s activities? To what extent have the programs supported/assisted institutions to increase and strengthen gender diversity? Why or why not? 4.2.1 Findings While EROL did not specifically focus on strengthening gender diversity in its activities with the justice institutions, in the Model Court Program, and with its citizen awareness campaign, it supported the creation of the Forum of Women Judges and Prosecutors. In collaboration with the US Embassy and USAID, EROL met with women judges and prosecutors to support the creation of the Forum of Women Judges and Prosecutors (Forum) in 2013. After several meetings organized by EROL, and with the assistance of the US Embassy in drafting the Regulation for the Forum, an assembly of women judges and prosecutors met on December 21, 2013, and adopted the Regulation. The Forum is open only to women and includes not only judges and prosecutors, but also women professionals who work as legal officers in the courts and prosecution offices, and graduates from accredited law schools. The Forum’s vision is to improve the role of women in the justice system by cultivating gender and ethnic equality. The Forum’s purpose and activities are, inter alia, to empower the presence of women in the justice system, specifically judges and prosecutors, through a dignified, unified voice and vision; promote equality and respect for all; raise public awareness of the contributions of women judges and prosecutors to the justice system; increase citizens’ awareness of the impact the justice system has on gender equality; encourage the 16 professional development of Forum members; support initiatives that improve legislation impacting gender issues; and support organizations and causes, which advance the status and progress of women in the profession. 4.2.2 Conclusions  The Forum for Women Judges and Prosecutors has the potential of being a powerful voice in increasing and strengthening gender diversity in the justice system. 4.3 QUESTION 3: What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on KJI, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar)? The impact of the programs on secondary legislation will also be important. 4.3.1 Findings Justice institutions received significant program assistance from EROL to meet important timelines for implementation of the judicial package laws related to court restructuring. As previously stated, the new LOC, which entered into force in 2011, called for a re-organization of Kosovo’s court system starting from January 1, 2013, and introduced a new three-tier court system with: 7 Basic Courts and 20 Branch Courts serving as court of first instance; one Court of Appeals; and a Supreme Court as the highest judicial body in Kosovo. The creation of a new Court of Appeals and the concomitant changes of jurisdiction necessitated appointments and re-appointments of judges and court personnel, and transfer of cases to the new competent basic courts and branches in order to ensure uninterrupted court services throughout Kosovo. In the first 2 years of the program, EROL supported the KJC, KJCS and the courts to compile an Implementation Plan for the LOC and Law on Kosovo Judicial Council. The LOC assigned to the KJC the responsibility to prepare and publish an implementation plan by December 1, 2011. The KJC adopted the plan in November 2011 and established a steering committee, an ad hoc implementation committee, and 6 thematic groups to work on: 1) human resources (judges and court administrative staff); 2) physical resources and materials; and 3) court administration (case files and registries) and logistical changes needed to ensure the transition into the new court structure. Through EROL’s assistance, KJC adopted key regulations on the internal organization and court restructuring in line with plans for the implementation of the court reforms required by the judicial package laws. The Implementation Plan for the Law on the Courts formalized the planning process and set out clear timelines for the implementation actions to ensure a smooth transition according to entry into force of the Law on Court in January 2013. The KJC, KJCS and 6 implementation working groups received assistance from EROL throughout 2012 to prepare for implementation of the new LOC. EROL also supported the KJC to create an inventory of about 240,000 cases pending in Kosovo courts operating under the previous LOC, and assisted with the physical transfer of about 9,600 transferred cases from the court of previous jurisdiction to the new one. The transition required a high level of effort from EROL’s personnel, including 26 Data Entry Assistants and 7 Regional Court Liaisons, which facilitated the entry of data from manually held case registers into a case tracking database, and provided day-to-day help to court staff. As a result of EROL’s assistance, the process of inventorying and transferring cases was completed without delays and all courts were able to transition and operate under the new legal framework without significant challenges on January 1, 2013. The newly-created Court of Appeals was able to start operating in compliance with the new LOC without undue delays and held its first hearing on January 11, 2013. EROL’s assistance to KPC has been much more limited. Pending the entry into force of the new Law on KPC and State Prosecutor, EROL offered considerable technical assistance in the assignment and transfer of prosecutors, transfer of cases to new prosecution offices, and the drafting of internal regulations on the operation of the KPC. EROL’s assistance was beneficial in preparing the regulations on prosecutorial transfers, prosecutorial appointments, and regulation on transfer of cases required by the introduction of the new prosecutorial system effective as of January 1, 2013. The fact that the transfer of over 120 prosecutors within the system was managed smoothly and without appeals is also attributed to the clarity and quality of the regulations drafted through EROL’s advice. 17 The ET found that the new judicial package laws did not alter the responsibilities of the MoJ and KCC and thus did not require EROL’s assistance to meet the timelines for court restructuring. However, in 2011, EROL facilitated a series of roundtable discussions with legal advisors of KJC and other justice institutions to enhance understanding and cooperation between the KJC and the OP with regard to the appointment and re￾appointment of judges, which have resulted in improved coordination of justice-related institutions. Following a series of roundtable discussions, EROL assisted the OP, KJC, and KPC in drafting regulations and clarifying procedures and criteria for the appointment, transfer and dismissal of judges and prosecutors, subsequently adopted in August 2011 prior to the entry force of the LOC. The ET, through KIIs with senior justice institutions officials, on-site observations, and FGDs, found that EROL’s assistance to both the KJC and KPC was crucial in meeting the timelines set forth in the new laws. Many court presidents, supervising judges and court administrators from all court levels, stated that without the guidance and hands-on assistance from EROL’s staff, the transition triggered by the new judicial package laws would have been very challenging for Kosovo’s justice system. Despite improvements achieved through EROL’s assistance, the capacity of the KJC and KPC to become effective in exercising prescribed responsibilities under the judicial package laws remains weak. Beyond the transition phase, the contribution of EROL in drafting secondary legislation related to KJC’s responsibilities under the judicial package laws has not been significant. Given that EROL’s assistance to the OP, MoJ, and KCC was moderate, as discussed in response to EQ 3, the evaluation focused on the impact of EROL’s assistance on the current capacity of the KJC and KPC to exercise their responsibilities under the new judicial package laws, and EROL’s impact on secondary legislation. Relative to the KJC, EROL’s assistance to the KPC has been much more limited due to the division of responsibilities between the EU-funded project supporting the KPC and EROL by means of the Memorandum of Understanding (MoU), discussed in response to EQ 5 and the fact that DOJ/OPDAT is also assisting the KPC. In the last 3 years, EROL has assisted the KPC in drafting several regulations, including a regulation on the structure and administrative support of the Office of Chief Prosecutor, Regulation on the Activity of KPC, Regulation on the Appointment and Transfers of Prosecutors, Regulation on the Disciplinary Committee of the KPC and Regulation on Prosecutorial Performance Evaluation Unit, Regulation on the Functioning and Activity of KPC and, more recently, on the Regulation of the Nomination of the Chief State Prosecutor, which was adopted in March pending the retirement of the current Chief State Prosecutor in June 2014. In 2014, KPC is planning to adopt a regulation on the classification and qualification of documents in the Prosecution Office of Kosovo. Through group discussions with the EROL team and KIIs with justice institutions’ officials, the ET learned that despite its short life and the lack of a fully-fledged Secretariat, the KPC has achieved progress in meeting the timelines for exercising its responsibilities. However, the KPC’s capacity in the areas of prosecutorial performance and discipline remains weak and this is recognized by the KPC leadership as well. As the governing body of Kosovo’s judiciary, the KJC has broad powers and responsibilities in administering the courts, including the appointment, promotion and dismissal of judges; managing and overseeing the judicial budget; determining policies, standards and instructions related to the training of judges and court personnel; providing and publishing information and statistical data on the judicial system; evaluating judicial performance; and enforcing judicial discipline. Under the LOC, Law on KJC, Criminal Code and Criminal Procedure Code, KJC is required to issue more than 30 pieces of secondary legislation in the form of regulations, instructions, and decisions to further detail the legal provisions of its areas of responsibilities.3 Through the assistance of EROL and other donors, KJC has adopted important regulations to fulfill its obligations under the new laws and codes, including regulation of internal organization of the courts, appointment of judges, performance assessment of judges, and the Disciplinary Committee’s performance. EROL contributed to the drafting of the Regulation for the Internal Organization of the Court, which entered into force on January 4, 2013. This regulation is considered to be the most important sub-legal act adopted by the KJC, setting forth the rules of the internal organization of the Basic Courts, Court of Appeals, and the Supreme Court. The codes of ethics for the KJC and court staff drafted with EROL’s expertise, and adopted by the KJC in 2012, were praised by the KJC leadership but the ET was not able to gather evidence to assess the impact of the codes of ethics on improved judicial ethics. A 3 See ANNEX F for a detailed list of the KJC regulations and their status as of 31 March 2014. 18 number of EROL-supported secondary legislation, such as regulation procedures for the appointment and re￾appointment of judges, and the regulation on judicial performance have been adopted by KJC and entered into force. EROL’s staff expressed strong concerns that, despite the initial success of adopting key regulations which enabled transitions to the new court structure, KJC is lagging behind in completing the regulatory framework in its areas of responsibilities. The ET also found that as of March 2014, KJC has yet to adopt important regulations required by the judicial package laws, such as the regulations on: 1) the disciplinary proceedings against KJC members; 2) KJCS internal organization and functioning; 3) removal of court presidents and supervising judges; 4) ensuring representation and recruiting of judges from the non-majority communities; 5) appointment and performance assessments of lay judges; and 6) certifying translators and interpreters in criminal trials. Such delays have also been criticized in the most recent EU Progress Report on Kosovo of October 2013.4 KJC has not yet adopted the regulation on the organization of preparatory exams for judicial candidates as well as policies, standards, and directives regulating the training of judges, lay judges, and staff. From KIIs it appears that the delay in the adoption of such key regulations is the result of several factors, two of which have been previously mentioned, i.e. the passivity of the KJC and the lack of a proactive approach by EROL. The ET also found that EROL’s assistance with secondary legislation is affected by the fact that while EROL cooperates closely with the EU-funded Support the Kosovo Judicial/Prosecutorial Councils project, the EULEX embedded advisors and other international projects assisting the KJC and KPC, and while the MoU, discussed in response to EQ 5 divides responsibilities among the EU and EROL programs, there are many sub￾legal acts where the delineation of responsibilities among donors in offering technical assistance to KJC is not clear, resulting in assistance with some sub-legal acts not being sufficiently provided. While numerous donors also make it difficult to evaluate the degree of EROL’s assistance with secondary legislation, the ET collected sufficient evidence to suggest that a more proactive approach by EROL in drafting the required KJC’s secondary legislation would have yielded significant results. According to KIIs with local and international officials, KJC’s failure to adopt secondary legislation on the international organization and functioning of its Secretariat is undermining its ability to exercise responsibilities. EROL has offered its assistance in this respect by reviewing the Secretariat’s internal organization and providing recommendations to the KJC. Due to the Secretariat’s important role, adopting a regulation on the scope of activity and internal structures of the KJCS should be an immediate priority for the KJC. KIIs with KJC staff and international advisors suggested that establishing a Research and Policy Development Unit within the Secretariat may be a good step to enhance its capacities to provide expertise and support to the KJC. In addition to the secondary legislation assistance, EROL supported 2 important initiatives, including the Strategic Plan for Judiciary, which has been in preparation for over 18 months, and the documents on media and communication (previously discussed in response to EQ 1), which were submitted to the KJC in 2012 but had yet to be approved as of March 2014. Although case backlog continues to be one of the most critical issues faced by the Kosovo judiciary, with recent estimates indicating that approximately 120,000 cases are pending resolution in Kosovo courts, while 120,000 cases are pending execution, the ET learned that the KJC’s past efforts to reduce case backlog have been largely unsuccessful. EROL has provided important assistance to the KJC for a new National Backlog Reduction Strategy, which it adopted in September 2013 as the successor to the National Backlog Reduction Strategy of 2007. The impact of the previous National Backlog Reduction Strategy is hard to measure as case backlog continues to be generated in recent years due to judicial vacancies that the KJC has not been able to fill. The new National Backlog Reduction Strategy requires all courts to have individualized action plans to reduce case backlog. The ET learned that EROL is currently conducting a study to determine the primary 4 See page 10 of the section on judicial system of the EU Progress Report on Kosovo issued by the European Commission on 16 October. Notwithstanding Progress Report’s reference to the failure of the KJC to adopt the regulation on certifying translators in court proceedings, the KJC justifies the lack its lack of action due to the legal inconsistencies. As a matter of fact, although it is widely understood that KJC is the responsible authority to adopt rules on court personnel including professional translators in court proceedings the new Criminal Procedure Code of 2013 (Article 215, Paragraph 2) assigns the responsibility on the certification of translators and interpreters in criminal proceedings to the Ministry of Justice and not the KJC. 19 causes of case backlog in order to inform KJC’s policy-making in this area. A comprehensive study of the gravity of the problem and its main causes is a step in the right direction. Another key challenge for KJC is the evaluation of judges. In 2013, KJC conducted an evaluation for permanent appointment of 75 temporary term judges; 74 judges successfully passed the evaluation. Through KIIs with judges and international advisors, the ET learned that KJC’s current system is incomplete and the established criteria are not conducive to an objective, merit-based and credible evaluation of judicial performance. Although the KJC has conducted over 30 disciplinary proceedings against judges in the last 3 years, the ET learned that its disciplinary process is also criticized for being too slow and ineffective due to the lack of a regulation on the Office of Disciplinary Counsel and the inactivity of the KJC’s Disciplinary Committee. Drawing on content analysis of KIIs with officials from the justice institutions, content analysis of EROL’s programmatic documents, and KIIs with EROL staff, the ET identified several reasons for KJC’s slow progress in exercising its responsibilities under the judicial package laws, including the evaluation and discipline of judges. KJC is a new institution, has undergone several reforms before and after independence, and its members lack experience. Continuous reforms and transitions have not allowed necessary breathing space for KJC and its committees to develop long-term policies for the court system. As for KJCS, its capacities to assist KJC in administering the court system are very limited and its internal organization is not conducive to long-term strategic planning in court administration and judicial policies. Therefore, EROL’s targeted technical assistance to the committee structure has not yielded the expected results. Although most justice institutions related KJC’s current weak capacity to its own institutional failures and the multiple transitions it underwent since its inception, several senior officials of justice institutions emphasized the fact that EROL could have done more in increasing KJC’s capacity. As a senior official in the justice sector stated “Compared to other international projects helping the justice institutions, EROL is doing well but its performance is far from what is expected of a US government project of this size and strength”. There is also an over-abundance of un-coordinated donor assistance to KJC, which not only takes a lot of time and attention from KJC and KJCS staff but also makes it hard for projects like EROL to deliver the assistance. Due to KJCS’ limited ability to provide support to its committees, KJC is far from exercising its responsibilities under new laws and needs considerable assistance to fulfill its role as the governing body of the judiciary. 4.3.2 Conclusions  EROL’s assistance to KJC, KPC, courts and prosecution offices, and OP was crucial and enabled these justice institutions to meet the timelines of their respective responsibilities during the restructuring of the court system in accordance with the new LOC (Law No. 03/L199), Law on Kosovo Judicial Council (Law No. 03/L223), Law on State Prosecutor (Law No.03/L –225) and the Law on Kosovo Prosecutorial Council (Law No. 03/L-224).  Despite EROL’s assistance, KJC’s current capacities in exercising responsibilities under the relevant laws are weak and need to be strengthened significantly. EROL’s impact in drafting secondary legislation related to KJC’s responsibilities under the judicial package laws has not been significant.  EROL’s assistance to the KPC has moderately improved its capacity in exercising its responsibilities under the relevant laws, but KPC’s capacity in the areas of prosecutorial performance and discipline remains weak. 4.4 QUESTION 4: What results have EROL’s and KLP’s partner institutions achieved through capacity building activities of the subject programs? How much has EROL and KLP utilized on the job training and coaching with partner Institutions (in comparison to classical training) and what are the concrete results on the ground? 4.4.1 Findings EROL’s capacity building activities focus on increasing KJI’s acceptance and understanding of the benefits of on-the-job training and coaching in comparison to classical training. KJI, with EROL’s input, organized trainings with the objective of focusing on the sustainability of training programs that require building staff capacity to conceptualize, design, develop, and conduct trainings using modern adult educational methodologies. 20 KJI worked with EROL in the development and application of ToT methods, which include programs designed for ILEP and CLEP. EROL’s efforts emphasize ToT programs and mentoring in combination with substantive teaching materials, which provides KJI the framework to develop lasting and transformative programs of study and core curriculum. The Bench Book Initiative establishes legal writing and reasoning competencies, serving as model secondary sources that KJI has integrated into its programmatic offerings. In addition, legal skills for judges and their staff, including trial/litigation management and legal research, utilize targeted, practical exercises and practicum. EROL delivered a Judicial Trial Best Practices program to Kosovo judges using an interactive methodology built around case studies. Two KJI trainers were prepared to deliver the Workshop for Judges on Mock Trial (Civil) and the ICT Considerations Module for the Kosovo Judiciary. KJI worked closely with EROL to develop a holistic faculty development program built on the foundation of adult educational theory by: 1) evaluating the existing ToT curriculum; 2) developing and presenting training on modern adult pedagogy; and 3) developing a Faculty Handbook to acquaint new and experienced faculty with the adult education theories and practices KJI follows, and provide a framework for needs assessments, curriculum development, e-Learning, and the development of a court management curriculum. KIIs suggest that EROL’s one-on-one trainings and coaching with staff and the small group trainings are now well received by attendees. “Introducing judges to a new way of thinking about education takes time,” stated one KI, “it’s a new mindset. They are requesting more on-the-job trainings. They want to put theory into practice.” EROL’s capacity building activities at the justice institutions (MoJ, OP, KCC, KJC and KPC) staff utilized classical training through a combination of training sessions, roundtables and workshops. In addition to the KJI trainings and on-the-job training with court personnel, EROL organized numerous capacity building activities aimed at enhancing the capacity of justice institutions personnel and media representatives. EROL organized several training courses on legal research and drafting with lawyers and legal researchers from MoJ Legal Affairs Department, DILC, OP, KCC, KJC and KPC. Training sessions focused on techniques of drafting legislation on the basis of the applicable legislation and the European standards, as well as secondary legislation drafting techniques. In addition to trainings, EROL has supported several workshops and training programs on legal research, legal drafting and legal analysis, which included lawyers and legal advisors from KPC, KJC and MoJ. In addition, EROL’s trainings and workshops offered to KCC were highly appreciated by the court leadership and legal advisors for combining substantive course material with interactive case study exercises on legal reasoning and argumentation. EROL’s capacity building events have included ToT modules, which aim at equipping legal advisors from the OP, MoJ, KJC and KPC to deliver training through KJI programs or other events. Trainings were commended for both quality of materials covered and the interactive methods of delivery. The ET could not determine the impact of EROL’s capacity building on the trainees. However, qualitative information gathered during KIIs with several participants suggests that training programs were mostly based on classical adult learning methods and the quality of training was good and beneficial to trainees. EROL has significantly engaged in classical training and on-the-job training to advance the technical skills and professional capacity of court staff and judges in the Model Court Program. The ET obtained meaningful information and feedback regarding the technical skills EROL has advanced in its capacity building activities, particularly classical and on-the-job training. The ET found during the 3 FGDs that a total of 35 court staff took trainings on a variety of topics, as shown in Figure 1, below. Figure 1: EROL Trainings and Courses Taken by Focus Group Discussion Participants Training Percentage Who Took the Training Database Entry 73% Electronic Case Registries 58% Court Management Skills for Court Administrators 40% 21 Figure 1: EROL Trainings and Courses Taken by Focus Group Discussion Participants Training Percentage Who Took the Training Model Court Standards for Court Refurbishment Process 20-30% Electronic Time Stamp 20-30% Training of Court Statistic Officers, IT Officers, and Case Management Office Head 20-30% Training on New Case Registry 20-30% Change Management Leadership Training for Court Administrators 20-30% Over 80% of the mini-survey participants rated EROL’s training as Very Good or Good, citing the interactive teaching methodology, the written materials provided during the training, and the quality of the instructors as the best aspects of the training, while 18% stated that the training was Satisfactory. While the court staffs admitted they lacked professionalism prior to the Model Courts Program, they believe they have substantially increased their capacity primarily due to the on-site and classical training, but also to the reconfiguration of the court staff offices. Additionally, prior to the Model Courts Program, they recognized there was an absence of customer service as they were ill-informed of the fundamental role of the courts and of their responsibilities. Approximately 85% of the mini-survey participants stated that they are applying the skills learned through EROL’s training in their daily job, as shown in Figure 2, below. Figure 2: Mini-Survey Respondents Applying Skills Learned Through EROL Trainings in Daily Jobs Skill Percentage Applying in Daily Job Performing responsibilities at higher level of aptitude 61% Providing better services to and communication with public 52% Having better understanding of court’s role to provide services to public 45% Working as team with other court staff 33% Understanding responsibilities of other court staff 30% Understanding court operations 27% Training on New Case Registry 20-30% Change Management Leadership Training for Court Administrators 20-30% “Today, court services are more efficient and more transparent through court refurbishment, court services, court staff professionalism, and public information signage,” stated a KI who summarized the opinion of many KIs and participants in the FGD and mini-surveys as EROL’s greatest contribution to the court system. EROL also provided on-the-job training through the 7 court liaisons, each assigned to a different Basic Court, who initially helped with the implementation of the LOC and then began assisting court staff with the data entry process, which continues today. There were suggestions that the court liaisons, who also serve as the focal point of communication between EROL and the courts, provide assistance with the development and implementation of the individualized case backlog reduction action plans. 4.4.2 Conclusions  KJI has benefitted from the capacity building activities of EROL, which has introduced on-the-job trainings and coaching. The efforts have been well-received and more on-the-job trainings have been requested.  EROL’s capacity-building events for trainees of different justice institutions largely utilized classical training methods to the benefit of the trainees.  EROL’s classical and on-the-job training in its Model Court Program has advanced the technical skills of court staff, and inspired court staff to expand their professional skills and to work collectively as a team. “I consider EROL trainings as perfect for improving the efficiency and professionalism of court staff.” 22 4.5 QUESTION 5: How effective and efficient are EROL’s and KLP’s activities in coordinating with other USAID and other donors’ programs? 4.5.1 Findings EROL coordinated with other USAID and other donors’ programs, yet the perception of EROL’s work by the international donor community is mixed. According to KIIs with justice institutions stakeholders and other USAID donors, such as the Contract Law Enforcement Program implemented by Checchi and Company Consulting, Inc. and other donors’ programs, particularly those who work closely with the KJC, KPC and MoJ, such as EULEX, GIZ, and EU projects, two of which are “Support to the Kosovo Judicial/Prosecutorial Council” implemented by Human Dynamics, and "Further Support to Kosovo Legal Education Reform" implemented by German Foundation for International Legal Cooperation (IRZ), EROL has a mixed reputation in coordinating with other donors. While some donors highly praised EROL for sharing documents and ideas, and engaging in MoUs which delineate overlapping responsibilities, others who were less positive noted that EROL does not coordinate with them as effectively as it did in the beginning stages of the program. In assisting the DILC of the MoJ, EROL worked collaboratively with the EU Twinning Project launched in 2012. In providing capacity building assistance to the DILC, EROL focused on civil matters whereas the EU Project focused on criminal and child abduction matters. In early 2012, EROL proactively pursued and developed a 17- page agreement with Human Dynamics, which set forth the expected results each was to achieve in their respective programs, and allocated the responsibilities among them to reduce overlapping assistance which could have been abundant since both missions were similar. For example, Human Dynamics was the lead on reviewing the current provisions of the KPC on the evaluation of prosecutors, while EROL was the lead on improving and drafting new criteria for the evaluation of prosecutors. EROL intends to develop a cooperative agreement with IRZ, which also has overlapping responsibilities in legal education on court administration. With regard to KJI, although effort was made to coordinate with USAID and other donors’ programs, a large number of international donors have assisted KJI with similar goals in capacity development through trainings and practical programs for the Judiciary as well as the management, staff, and instructors. Major donors include USAID, United Nations Development Programme (UNDP), EU, OSCE, Council of Europe, various UN agencies, and GIZ among other donors. According to KIIs, many programs for KJI are developed with the same objectives. It is felt that many donors focus on KJI because of the relative ease in providing trainings, rather than addressing the more challenging development tasks. Other KIIs indicate that there is a lack of coordination and personalized assessment focusing on how the training will actually impact Kosovo. One KI stated, “In the assessment phase, consultation and coordination should be done with beneficiaries as well as donors.” Another KI stated, “It is crucial that a successful working group be established and that everyone be involved. Synergy is important among donors. We must give better service to Kosovo.” While EROL was not singled out as one of the errant international donors, these issues were stated many times during a significant number of KIIs. It should be noted that a KI who represents a significant international donor in Kosovo praised EROL for its “unprecedented’’ level of collaboration and cooperation. While conducting KIIs about donor coordination, the ET heard several comments that suggest a mixed perception of EROL’s work in Kosovo. EROL received extremely positive comments on its Model Court Program. Every KII with whom the Model Court Program was discussed, heaped praise on EROL and its contribution to the courts of Kosovo. EROL also received praise for its proactivity in the beginning of the program, but is perceived by many as being passive. Without quoting specific statements made about EROL, a common theme from the KIIs, even those who may have had positive comments about EROL’s coordination with other donors, questioned EROL’s passivity work with the justice institutions concluding that, relative to other US Government programs, EROL has not met the expectations of the justice institutions. 4.5.2 Conclusions  While EROL coordinates with other USAID and other donors’ programs, this coordination has not been effective and it is not to the degree which generates universal respect among the donors. 23 4.6 QUESTION 6: Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? 4.6.1 Findings EROL’s personnel changes, the concomitant lack of continuity in the provision of services, and reliance on numerous short-term advisors have been identified as a deficiency by the justice institutions. Since the inception of the program, EROL’s leadership and key personnel have changed frequently. This has been identified as a deficiency by the justice institutions as staff turnover was associated with the loss of institutional memory within the program, resulting in gaps and backsliding of the assistance provided. New senior staff members have been recruited without significant delay; however, the lack of continuity, combined with the additional time that newly-recruited staff members, as well as numerous short-term technical assistance (STTA) advisors need in order to understand the Kosovo context and build professional relationships, have caused unnecessary delays. There were, however, no complaints regarding the quality of assistance provided to the justice institutions. While EROL has noted that its use of STTA advisors is considerably less than that which is budgeted for which it should be commended, it does not diminish the perception by the justice institutions that STTA advisors are overused by EROL and that its current long-term staff is sufficiently experienced and qualified to engage in activities that some STTA advisors have conducted. 4.6.1 Conclusions  EROL’s staff changes and reliance on short-term advisors caused disruptions in the provision of services to the justice institutions, although the services that were provided were considered to be of high quality. 4.7 QUESTION 7: What lessons learned can be used in furtherance of ongoing programs and the planning of future USAID programs? 4.7.1 Findings There are considerable donors and international programs supporting ROL activities in Kosovo that have overlapping mandates and represent different legal systems; conflicting directives within and among donors; abundance of international experts, full-time and short-term; and various programmatic timelines. The ET learned from KIIs with officials within the justice institutions that it was burdensome for them to coordinate the international donors, many of whom, according to the KIIs, pursued their own goals, independent of other donors, and provided technical assistance relating specifically to their own legal system, particularly in the drafting of laws and regulations, resulting in confusing and conflicting provisions within the same law or regulation. The ET was also told that, given the abundance of international legal advisors from various donors and countries – many of whom are embedded in the justice institutions, while others are providing long- and short￾term assistance from project offices – the officials of the justice institutions are spending considerable time responding to the requests and needs of long-term advisors, and educating short-term advisors, rather than focusing on their own responsibilities, which they believe is one of the major reasons international assistance provided to the justice system, including EROL’s, is not as effective and efficient as it could be. Additionally, due to the abundance of donors, some justice institutions officials in the KIIs were unable to definitively state which donor provided specific trainings, confusing JSP with EROL, and which ROL programs supported by USAID and the EU provided advice and guidance on laws, regulations and policies. 4.7.2 Conclusions  It is difficult to attribute impact and benefit specifically to EROL as there are numerous international donors who provide similar assistance to the justice institutions. Furthermore, many officials within the justice institutions are often unable to identify which donor provided trainings and workshops, technical advice and guidance. 24  The overabundance of donors and international programs in supporting ROL activities has unintended consequences and has impeded progress. EROL’s programmatic effectiveness and efficiency is reduced in those areas in which there are many donors. Its success in the Model Court Program and its citizen awareness campaign are due to several factors, of which the most important is that it worked in an environment without substantial involvement by other donors.  While informal agreements exist among donors, there is no formal coordination mechanism among all ROL donors, which would offer a meaningful consultation and division of responsibilities among the donors. 5.0 EROL SPECIFIC FINDINGS AND CONCLUSIONS 5.1 QUESTION 1: What is the level of satisfaction of EROL’s court users’ survey compared vs. Public Pulse survey supported by USAID? 5.1.1 Findings The level of satisfaction of EROL’s court user’s survey of July 2013 is approximately 70% whereas the level of public trust in the judiciary measured by the Public Pulse Report (Pulse) survey of August 2013, supported by USAID and UNDP, is approximately 17%. EROL commissioned a court user satisfaction survey in 2013 to measure the public’s experience and satisfaction with court services and facilities in order to establish baselines for future measurements of court users’ knowledge, attitudes, practices and behaviors. One of the key objectives of the EROL’s survey was to measure the trust of court users. The ET identified incompatibility between the two surveys. For the purpose of the EROL’s survey, respondents were considered as court users while visiting a court and not self-identifying as court staff, probation staff, judge, prosecutor, police officer or media. Over 1,300 court visitors were randomly interviewed in the 7 Basic Courts and their respective branches, to capture their impressions and satisfaction with the court services and facilities they had just encountered. The respondent pool only includes those court visitors who were willing to be interviewed. If those who refused to answer the survey questions had been included in the response rate, EROL may have received a lower satisfaction rate as one could speculate that they did not respond due to either being unhappy or neutral with the court services. With the Pulse survey, 1,290 Kosovar citizens over 18 years old, who represented a sample of all Kosovo municipalities and ethnic minorities in both urban and rural areas, were surveyed with the purpose of measuring their trust in the judicial system but not their level of satisfaction with court services. Accordingly, by their nature, the surveys are incomparable. 5.1.2 Conclusions  While court users’ satisfaction is considerably higher in the EROL survey than in the Pulse survey, which measured the level of public trust in the judicial system, it is impossible to compare the level of satisfaction between the two surveys due to issues of incompatibility. 5.2 QUESTION 2: Is the trainees’ knowledge retention level of the EROL trainings supported at the KJI measured and what are the results of it? 5.2.1 Findings EROL measurement of trainees’ knowledge retention level is not measured. KIIs report that knowledge retention is not currently being measured by EROL. KJI keeps record of attendance, summary of training, and evaluations of all trainings carried on at KJI. 25 5.2.2 Conclusions  Knowledge retention is not being measured by EROL, therefore results cannot be analyzed. QUESTION 3: To what extent is the court case data base system populated and used for court administration throughout Kosovo? 5.3.1 Findings The case registry database developed by EROL as part of the Model Court Program has been implemented and is utilized for court administration to varying degrees in all courts of Kosovo. There is a disparity among the statistics generated by EROL’s case registry database, by the courts and by the KJC, which is a result of various factors, none of which relate to the design of EROL’s case registry database. EROL’s case registry database was designed so that reliable and uniform statistical data is collected for the purpose of inter alia identifying general and targeted judicial workload levels, reducing case backlog, and developing judicial performance standards and measures. In addition to EROL’s case registry database, individual courts continue to maintain their manual court registry books, with considerably more categories than EROL’s database of 6 categories, resulting in a broader spectrum of statistics. The KJC relies on both sources to generate its own statistics and, as a result, there is disparity among the statistics generated by each source. The ET learned through KIIs with EROL staff, relevant officials of the KJC, and Court Presidents, Supervisory Judges, Court Administrators and deputy Administrators that the degree the case registry database is utilized for court administration purposes depends upon the leadership of the courts. In visiting 9 courts and conducting KIIs with judges and EROL staff, the ET learned that Court Presidents of the Ferizaj and Gjilan Basic Courts, and Supervisory Judges of the Podujeva and Glogoc Branch Courts, were proactively engaged in analyzing the statistics generated by EROL’s case registry database to address the courts’ information needs, and to develop and implement case backlog reduction action plans. The ET was informed that while EROL’s data entry assistants were instrumental in populating and verifying the database information that has been inputted from the manual registry books until January 2014, inaccurate data entries are inevitable due to various factors. First and foremost, according to many KIIs, is human error, which several KIs believed could be reduced by continual training and by computer-generated filtering methods that identify common data entry errors. During FGDs and from the mini-surveys conducted with 35 court staff, the ET learned from those court staff who are primarily responsible for data entry that other factors that caused errors in EROL’s web-based court registry database, included: 1) the duplication or multiple entry data of the same case; 2) insertion of inaccurate dates; 3) the absence of any data for cases; and 4) the heavy workload of court data-entry clerks, particularly those who work in larger courts, such as the Pristina Basic Court. Since the database is web-based, several participants in the FGDs commented that the periodic absence of internet service prevented them from inserting the data in a timely manner. 5.3.2 Conclusions  The case registry system is maintained as accurately as possible and is utilized in varying degrees by courts for court administration. It is an essential first step in the development of a fully-automated case management information system. 6.0EROL RECOMMENDATIONS  EROL should significantly decrease assistance to MoJ, OP, KCC and KPC, and redirect it to KJC and courts.  EROL’s assistance to the MoJ should focus on supporting the process of revising or drafting justice-related laws and build the capacity of the SAO.  EROL should continue to offer assistance to the legal staff of the OP and facilitate coordination with other justice-related institutions. 26  At the KJC, EROL’s assistance should focus on increasing KJC’s and KJCS’ capacity in developing and implementing policies for a more efficient and effective administration of the courts. Future assistance is also needed to improve KJC’s capacities in developing judicial policies on caseload management, case backlog reduction, judicial discipline, judicial statistics, and data-based strategic planning.  The in-country team of EROL should engage more actively with the KJC and KJCS as opposed to extensive STTA engagement.  KJI needs support of the major donors to continue but the effort needs to be meaningfully organized.  The Model Court Program is not yet sustainable and should continue to receive USAID’s support. The continuation of the Model Court Program should focus on fully implementing court administration policies within all courts of Kosovo, and on continued training of all court personnel, including judges, on court administration and case management. As the new CMIS is being implemented, EROL’s assistance will continue to be necessary and adaptable to the court administration policies associated with KJC and the CMIS.  EROL’s assistance to the Forum for Women Judges and Prosecutors should focus on increasing the membership of the organization, strengthening its internal management and operations, and building alliances in international organizations of women judges and prosecutors.  EROL should expand outreach activities in non-majority communities and, should the situation allow, launch a sustained public outreach in Northern Kosovo.  While informal agreements exist among ROL assistance programs, the existence of a formal coordination mechanism among the major ROL donors, particularly the EU and USAID, would offer a meaningful consultation and division of responsibilities among the donors, thereby reducing overlapping and conflicting projects.  Given that the capacity of the Kosovars that the ET interviewed - all have benefited from JSP’s and EROL’s interventions – is sufficiently high to assume leadership positions in USAID ROL programs, serious consideration should be given to reducing the number of international senior staff positions and increasing the number of local senior staff positions in future USAID ROL programs. 7.0 KLP OVERALL FINDINGS AND CONCLUSIONS 7.1 QUESTION 1: How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of KLP’s program assistance components? To what extent have the programs’ met their five stated objectives and how effective have the programs’ interventions been in achieving the program’s stated objectives? Why or why not? To what extent have projects implemented the advanced ROL objectives? Are these activities beneficial to the institutions? Are they sustainable? If not, why not? 7.1.1 Findings KLP’s program assistance to the UPFL and Iliria College focuses on broad institutional support, skills training, career development, and academic research. KLP’s program assistance to the UPFL and Iliria College targets the strength and quality of legal education, along with the USAID objective of empowering Kosovars to consolidate a functioning democracy. The ET found that it is apparent that within KLP’s legal education component, quality support for a capable and active legal education was provided by working with the UPFL and Iliria College Faculty of Law to support clinical programs and practical skills development for law students. Data indicates that the number of Masters level students enrolled in legal clinics and of law faculty students obtaining internships have increased from Year 1 to Year 2. Figure 3: Increase in Law Student Legal Clinic Enrollment and Law Students Obtaining Internships Year 1 Year 2 Masters level law students enrolled in legal clinics 40 91 Law faculty students obtaining internships 151 373 27 KLP assists the faculties of the respective institutions to strengthen their academic research activities, and supports the establishment of linkages between them and public and private organizations through CDCs with the aim of improving student opportunities for internships and employment. Data indicates that in Year 2, partner institutions published a total of 5 policy papers. KIIs stated that KLP actively supports the establishment of functional cooperation among the faculties and other organizations related to the legal profession, such as KCA and KJI, for the benefit of students (internships, job placement etc.). A primary goal of the legal education component is to support high quality legal education and thus a future generation of capable and active legal professionals in Kosovo. As one KII stated “it all begins with strong and structured education.” KLP successfully continued the practical skills clinical programs for Masters level students at the UPFL, which were first established by the USAID/Legal Profession Development Initiative (LPDI) program. In addition, KLP facilitated the establishment of CDCs in the targeted law faculties that link to public and private organizations in order to promote and improve the career prospects for students and graduates. Further, the program worked with the faculties to develop their academic research capacities and then promote the use of this research for legal policy making in Kosovo. KLP’s Legal Education component strengthened the quality of legal education in Kosovo by expanding the availability of practical skills training at the Masters level, identifying professional opportunities for students, and strengthening university academic research capacity. Masters level legal clinics were improved and the number of internship opportunities for students at UPFL and Iliria College were increased. KLP handed over responsibility for the CDC to UPFL in March 2013. After wrapping up USAID’s assistance, provided since the CDC’s establishment in September 2010, CDC now has the capacity to sustain student career services. Site observation and KIIs indicate that the Offices of Clinical Studies were established and functional at both UPFL and Iliria College to provide sustainable coordination and a resource center for professors teaching and students enrolled in legal clinics. KLP assisted a total of 5 legal clinics, which benefitted 91 students. KLP’s Legal Education Expert reviewed and updated all curricula for KLP-supported legal clinical courses, resulting in improved teaching methodology, course content, and student exposure to practical skills. A KII stated that “introducing these new teaching methods creates a new mindset for the University of Pristina; we expected that it would take longer for the faculty and students to engage.” A MoU signed with Iliria College resulted in the establishment of the Office of Clinical Studies (OCSs) and Masters level civil law clinic. During 2013, 187 Masters students, 3 times as many at similar events in 2011, participated in career exchange roundtables implemented by UPFL and Iliria College CDCs through the Professional Panel Series. KLP internships continued through the UPFL CDC. During Year 2, KLP provided direct opportunities to 7 Masters law students. Five former KLP interns have now found full time employment with banks, law firms, and other organizations. KLP has provided key support to KBA on strengthening its recently formed Mandatory Continuing Legal Education (MCLE) system; strengthening its disciplinary system; designing and implementing many legislative reforms; and launching a Bench-Bar system to provide justice sector practitioners with the opportunity to establish inter-institutional cooperation for problem solving. KIIs with KBA officials informed that KLP, with the expertise of US lawyers, has supported KBA on its structural and administrative reforms. The main focus of such assistance has focused on KBA’s Statute, Code of Ethics, Disciplinary Code, and creation of MCLE, Disciplinary Committee, and GMC. KLP helped evolve KBA from a licensing organization to a membership organization. To do so, KLP assisted KBA with the revision of the Law on Bar, improved KBA’s efficiency, and increased the disciplinary system’s transparency by providing KBA with a more functional, responsive operational framework. KLP provided the main support for KBA’s newly-formed MCLE system. KLP’s legal profession component provides a large number of interventions designed to enhance KBA’s quality and, as discussed during KIIs, has been effective in designing and implementing activities that ultimately contribute to improving legal profession in Kosovo. 28 KIIs with KBA management discussed how KLP assisted KBA in taking major steps to improve its regulatory framework for professional ethics and discipline. With assistance from a DLA Piper Global Law Firm pro bono expert team, KLP helped KBA to re-draft the Regulation on Discipline, and recommended complementary adjustments to its KBA Statute and Code of Ethics. Evidence from KIIs and its performance indicators indicates that KBA has, for the first time in its recent history, directly achieved its structural planning of the annual budget and tracking mechanisms of expenditures. Furthermore, our KIIs revealed that sustainability of KBA is the most challenging factor they will face in the future. “We feel like we have achieved a lot in the last two years with KLP's hands-on support but I am not sure there is a will to sustain the progress after KLP.” According to KIIs and reviewed data, KLP has also helped KBA to strengthen the efficiency and transparency of its disciplinary system; worked to strengthen its MCLE and other training programs; improved outreach to the legal community on MCLE requirements; and assisted KBA in operationalizing a broadly accessible referral mechanism for legal advice. Moreover, KLP supported the KBA GMC to draft a work plan, budget, and activity schedule for each year. For the first time, KBA organized a donor meeting at the KLP premises. KLP provided advice to KBA on updating and distributing existing pamphlets on how to file complaints against lawyers; assisted KBA in updating complaint forms, making them widely available at KBA regional branches and on the KBA webpage, and developing and publishing an annual report on discipline; and assisted KBA in improving online publications on professional ethics rules and disciplinary procedure. Jointly with KBA, KLP provided and facilitated discussions on the better regulation of non-licensed legal professionals. KIIs and background research shows that KLP assisted KBA to function properly through the creation of the Structural, Administrative and Legislative reform, a disciplinary system, and the KBA Strategy Package, which includes the KBA Statute, Code of Ethics, and Disciplinary Code. The strategic planning sessions and strategic planning documents were done with the expertise of KLP experts. KBA’s Statute and Code of Ethics were amended accordingly after the new changes on the Law on the Bar. Further, KBA organized and launched a budgeting process for the first time with KLP’s support. KLP assisted KBA in organizing and launching a new system of Bench-Bar meetings in all Kosovo regions to provide justice sector practitioners with an opportunity for problem solving. Bench-Bars have resulted in 36 KBA initiatives, such as roundtables and meetings with judges, prosecutors, and police directorate officials in order to establish inter-institutional cooperation. One such roundtable was held in order to familiarize lawyers with the non-compliance to MCLE requirements. Bench-Bar meetings are sustainable and their organization has been fully transferred to KBA. 7.1.2 Conclusions  Justice Institutions in Kosovo are strengthened and have benefited from the implementation of KLP’s interventions. Through strengthened legal education and career enhancement, KLP has met ROL objectives of empowering Kosovars to consolidate a functioning democracy. KLP’s efforts become sustainable by education, mentoring programs, academic publications, and job placement activities.  KLP has contributed to the strengthening of the KBA by strategically improving its legal framework. KLP’s contribution was instrumental in redesigning and strengthening the MCLE – supporting compliance, training standards, and disciplinary system and revitalizing the MCLE program previously supported by ABA-ROLI. KLP improved the reorganization of KBA’s membership to improve the effectiveness of mentoring and professional development of members; strengthened professionally the MCLE and other training programs; and improved outreach to the legal community on MCLE requirements. 7.2 QUESTION 2: How have the Justice Institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of KLP’s activities? To what extent have the programs supported/assisted institutions to increase and strengthen gender diversity? Why or why not? 7.2.1 Findings KLP’s incorporation of USAID’s Gender Equality and Female Empowerment policy into its yearly activity planning have led counterpart institutions, including KBA and university law 29 faculty, to mainstream gender concerns by planning strategically and participating in year-round activities to address equality and empowerment. Female enrollment in the Masters Civil Clinic at Iliria College, which KLP supports, increased from 26% during the 2012/2013 academic year to 55% during the 2013/2014 academic year. During this time, the number of women taking the bar exam increased from 37% to 52%. Although KLP’s efforts have contributed to these increases, KIIs highlighted the fact that other donors, such as EU and GIZ, also contribute to gender initiatives. Due to the efforts of UPLF and Iliria College, with KLP support, to empower women with the knowledge and tools to identify, apply, and secure internships in professional opportunities resulted in females securing two￾thirds of law school’s student internships in 2013. Throughout its implementation, KLP has actively promoted gender equality and minority outreach by supporting women lawyers, recent women graduates, and women students from UPLF and Iliria College to become more active in the legal profession. Following accomplishments in 2012 and 2013, KLP recently encouraged recognitions and celebrations for International Women’s Day on March 2014. Despite the increase in females attending law school and taking the bar exam, background research and KIIs show that the low number of minorities and women admitted into the practice of law remains problematic. The disengagement of Kosovo Serb professionals since the unilateral declaration of independence on February 17, 2008, continues to present a challenge for minority participation in Kosovo’s legal educational initiatives. The political situation between Kosovo and Serbia continues to be tense and, currently, most Kosovo Serb students pursue legal studies at the University of Mitrovica or in Serbia. Therefore, as KIIs suggested, although KLP strives to identify opportunities for engagement and participation with minority students as they arise, very low numbers of minority students participate in KBA and study at the Law Faculties. The issues of women lawyers have less to do with their adequate representation in these organizations, but rather with the special problems women encounter when they open a private practice and the difficulties they face in obtaining a sufficient number of clients. The ET found that KLP focuses on the role of women and minority communities in the legal sector and the access of women and minorities to justice. KLP attempts to take into account the role of women and minority communities in all of its components and, in turn, tries to facilitate their involvement and educational and professional development. The incorporation of USAID’s new Gender Equality and Female Empowerment Policy into activity planning in Year 1 resulted in the development of KBA’s first multi-year Strategic Plan and activity timelines to promote gender equality in the legal profession and attract female law students. In Year 2, KLP supported the Professional Panel Series to provide students with concrete practical information that prepares them for entry into the workforce and competitive application processes. Year 2 professional panels focused on: 1) exposing students to the vast experience of professional experts and volunteers from the US; and 2) empowering female law students. With KLP’s support, 7 professional panels were organized in Year 2, with a total of 187 students attending, 143 of which were women. Research highlighted that KLP also incorporated USAID’s Gender Equality and Female Empowerment policy into Year 2 activity planning within the project’s framework of legal education and legal profession objectives. As a result, counterparts, including KBA and university law faculty, are mainstreaming gender concerns by planning strategically and participating in activities to address equality and empowerment year-round rather than only once a year for International Women’s Day events. Through the multi-year strategy, KBA is aiming to increase the number of licensed female lawyers, provide greater opportunities for female lawyers and law students to participate in the profession, increase legal literacy, and empower marginalized groups. In 2013, KLP interventions facilitated internships for 384 law students, more than twice the number in 2012 (165). Two-thirds of law student internship recipients in 2013 were female. The high number of female law students applying to and securing internships resulted from KLP’s efforts, along with those of the CDCs at UPFL and Iliria College and justice sector institutions via MoUs with universities, to empower young women with the knowledge and tools necessary to identify, apply, and secure internships and professional opportunities. Since KLP’s start in January 2012, 549 students have been placed in internships, including 349 women. 30 KLP assistance reactivated the KBA Gender and Minority Committee (GMC) with the purpose of increasing the number of women lawyers in its membership and leadership. KLP assisted KBA in launching programs and trainings on gender representation, including: a mentoring program; training with female students in the faculty of law; and workshop with recent female law graduates and female lawyers. KIIs informed the ET that KLP initiated assistance to the KBA GMC with the main purpose of increasing the number of women lawyers and minorities participating in KBA overall membership and leadership. KLP assisted and strengthened the KBA GMC in developing a results-oriented work plan. KLP’s assistance to KBA was, for the first time, on drafting and developing the Strategic Plan for Gender Equality and Female Empowerment. However, the GMC developed a 3-year strategy in 2012 and has developed work plans and budgets for both 2013 and 2014. The KBA has approved those work plans and budgets. In addition, KLP assisted KBA to launch the following programs and trainings with regard to gender representation: mentoring program; training for non-minority community; training with female students of the Faculty of Law; workshop with recent law graduates and female lawyers; paid visits of non-majority lawyers in Kosovo regions; and visit to women in prisons, all within the general operating budget of the KBA. KLP also drafted, with the support of two US Gender Experts, a White Paper – the “New Perimeter Legal Profession Project: Strategic Plan for the Kosovo Chamber of Advocates Gender and Minority Committee.” The two Gender Experts held a strategic workshop with the KBA GMC, and drafted a strategic plan to provide it with an instrument to use for planning and implementing its work to increase female and minority membership in the Committee. 7.2.2 Conclusions  Throughout its implementation, KLP has actively promoted gender equality by supporting women lawyers, recent women graduates, and women students from UPLF and Iliria College to become more active in the legal profession. As a result of its efforts, the justice institutions with which it works have increased their gender diversity by creating gender related committees, launching programs, providing trainings, and mainstreaming gender in their activities. However, other major donors such as EU and GIZ are involved with increasing gender initiatives, thus it is difficult to link all improvements directly to KLP alone; regardless, it is clear that KLP’s efforts are strategically focused and successful.  As a direct impact through the involvement of KLP, KBA has re-activated the GMC. 7.3 QUESTION 3: What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on Kosovo Judicial Council, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar. The impact of the programs on secondary legislation will also be important. 7.3.1 Findings KLP provided KBA with expertise on all internal regulations which have been harmonized with the Law on the Bar and the KBA Statute and which are adopted by the KBA Managing Board. According to data and KIIs, KLP has provided KBA with expertise to review the draft Law on the Bar; KLP helped lead to the adoption of a new regulation on MCLE. During Year I, KLP assisted KBA in re-drafting the Regulation on Discipline, and recommended complementary adjustments to the KCA Statute and Code of Ethics. Further, with KLP’s support, the new KBA Statute and Regulation on Disciplinary Procedure have been harmonized with the Law on Bar and adopted in November 2013. KIIs and site visits informed the ET that the following secondary legislation was adopted by KBA with KLP’s major professional assistance: Regulation on Disciplinary Procedure; Regulation on Financial Management of KBA; Regulation on the Exam in Code of Professional Ethics of Advocates; Regulation on Spatial and Technical Conditions of Advocacy Offices; Regulation on Mandatory Continuous Legal Education for Lawyers in Kosovo; Regulation on KBA Regional Branches; Regulation on Specialization; and Regulation for Organization and Work of the Committees. 31 7.3.2 Conclusions  KLP’s activities effectively contributed to the re-organization of KBA, with the main work on the legislative amendments to the Law on the Bar and to the secondary legislation. 7.4 QUESTION 4: What results have KLP’s partner institutions achieved through capacity building activities of the subject programs? How much has KLP utilized on the job training and coaching with partner institutions (in comparison to classical training) and what are the concrete results on the ground? 7.4.1 Findings KLP’s capacity building activities with UPFL and Iliria College have resulted in the incorporation of clinical programs and practical skills development into regular faculty curriculum, strengthening of academic research activities, and linkages between faculties and public and private organizations to improve student opportunities for internships and employment. Positive developments are noted in the area of legal education. Some of these include the incorporation of clinical programs into the regular faculty curriculum, support to academic staff with interactive teaching methods, and establishment of CDCs to help faculty students with internships and employment opportunities. According to KIIs, site visits, mini surveys, and FGDs, KLP works closely with UPFL and Iliria College to support clinical programs and practical skills development for law students. KLP supports the faculties to strengthen their academic research activities and supports the establishment of linkages between these faculties and public and private organizations through CDCs, which improve the flow of information to help improve the student opportunities for internships and employment. KLP provides focused support for UPFL Masters level practical skills clinical programs and to improve practical skills curriculum in Iliria College. The CDCs that KLP established provide advice to students on choosing career paths and maximizing their opportunities in a limited job market. Additionally, KLP works closely with the targeted law faculties to improve their academic research capacities and promote their use in the legal community in terms of legal policy making. KLP was instrumental in the activation of KBA’s MCLE, which has resulted in 92% lawyer compliance in 2012 compared with 72% in 2011. KLP led the establishment of the Office of the Disciplinary Prosecutor and reforms of the Disciplinary Committees within KBA. KLP utilized and launched the new Disciplinary Regulation that introduced important changes and disciplinary efficiency within KBA. Major achievements of the new system, which was led by KLP, are the establishment of the Office of the Disciplinary Prosecutor and reform of the KBA Disciplinary Committees. One of the best things about KLP is that they are not only always responsive to provide expertise but also to follow up the implementation of the advice. In addition, KLP launched Bar-Bench Meetings in all regions of Kosovo, as previously discussed. 7.4.2 Conclusions  KLP’s capacity building activities have resulted in Masters Level law students that have better practical skills. KLP has established cooperation mechanisms between the Law Faculties and other relevant legal institutions, and strengthened career centers and academic research capabilities.  With the changes made as a result of the implementation of some of the initiatives by KLP, the justice system in Kosovo has made progress in improving cooperation between the justice institutions and promoting the ROL. 7.5 QUESTION 5: How effective and efficient are KLP’s activities in coordinating with other USAID and other donors’ programs? 7.5.1 Findings KLP activities focus on coordinating with other donor programs, including those by the European Commission Liaison Office (ECLO), Swiss Cooperation and Development Office 32 (SCDO), GIZ, EU, Organization for Security and Co-operation in Europe (OSCE), United Nations Children’s Fund (UNICEF), and United States Department of State (State Department). As stated in USAID’s overall strategy, KLP is committed to increasing the effectiveness of its assistance and strengthening donor coordination. For example, KLP leverages its existing relationships with European donors, including ECLO, SCDO, and GIZ, to ensure that EU principles, standards, and knowledge are integrated into activities. This includes facilitating EU law curriculum development for integration into legal clinics and coordination between KBA and the newly established Chamber of Notaries. In an environment where personal contact, exchange, and relationship building are most important for programmatic success, KLP is fortunate to have a CoP who has successfully navigated the international donor community for a lengthy period of time. Repeated KIIs mentioned that KLP’s success can be directly attributable to the CoP’s lasting reputation and length of time he has spent in Kosovo. Currently, 3 international donors are assisting KBA in addition to KLP – OSCE, UNICEF, and State Department. Coordination activities between the various donors are sufficient and KBA holds regular semi￾annual meetings with other international donors in Kosovo. KIIs and data show that OSCE provides support on an ad hoc basis depending on funding and eligibility; for example its 2014 budget is not yet approved. UNICEF provided assistance for the Manual for Juveniles following adoption of the new Criminal Code and Criminal Procedural Code on January 1, 2013. The State Department Justice and the People project provided pamphlets, which had been developed in the context of its citizen awareness activities. KLP is ensuring the pamphlets are available and assisting KBA with its website information, forms, etc. 7.5.2 Conclusions  KLP’s activities in coordinating with USAID and other donors programs are effective and efficient.  KLP’s CoP is credited for his knowledge of the country’s legal education environment and his ability to navigate himself in the international donor community. 7.6 QUESTION 6: Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? 7.6.1 Findings Identified deficiencies regarding implementation of KLP’s current objectives include the necessity to frame activities within the constraints of the academic and funding cycles and the turnover or criticism of key figures at partner institutions. KLP’s programmatic deficiencies include the reality that activities with law faculties must be framed within the constraints of the academic and funding cycles. While both UPFL and Iliria College have expressed commitments to enhance practical skills training, promote career opportunities, and research visibility for graduate students, some delays may be experienced in effective program implementation as funding allocations and accreditations proceed according to their set schedules. Collected data and KIIs illustrate the fact that KLP frames all program activities with a focus on institutions rather than personalities. As pointed out in KIIs, the KBA presidency rotates periodically, and the dean position at UPFL is an envied position that can easily come under attack. As a result, KLP seeks to involve broader groups of internal stakeholders in these institutions to avoid dependency upon leaders who may or may not maintain their current positions. 7.6.2 Conclusions  The deficiencies of the necessity in adhering to the constraints of the academic and funding cycles and of the turnover of key figures at partner institutions have been identified and remedied to the best of KLP’s abilities and on-the-ground realities. 33 7.7 QUESTION 7: What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs? 7.7.1 Findings Prior to KLP’s assistance, mentorships were missing from law programs, including: students were placed into internship programs without an assigned mentor; OCS did not have an official mentor; and Masters level students were not assigned mentors for their research and publication efforts. Noticing these voids, KLP introduced mentorships to UPFL and Iliria College to sustain its efforts in legal education. From KIIs and research data, monitoring and mentoring needs to be continued to assure KLP’s efforts are fully utilized. The Coordinator of the OCS at the University of Pristina needs mentoring as well as the interns. Monitoring and on-site visits to classrooms should be conducted to ensure that curricula and advanced teaching methodologies are being followed. Research and publication of Masters level students and professors still requires mentoring and attention. Partnering with additional US and European universities could provide additional exposure for both students and faculty. 7.7.2 Conclusions  Establishing mentoring programs where they have not previously existed is beneficial to legal education projects as the mentoring allows students an opportunity to put learned classroom theory into workplace practice. Mentoring programs also prove beneficial to legal education reform that includes a newly introduced curriculum and teaching methodology because they confirm that the introduced changes are being continued.  Strengthening and expanding new and existing partnerships with US and European Law schools is beneficial for legal education reform efforts as these relationships can cement the requirement for providing extended mentoring and monitoring needed to sustain development efforts. 8.0KLP SPECIFIC QUESTIONS 8.1 QUESTION 1: To what extent is the management, quality and availability of the legal clinics at the Law Faculty of the University of Pristina and the Iliria College better as a result of KLP assistance? 8.1.1 Findings KLP’s efforts have focused on increasing the management, quality, and availability of legal clinics at the Law Faculties by establishing Offices of Clinical Studies, re-establishing the Civil Law Legal Clinic at UPFL, improving legal clinics to be able to assist more students, updating curriculum, and improving teaching methodology and course content. Site observation and KIIs indicate that the Offices of Clinical Studies were established and functional at both UPFL and Iliria College to provide a resource center and sustainable coordination for professors teaching legal clinics and the enrolled (or interested in enrolling) students. KLP assisted a total of 5 legal clinics at UPFL and Iliria College, which provided 91 students with practical professional experience, including trial practice, client interviews, and other aspects of lawyer-client relationships. KLP’s Legal Education Expert reviewed and updated curricula for all KLP-supported legal clinical courses, resulting in improved teaching methodology, course content, and student exposure to practical skills. All relative Masters level course curricula were drafted and adopted. The Civil Law Legal Clinic at UPFL was re￾established after a 2-year break. The Masters level Legal Clinic in Civil Law was introduced in academic year 2013-2014. A MoU signed with Iliria College and other activities resulted in the establishment of the OCS and Masters level civil law clinic. KLP oversaw the improvement of Masters level legal clinics and the increase in the number of internship opportunities for students at UPFL and Iliria College. KLP’s work with clinical legal education at UPFL included establishing the Office of Clinical Students. From on-site observations, the clinic is functional. 34 8.1.2 Conclusions  KLP’s efforts have greatly added to the management, quality, and availability of legal clinics at Law Faculties of the University of Pristina and Iliria College, allowing them to provide a better environment for law students. 8.2 QUESTION 2: How has the academic research capacity at the Law Faculty of the University of Pristina improved as a result of the KLP assistance? 8.2.1 Findings The 3 main ways in which KLP has assisted in improving the research capacity at UPFL include: 1) assisting in writing a draft statute to establish a research institute to assist in publishing academic papers; 2) distributing the Blue Book, which provides proper legal citations, to law students and faculty; and 3) introducing law students and professors to the Social Science Research Network (SSRN) which is an avenue for the publication of social science papers. KLP has undertaken many activities to improve the academic research capacity at UPFL. First, KLP assisted in writing a draft research institute statute to secure the establishment of a Research Institute at the University of Pristina to assist the school’s academic community in publishing academic papers. However, it should be noted that at the time of this evaluation, the institute was not set up and there had been no publications in academic journals. Although the regulation to create the Research Institute has been adopted by the UPFL’s Council, it has not yet been approved and adopted by the University of Pristina Senate due to changes in the Senate Rector and Vice Rector, who are serving on an interim basis for 6 months and lack the ability to make approvals or changes in regulations and curriculum. Second, KLP distributed the Blue Book, which shows proper citation formatting for all legal publications to law students and faculty members to assist in them in writing legal papers. Finally, KLP introduced law students and professors to the SSRN, which is an avenue for the publication of social science papers. For the first time, through this connection, a Masters level law student from UPFL published a research paper on SSRN. KIIs noted that KLP is helping to develop an academic mindset among professors and teaching staff at UPFL, along with a sense of pride in their scholarly professions. 8.2.2 Conclusions  KLP’s efforts to improve the academic research capacity at UPFL are off to a good start; however, much work remains to be done. Although KLP has set the stage for huge improvements in UPFL’s research capacity, including starting the process to establish a research institute and facilitating beneficial relationships for the publication of academic research, these require further assistance in order to achieve actual publication. 8.3 QUESTION 3: What have been the effects of KLP support for Iliria College? 8.3.1 Findings KLP strong focus on Iliria College support and the addition of the legal clinics, has led to a noticeable jump from strictly literature (theory only) style learning to clinical education (practical experience) and greatly improved research capacities for both faculty and students. From KIIs, on-site observations, student mini-surveys, and faculty FGDs, the effects of KLP support at Iliria College are well-received and positive. KIIs state the following:  Research capacities are greatly improved with both faculty and students. Research is now part of student evaluation and is calculated with final grade.  The way of teaching is changed, pointing out “It is now self-directed and more interactive.”  Communication between faculty and students is improved. Students are more encouraged and positive.  Overall the environment is better. There is a new way of thinking and an awareness of the level of quality legal education. 35  The CDC and the clinics have taken Iliria College from 0 to 9 in better serving the students. The ET conducted a mini-survey with 22 students (11 females and 14 males) from Iliria College focusing on their experience with the Legal Clinics and CDCs. Typical responses to the question, “How has the CDC served you?” included that it has assisted in internship placement, training, career and professional planning, gaining insights into the work of practicing lawyers and notaries through lectures, confidence building, and strengthening convictions. When asked, “Have you attended any Legal Clinics and what was your experience?” students provided the following responses: “It was a very good experience which will help me in my profession…it impacts our perception of the judicial practice and professional experiences. I have learned a lot by doing different activities, such as drafting contracts, rulings, and decisions.” “It allows us to apply theoretical knowledge into practice by writing different documents and attending speeches/lectures of different professors, judges, and attorneys about their experience in their legal professions, which is very good.” When asked: “Have you attended any Legal Clinics and what was your experience?” a total of 20 students responded positively. Two students had not attended clinics. Professors noted during the faculty FGDs that with the addition of the legal clinics, there has been a noticeable jump from strictly literature (theory only) style learning to clinical education (practical experience). The task will be to hold and update this level when needed. “It is possible for students to study, but not learn. They must be 100% capable to do their jobs. KLP/NCSC assistance is invaluable to us.” 8.3.2 Conclusion  KLP’s support to Iliria College is visible and profound; however mentoring and monitoring support is still needed. 8.4 QUESTION 4: How has KLP contributed to improving the functioning of the Mandatory Continuing Legal Education and the disciplinary system at the Kosovo Chamber of Advocates? 8.4.1 Findings KLP engaged in many activities to improve MCLE and the disciplinary system of KBA KLP was instrumental in strengthening the functioning of MCLE – supporting compliance, training standards, and disciplinary system – which was launched by KBA when it was supported by ABA-ROLI. MCLE requirements were initially introduced for all lawyers in 2010 and went into effect on January 1, 2011. KLP assisted KBA in drafting its first MCLE strategy and annual program outline. According to KII observations, KLP provided crucial assistance to support and direct the MCLE disciplinary system (Regulation on Discipline). Additionally, the establishment of the Office of Disciplinary Prosecutor on January 1, 2014 is a major step toward fulfilling KBA’s mandate. KLP’s international experts designed and delivered many training workshops to KBA members on the MCLE Disciplinary Committee to introduce more methodologies in trainings. KLP assisted the Disciplinary Committee to introduce the Disciplinary Prosecutor. According to KIIs, KLP designed and delivered the first ever certification training program for KBA trainers. The project also drafted the Regulation of Specialization, later adopted by the KBA’s Managing Board. KLP assisted KBA in the drafting regulations on the work of the Disciplinary Committee, which it then adopted in 2014. According to these regulations, KBA decreased the number of mandatory training hours for lawyers over age 70 to 5 points, while for others it remains at 10 points. New lawyers with 3 years of experience or less must have 15 points in order to pursue MCLE trainings. Information from KIIs shows a significant improvement in compliance with fully implementing KBA’s Regulation on MCLE and fining non-compliant lawyers. As a result, more than 90% of KBA’s membership now complies with MCLE requirements. KLP established a new system for mentoring KBA disciplinary staff to ensure disciplinary cases are recorded and tracked. Following adoption of the Regulation on Discipline, KLP has provided advice to the KBA Disciplinary and Ethics Committee to functionalize the new disciplinary standards and procedures. Throughout 36 the program, the ET found that specific interventions were undertaken for development of forms, protocols, and guidelines for implementation of the Regulation on Discipline. A shared concern during the KIIs with lawyers was that notwithstanding the high level of expertise and commitment from KLP project - still needs a lot more institutional strengthening as the past experience has shown that progress stops after internationals go. 8.4.2 Conclusion  As a result of regulations drafted by KLP and KBA, 90% of KBA’s membership now complies with MCLE requirements. Although the number of the KBA members complying with the MCLE’s requirements has increased significantly, 72% (2011), 92% (2012), and 93% (2013), the general awareness of what constitutes a disciplinary violation amongst lawyers is still weak.  In general, the disciplinary system has been improved through training workshops developed and delivered by KLP, and the Disciplinary Committee and the Office of the Disciplinary Prosecutor are more active but they lack the proper expertise and need further strengthening to enforce lawyers discipline. 8.5 QUESTION 5: How has gender and ethnic diversity improved within the Kosovo Chamber of Advocates membership as a result of the KLP assistance? 8.5.1 Findings KLP has directly assisted on restart of the GMC. KLP supported the KBA GMC in organizing a series of meetings intended to help draft the work plan, budget, and activity schedule. KLP’s greatest support to KBA was in assisting GMC to develop a policy paper recommending KBA leadership options for facilitating the membership of underrepresented groups, including women and ethnic minorities. The adoption by the KBA General Assembly of the GMC Strategic Plan, whose development KLP directly supported, marks the first step in the implementation of its 2013 Work Plan. In Year 1, the ET found that KLP assisted GMC in developing and adopting an action plan to attract membership from under-represented communities, including women, young graduate, and minority lawyers. Further, KLP assisted GMC in implementing activities in its 2013 Work Plan and in the 3-Year Strategic Plan. KBA GMC’s mentorship program for women lawyers was recently re-launched with 12 women KBA lawyers mentoring 12 young women lawyers who are either recent law school graduates or at in law school at the Masters level. According to many KIIs, GMC’s current budget of 10,000 Euros, which is allocated by KBA, is insufficient to deal with its needs. However, this is the only KBA committee with an allocated budget. KLP has played a crucial role in enhancing GMC’s capacities by proactively assisting it to reach out to young graduate female lawyers to enhance their visibility outside of KBA. Despite these efforts, however, the representation of women and minorities remains KBA’s biggest challenge. Currently, only 8% of KBA members are Kosovar ethnic minorities. Minority representation remains low primarily because of the current political situation in Kosovo. For the first time, KBA met with Serbian lawyers in Mitrovica and Pristina to request their membership. Minorities pay an annual KBA membership fee that is 50% lower than non-minority members and women are permitted to pay their KBA as well as MCLE fees in installments. 8.5.2 Conclusion  As a direct impact through the involvement of KLP, KBA established the GMC. GMC is now functional and is continuously increasing the number of women lawyers and minority participation in KBA leadership and overall KBA membership. KLP supported the KBA GMC in organizing a series of meetings intended to help draft the work plan, budget, and activity schedule. Out of 570 current registered members of KBA, only 62 are women lawyers (12%). 9.0KLP RECOMMENDATIONS  KLP’s continuing focus needs to be on the sustainability of clinics, CDC, and research capabilities. Partner institution support should be strengthened.  Continued robust support needs to be given Legal Education. 37  Focus should be on Legal Specializations from curriculum to Research Institutions. Curriculum, teaching methodology and Academic Institutions need to promote Legal Specialization. Students should be able to graduate with the acceptable academic credential to be Tax Lawyers, Family Lawyers, Criminal Lawyers, Commercial Lawyers and so on. Ideally this effort should begin at LLB studies, and should be pronounced at LLM level. The absence is causing challenges not only in the practice of law but also at the judicial level. Currently Judges are required to make decisions in areas of law they do not know well.  Academic Research Institutions need further support. These institutions should support specializations as well; for example, a Fiscal Law Academic Research Institution.  Partnerships with US Universities/Law Schools should be actively promoted, for students as well as Professor exchanges.  KBA and MCLE should be maintained with USAID support as KBA lacks the professional ability to do so. In order to achieve sustainable results from its activities, KLP should continue to work with KBA on regulatory reform.  KLP should further assist KBA on the quality of the MCLE Training Center and on improving the quality of further legal education.  Additional assistance to the KCA Gender and Minority Committee is necessary for it to become more engaged in empowering women and underrepresented communities and to help it increase the number of KBA women interns.  Future KLP’s support to the KBA is necessary in increasing the quality of MCLE and strengthening the Office of the Disciplinary Prosecutor. According to the disciplinary provisions of the KBA, the new control system of MCLE trainers should be established. The quality of the MCLE trainers should be improved, as they are the main weakness of the current format. Well-known experts from the various fields of law should be invited to deliver trainings at MCLE. MCLE needs further support to function at a higher level, serve the needs of the lawyers, and become sustainable.  Should the political circumstance allow, KLP and KBA need to focus outreach efforts to integrate the Kosovo Serb lawyers in the legal profession of Kosovo through licensing, training delivery in native language and facilitating the full integration of Kosovo Serbs in the legal community. 10.0 GENERAL RECOMMENDATIONS  If the result of the Normalization Agreement between Kosovo and Serbia create the conditions for the restoration of court institutions in the North, USAID should utilize its convening power and EROL’s successful experience in implementing court re-structuring to facilitate the implementation of transitional arrangements for establishing effective and efficient courts operations in the North.  USAID’s future assistance in the ROL area in North Kosovo (through EROL, KLP or otherwise) should dedicate a strong donor coordination component between US, EU and bilateral donors in order to avoid the negative consequences of un-coordinated assistance.  USAID should promote the establishment of the formal donor coordination mechanism where donors share their plans and clearly delineate the areas of assistance in the justice sector in North Kosovo before embarking on implementation. This mechanism should take into account the lessons learned from the un￾coordinated donor assistance in the justice sector of Kosovo and avoid overlapping projects and conflicting advice through periodic coordination meetings during the implementation phase.  In accordance with the standards of the Model Court Program, EROL program should work closely with KJC to complete the ongoing physical intervention in the court building in Mitrovica and start the refurbishment of Branch Court buildings in Leposavic and Zubin Potok.  EROL should work closely with the leadership and Basic Court of Mitrovica and the KJC to coordinate transitional arrangements for establishing courts in the North such as recruitment of judges and court personnel, training programs, file transfers, record management and court administration.  Upon resumption of basic court operations in the North, EROL should facilitate the participation of court leaders in Mitrovica, Leposavic and Zubin Potok in the KJC workshops with court presidents and prepare 38 plans for the establishment of case management offices, installation of tracking database and train judges and court administrators on court administration.  The KLP and KBA should establish closer relationships with Kosovo Serb lawyers, by inviting judges, prosecutors and KBA Kosovo Serb members in the workshops, round-tables – in order to facilitate their integration in the legal community of Kosovo.  USAID’s future assistance in the ROL should include a robust minority outreach component targeting the Kosovo Serb and other non-majority communities of Kosovo through national broadcaster RTK 2 pro bono – which gives 85% of its program in Serbian language, while 15% for other minority groups. ANNEXES ANNEX A: SCOPE OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK C.1 INTRODUCTION Name of Activities to be Evaluated: Effective Rule of Law Program (EROL) Kosovo Legal Profession Program (KLP) Implementer: Checchi Consulting/ National Center for State Courts (NCSC) Award Number: AID-167-C-11-00001/AID-167-A-12-00001 Contract/Agreement Value: $23,614,701.00/ $1, 989,970 Life of Program: March 2011- March 2015/January 2012- January 2015 Period to be Evaluated: March 2011-present C.2 BACKGROUND These projects began as follows: EROL started on March 23, 2011 and will end on March 22, 2015; KLP started on January 3, 2012 and will end on January 2, 2015. These activities sought to assist/support the strengthening of the Kosovo Judicial Council (KJC) including specific courts through Model Court component, the Kosovo Prosecutor’s Council (KPC), the Ministry of Justice (MOJ) and Kosovo Judicial Institute (KJI). Initiatives also provided technical assistance to enhance the capacity of justice sector personnel, lawyers and law students, strengthen the KJC’s ability to fulfill its independent oversight and disciplinary functions, and help streamline and modernize the court system. Moreover the KLP is a three-year program which main objective is improvements in legal education, the capacities of the legal profession and public awareness and outreach. The KLP assisted the Kosovo Chamber of Advocates (KCA), Faculty of Law Pristina University and Iliria College. C.3 PROGRAM OBJECTIVES USAID has been instrumental in building the institutions and aiding in developing legislation that underpin the State of Kosovo. Improving the function of these institutions, including their legal framework, is imperative to Kosovo’s continued progress towards Euro-Atlantic integration. The EROL is a four-year program that focuses on making the justice system more independent, accountable, efficient, and effective. The centerpiece of this program is bolstering the justice sector’s operational capacity, focusing on the courts, the Kosovo Judicial Council (KJC), the Ministry of Justice (MOJ), and the Kosovo Judicial Institute (KJI). This includes refurbishing eight Model Courts, which will help to guarantee equal access to justice for all of Kosovo’s citizens. The bulk of EROL’s activities are aimed at providing assistance to justice sector institutions to implement the Law on the Courts and the transition to the new court structure, which took take place in January 2013. The KLP is a three-year program that has three objectives: (1) improve the quality of legal education; (2) improve the capacities of the legal profession; and (3) improve public awareness and outreach. KLP initiated and maintains continuing legal education activities to improve the practical skills of law school students and lawyers throughout Kosovo. KLP is also set to increase academic research capacity and establish Career Centers to promote better career prospects for students of both public and private university law faculties. In addition KLP works with the Kosovo Chamber of Advocates to develop a system to better regulate licensing of legal professionals and institutionalize of disciplinary mechanisms as well as strengthen the Chamber’s mandatory continuing legal education requirements. C.4 PURPOSE OF THE EVALUATION The main purpose of this performance evaluation is to provide USAID/Kosovo with an objective external assessment of the management and performance of the: 1) EROL program that started on March 23, 2011 to the present date; and 2) the KLP program that started on January 3, 2012 to the present date. C.5 EVALUATION OBJECTIVE USAID Kosovo is interested to evaluate EROL and KLP programs. This process should include two main tasks: (1) Evaluate EROL’s and KLP’s current performance; and (2) Recommend adjustments to both activities through the end of current awards; The ET is tasked with assessing the efficiency, impact, sustainability, and relevance of the activities implemented through the EROL and KLP programs. The ET will examine the overall impact of the activities on the target institutions and validate/observe the progress made in achieving the results and objectives as specified in the EROL and KLP awards and the adjusted strategic plan. The ET will review actual versus planned progress in attaining the anticipated results; identify and analyze problems, delays and other issues related to project implementation; document lessons learned; and make recommendations for future USAID assistance in the justice sector. The results of this evaluation will provide feedback to USAID/Kosovo for possible corrections for the remaining the life of the EROL and the KLP awards, which is expected to end by March 2011- March 2015/January 2012- January 2015. C.6 TARGET STAKEHOLDERS The target audience for this assessment includes USAID/Kosovo staff, especially the Democracy and Governance Office, its implementing partners, local stakeholders, and local beneficiaries. Local stakeholders include the Kosovo Judicial Council, Kosovo Prosecutors Council, Ministry of Justice, Kosovo Judicial Institute, Office of the President of Kosovo, Constitution Court, Presidents and Court Administrators of completed Model Courts, the European Union Office and their implementing partners representatives’, Kosovo Chamber of advocates, and the Faculty of Law/University of Pristina, “Iliria College”. C.7 SCOPE OF WORK The Contractor will provide a four person team to conduct the EROL and KLP Performance Evaluation. The team will develop and adopt an approach that elicits and analyzes information, provides key findings, conclusions, recommendations and lessons learned. The Contractor will design and execute the evaluation to generate detailed knowledge about the performance of the EROL and KLP projects, to measure accountability, project outcomes and benefits, and make recommendation about continued USAID involvement. The Contractor will develop an evaluation plan, including a draft Work Plan, that is most appropriate and feasible to accomplish the objectives set forth. The Plan will include the description of methods and procedures that will be used in gathering and analyzing both qualitative and quantitative data. The Contractor will collect data and information from the widest possible stakeholder group, including project participants, current employees, implementing partners, direct beneficiaries, and other donors. After contract award, at the request of the Contractor, USAID/Kosovo will provide an initial list of the stakeholders and their contact information. The Contractor will disaggregate collected data by sex to the greatest extent possible in order to ascertain how the project impacted men and women; how the activities affected the status and roles of women and men within the areas of intervention (for example roles in decision-making); how results of the work affected men and women differently; and what specific benefits of the program can be uniquely and specifically attributed to targeting women. USAID/Kosovo will provide the Contractor with key documents and background material relevant to Kosovo’s judicial sector and the applicable USAID project documentation, as well as any available documents deemed necessary to the Contractor to be familiar with the EROL and KLP activities. Key documents include: Quarterly Reports, Annual Reports, Work Plans, and Statistical Reports. C.8 EVALUATION QUESTIONS The Contractor must address the following key questions and may include others as necessary to meet the objectives of the evaluation. In addressing all evaluation questions the Contractor will do so in a manner and order that it determines to be most effective, efficient, and encompassing of all relevant stakeholders. The key questions for the evaluation include, but are not limited to:  How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of EROL’s and KLP’s program assistance components? To what extents have the programs’ met their five stated objectives and how effective have the programs’ interventions been in achieving the program’s stated objectives? Why or why not?  How have the Justice Institutions in Kosovo specifically strengthened their gender diversity as a result of the implementation of EROL’s and KLP’s activities? To what extent have the programs have supported/assisted institutions to increase and strengthen gender diversity? Why or why not?  What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on Kosovo Judicial Council, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar. The impact of the programs on secondary legislation will also be important ?How much has EROL and KLP utilized on the job training and coaching with partner Institutions (in comparison to classical training) and what are the concrete results on the ground?  What results have EROL’s and KLP’s partner institutions achieved through capacity building activities of the subject programs? To what extent have projects implemented the advanced ROL objectives? Are these activities beneficial to the institutions? Are they sustainable? If not, why not?  How effective and efficient are EROL’s and KLP’s activities in coordinating with other USAID and other donors’ programs?  Based on the review of EROL’s and KLP’s implementation and results, what recommendations are there for possible future USAID programming and/or other donors or governments in promoting efficient Rule of Law in Kosovo? What recommendations are there for enhancing the implementation of the Court Restructuring process?  Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? What are recommendations and lessons learned?  What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs? Questions specific to the EROL program: 1. What is the level of satisfaction of EROL’s court users’ survey compared vs. Public Pulse survey supported by USAID? 2. Is the trainees’ knowledge retention level of the EROL trainings supported at the KJI measured and what are the results of it? 3. To what extent is the court case data base system populated and used for court administration throughout Kosovo? Questions specific to the KLP program: 1. To what extend is the management, quality and availability of the legal clinics at the Law Faculty of the University of Pristina and the Iliria College better as a result of KLP assistance? 2. How has the academic research capacity at the Law Faculty of the University of Pristina improved as a result of the KLP assistance? 3. What have been the effects of the KLP support for the Iliria College? 4. How has KLP contributed to improving the functioning of the Mandatory Continuing Legal Education and the disciplinary system at the Kosovo Chamber of Advocates? 5. How has gender and ethnic diversity improved within the Kosovo Chamber of Advocates membership as a result of the KLP assistance? C.9 METHODOLOGY The contractor will design and execute an evaluation to generate detailed knowledge about the magnitude and performance of the EROL and KLP programs, to measure accountability and benefit, and to inform future activities. The evaluation should contain information to inform the allocation of resources across and within sectors. It is anticipated that the evaluation methods will include and rely on a mixture of methods, including documentation review, small surveys, and in￾person or telephone interviews with key informants in the U.S. and in-person interviews in Kosovo. The Contractor will review all of the available documents made available by USAID Kosovo prior to departure. Upon review of the documentation, the contractor will develop an evaluation framework (including a draft evaluation Work Plan) that is most appropriate and feasible to accomplish the goals outlined in the Scope of Work. In considering the evaluation design, the Contractor will incorporate diverse information gathering approaches in order to reach the widest possible sample of the main target audiences. In preparing a data-gathering approach, questions should be tailored to reflect, as appropriate, the specific roles of the stakeholders. The data analysis plan will include how interview and/or focus group interviews will be transcribed and analyzed; what procedures will be used to analyze quantitative data from surveys and qualitative data from key informant and other stakeholder interviews; any methodological limitations; and how the evaluation will weigh and integrate qualitative data with quantitative data. All data will be disaggregated by sex and minority as appropriate. C.10 TIMELINE This evaluation is expected to last up to 60 days. The exact dates of beginning are o/a March 3, 2014 and of ending no later than May 05, 2014. Tasks Team Leader Expat Advisors (2) Locally Hired Advisor Preparation and Research ( includes draft of an initial work plan and evaluation design) 6 days 3 days 1 day Round trip travel (US-Kosovo-US) 4 days 4 days On-site research and data collection in Kosovo. Meeting with USAID/Kosovo Mission’s M&E Specialist and with DGO Office; project stakeholders; Draft Report and Out-brief to USAID/Kosovo Mission 18 days 18 days 20 days Final Report due 8 days 4 days 3 days TOTAL 36 days 29x2 24 days C.11 RELATIONSHIP & RESPONSIBILITIES In accordance with USAID Evaluation Policy, this task order will be managed by the USAID/Kosovo Program and Project Office. Primary point of contact is Amy Southworth, Monitoring and Evaluation Specialist, Program and Project Office, asouthworth@usaid.gov Secondary point of contact is Aferdita Nimani, Program and Project Office, animani@usaid.gov. The Mission’s Monitoring and Evaluation (M&E) Specialist, or his/her designee, will be the designated Contracting Officer’s Representative (COR) for this award. Upon arrival in Kosovo, the Contractor shall meet with the M&E Specialist and representatives from the USAID/Kosovo Economic Growth Office prior to starting any work. The Contractor will be responsible for administration, scheduling, transport and translation. C.12 ADDITIONAL REQUIREMENTS An acceptable report will meet the following requirements as per USAID rules and procedures (please see: http://www.usaid.gov/sites/default/files/documents/1870/How-to- Note_Preparing￾Evaluation-Reports.pdf The following considerations should also be included:  The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what program activities were most successful in achieving the desired results, what did not work and why;  The evaluation report should address all evaluation questions included in the scope of work;  The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline shall be agreed upon in writing by the USAID Mission M&E Specialist;  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report;  Evaluation findings will assess outcomes and impacts using gender disaggregated data.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparative groups, etc.);  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions;  Findings should be specific, concise and supported by strong quantitative or qualitative evidence;  Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed;  Recommendations need to be supported by a specific set of findings; and  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. All quantitative data collected by the ET must be provided in an electronic file in easily readable format agreed upon with the COR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. [END OF SECTION C] ANNEX B: EVALUATION DESIGN MATRIX 1 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis General Questions Evaluation Question 1 1.1 How have the Justice Institutions in Kosovo been strengthened and benefited from the implementation of EROL’s and KLP’s program assistance components?  USAID RFP for EROL  EROL’s Work Plans  EROL’s PMEP Reports  EROL’s Quarterly Reports  EROL’s Annual Reports  Other documents provided by EROL throughout the evaluation  USAID’s RFP for KLP  KLP’s Work Plans  KLP’s PMP Reports  KLP’s Quarterly Reports  KLP Annual Reports  Other documents provided by KLP throughout the evaluation  Law on Courts  Law on the Kosovo Judicial Council  Law on State Prosecutor  Law on the Prosecutorial Council of Kosovo  Law on the Special Prosecution Office  Law on the Bar  Substantial secondary regulations, such  Document Review  Key Informant Interviews (KIIs) with USAID, Checchi Consulting, National Center for State Courts, EROL and KLP staff  KIIs with key representatives of the Justice Institutions, specifically KJC, KPC, Office of the President, KJI, Constitutional Courts, KBA, University of Pristina and Iliria College.  KIIs with beneficiaries of EROL and KLP, such as attorneys, judges, prosecutors, court administrators, law professors, and law students.  KIIs with representatives of other donors, EULEX, EUOK, Norwegian  Analysis of targeted results, outputs and outcomes of EROL and KLP  Verification of EROL and KLP reporting  Non-EROL and non-KLP documents  Independent research  Opinions elicited during the KIIs, FGDs  Recommendations elicited during the KIIs and FGDs  Quantitative analysis of the results of the mini-surveys 2 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis Regulation on Organizational Matters and Activities of the Courts  Concept Papers of the MOJ  Strategic Plans of the justice institutions  Codes of Ethics of the justice institutions development office  Focus Group Discussions (FGDs) with judicial trainees of EROL’s training, prosecutors, judges, court administrators of model courts, members of the KBA and law professors  Mini-survey of law students, and court users  On-site observation of the activities of EROL and KLP  Site visits to Pristina, Gjakove, Prizren, Peja, Suhareke, Istok, Ferizaj, Mitrovica Gilan and Kline. 1.2 To what extents have the programs’ met their five stated objectives and how effective have the programs’ interventions been in achieving the program’s stated objectives? 1.3 Why or why not? Evaluation Question 2 2.1 How have the Justice Institutions in Kosovo specifically strengthened their  SAME DATA SOURCE AS ABOVE  SAME METHODOLOGY AS  SAME DATA ANALYSIS AS ABOVE 3 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis gender diversity as a result of the implementation of EROL’s and KLP’s activities? ABOVE 2.2 To what extent have the programs supported/assisted institutions to increase and strengthen gender diversity? 2.3 Why or why not? Evaluation Question 3 3.1 What is the current capacity and timeline for Justice Institutions to become effective in exercising prescribed responsibilities under the judicial package laws (i.e. Law on Courts, Law on Kosovo Judicial Council, Law on State Prosecutor, Law on Prosecutors Council and the Law on Bar)? The impact of the programs on secondary legislation will also be important.  SAME DATA SOURCE AS ABOVE  Annual Court Reports  SAME METHODOLOGY AS ABOVE  SAME DATA ANALYSIS AS ABOVE 3.2 How much has EROL and KLP utilized on the job training and coaching with partner institutions (in comparison to classical training) and what are the concrete results on the ground? Evaluation Question 4 4 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis 4.1 What results have EROL’s and KLP’s partner institutions achieved through capacity building activities of the subject programs?  SAME DATA SOURCE AS ABOVE  SAME METHODOLOGY AS ABOVE  SAME DATA ANALYSIS AS ABOVE 4.2 To what extent have the projects implemented the advanced ROL objectives? 4.3 Are these activities beneficial to the institutions? 4.4 Are they sustainable? If not, why not? Evaluation Question 5 5.1 How effective and efficient are EROL’s and KLP’s activities in coordinating with other USAID and other donors’ programs?  SAME DATA SOURCE AS ABOVE  KIIs with key representatives of the Justice Institutions, specifically KJC, KPC, Office of the President, KJI, Constitutional Courts, KBA, University of Pristina and Iliria College.  KIIs with representatives of other donors, EULEX, EUOK, Norwegian development office  SAME DATA ANALYSIS AS ABOVE  Evaluation Question 6 6.1 Based on the review of EROL’s and KLP’s implementation and results, what recommendations are there for  SAME DATA SOURCE AS ABOVE  KIIs with USAID, Checchi Consulting, National Center for  SAME DATA ANALYSIS AS ABOVE 5 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis possible future USAID programming and/or other donors or governments in promoting efficient rule of law in Kosovo? State Courts, EROL and KLP staff  KIIs with key representatives of the Justice Institutions, specifically KJC, KPC, Office of the President, KJI, Constitutional Courts, KBA, University of Pristina and Iliria College.  On-site observation of the activities of EROL and KLP  Site visits to Pristina, Gjakove, Prizren, Peja, Suhareke, Istok, Ferizaj, Mitrovica Gilan and Kline. 6.1 What recommendations are there for enhancing the implementation of the court restructuring process? Evaluation Question 7 7.1 Which of the identified deficiencies in the overall implementation of the programs’ current objectives can be remedied in the remaining life of the program?  SAME DATA SOURCE AS ABOVE  KIIs with key representatives of the Justice Institutions, specifically KJC, KPC, Office of the President, KJI, Constitutional Courts, KBA, University of Pristina  SAME DATA ANALYSIS AS ABOVE  6 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis and Iliria College.  On-site observation of the activities of EROL and KLP  Site visits to Pristina, Gjakove, Prizren, Peja, Suhareke, Istok, Ferizaj, Mitrovica Gilan and Kline. 7.2 What are recommendations and lessons learned? Evaluation Question 8 8.1 What lessons learned can be used in furtherance of ongoing program and the planning of future USAID programs?  SAME DATA SOURCE AS ABOVE  Independent research on similar programs in other similarly-situated countries  KIIs with key representatives of the Justice Institutions, specifically KJC, KPC, Office of the President, KJI, Constitutional Courts, KBA, University of Pristina and Iliria College.  SAME DATA ANALYSIS AS ABOVE EROL Specific Questions EROL Question 1 1.1 What is the level of satisfaction of EROL’s court users’ survey compared vs. Public Pulse survey supported by USAID?  Public Pulse Survey  EROL’s Court User’s Survey  Mini-Survey of Court Users  On-site observation of the activities of EROL  Site visits to Pristina, Gjakove, Prizren, Peja, Suhareke, Istok, Ferizaj,  Quantitative analysis of the results of the mini-surveys  Opinions elicited during site visits 7 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis Mitrovica Gilan and Kline. EROL Question 2 2.1 Is the trainees’ knowledge retention level of the EROL trainings supported at the KJI measured and what are the results of it?  Training curriculum and modules of EROL’s trainings  FGD with trainees of EROL’s trainings  Recommendations elicited during the FGD EROL Question 3 3.1 To what extent is the court case data base system populated and used for court administration throughout Kosovo?  NA  NA  NA KLP Specific Questions KLP Question 1 1.1 To what extent is the management, quality and availability of the legal clinics at the Law Faculty of the University of Pristina and the Iliria College better as a result of KLP assistance?  Legal Clinic’s Intake Forms  Legal Clinic’s assessment sheet/forms  Legal Clinic’s information forms   KIIs with USAID, National Center for State Courts, University of Pristina and Iliria College law professors  Mini-survey of law students  On-site observation of the activities of KLP  Opinions elicited during the KIIs  Quantitative analysis of the results of the mini-survey  Opinions elicited from on-site observations KLP Question 2 2.1 How has the academic research capacity at the Law Faculty of the  Any available scholarly research from faculties  KIIs with USAID, National Center for  Opinions elicited during the KIIs and on-site observations 8 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis University of Pristina improved as a result of the KLP assistance?  Faculty Handbook  Faculty newsletters  Dean’s Annual report (s)  Director of Research reports State Courts, University of Pristina and Iliria College law professors  On-site observation of the activities of KLP  KLP Question 3 3.1 What have been the effects of the KLP support for the Iliria College?  Student Handbook  Faculty Handbook  Admissions Criteria  Curriculum information  KIIs with USAID, National Center for State Courts, and Iliria College Dean of Law Faculty and law professors  Mini-survey with Iliria College law students  On-site observation of the activities of KLP  Opinions elicited during the KIIs and on-site observations  Quantitative analysis of the results of the mini-survey KLP Question 4 4.1 How has KLP contributed to improving the functioning of the Mandatory Continuing Legal Education and the disciplinary system at the Kosovo Chamber of Advocates?  Available CLE curriculum  CLE Registration and attendance forms  Available data on disciplinary activities  KBA regulatory framework on CLE and Disciplinary System  KBA Training Center Plans and Curricula  KIIs with USAID, National Center for State Courts, and KLP staff  KIIs with key representatives of KBA  On-site observation of the activities of KLP  Mini-Survey/KBA  Opinions elicited during the KIIs and on-site observations  Quantitative analysis from answers received from Mini￾Survey 9 | Page EROL and KLP Evaluation Design Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis  FGDs with KBA members KLP Question 5 5.1 How has gender and ethnic diversity improved within the Kosovo Chamber of Advocates membership as a result of the KLP assistance?  Membership Criteria  Membership forms and follow-up data  Mission statement(s)  Newsletters  Annual report  KIIs with USAID, National Center for State Courts, and KLP staff  KIIs with key representatives of KBA  On-site observation of the activities of KLP  FGD with members of KBA  Opinions elicited during the KIIs and on-site observations  Responses and comments received from FGD ANNEX C: SCHEDULE OF MEETINGS AND INTERVIEWS BY SUB￾EVALUATION TEAM SCHEDULE FOR MARY NOEL PEPYS & ROLAND GJONI MARCH 17 – APRIL 7, 2014 Name, Title and Contact Info Time Monday 17.03.2014 USAID TEAM Tanya Urquieta, Director of Democracy & Governance Office Matthew Carpenter, Program Office Washington DC Gresa Caka, Project Management Specialist DGO office Melita Cacaj, Monitoring & Evaluation Specialist 10:00 am -11:30 am Harold D. Dampier Jr., Chief of Party KLP Petrit, National Local Advisor 03.00 pm -05:30 pm Tuesday 18.03.2014 Kelly Gavanagh, Chief of Party of EROL Andrea Muto, Deputy Chief of Party of EROL 09:00 am -12:30 am 01:30 pm -05:00 pm Wednesday 19.03.2014 Anne Trice, Court Administration Advisor of EROL Kelly Gavanagh, Chief of Party of EROL 10:15 am -12:00 am Ehat Miftaraj, Director of Unit for Evaluation of Persecutors Performance 01:00 pm -02:00 pm Gazmend Citaku, Acting Head of the Department of International Legal Cooperation of the MoJ 02:30 pm - 03:30 pm Feke Sadikaj, State Advocacy Office 03:30 pm -04:30pm Thursday 20.03.2014 Nexhmedin Kelmendi, Deputy Head Legal Unit KCC Radomir Ljuban, Legal Advisor at Constitutional Court Sevdail Kastrati, Legal Advisor at Constitutional Court 09:00 am -10:15 am Muharrem Mustafa, Acting Director of the Legal Office of the President Selim Selimi, Legal Advisor to the President 10:30 am -12:00 am Enver Fejzullahu, Model Court Specialist Feriz Berisha, Appeal Court Administrator 02:30 pm -04:30 pm Friday 21.03.2014 Fatmir Rexhepi, KJC Head of ICT 09:00 am -10:00 am Enver Peci, KJC Chairman Astrit Hoti, Acting Director Legal Department KJC 10:00 am -11:00 am Albert Avdiu, Director of the KJC Secretariat 11:00 am -12:30 am Besnik Ramosaj, KJC Head of Statistics Department 02:00 pm -03:15 pm Muhamet Kurtishaj, Director KJC Model Courts Engineer 03:15 pm -04:30 pm Saturday 22.03.2014 Visar Morina, Legal Advisor of EROL Mark Hugh, Special Technical Advisor Andrea Muto, Deputy Chief of Party of EROL Enver Fejzullahu, Model Court Specialist 11:00am – 02:00 pm Monday 24.03.2014 Hajredin Morina, Court Administrator Malishevo Branch Liman Javori, Senior Assistant / Data Entry 09:30 am – 10:00 am Malishevo Branch Court Tour 10:00 am – 10:30 am Rahovec Branch Court Tour 10:30 am – 11:00 am Meeting at Rahovec Branch Court Jonuz Bugari, Civil Law Judge 11:00 am – 11:45 am Focus Group Discussion (Basic Court and 2 Branch Court Staff) Manushaqe Hasanaj, Court Statistics Rahovec Branch Rudin Elezi, Court Administrator Malishevo Muharrem Perboqi, Ass. Administrator Rahovec Branch Hajredin Morina, Ass. Administrator Rahovec Branch Hektor Vula, Head of Registry Liman Javori, Civil Registry Clark Bujar Sylka, Civil Registry Clark Nexhibe Balaj, Head of Judical Support Office Haxhi Hoti, Ass. Administrator Suha Reka 01:00 pm – 02:30 pm Miziri Shabanaj, Supervising Judge Rahovec Branch 02:30 pm - 03:00 pm Group Discussion at central case registration office Rahovec Ivana Bojic, Data Entry Officer Afrim Gashi, Senior Data Entry Officer Murat Derguti, Archive Officer Samedin Tahiraga, Translator A/S Shkelzen Zekaj, Interim 03:00 pm - 03:30 pm Mark Hugh, Special Technical Advisor 04 30 pm - 06:00pm Tuesday 25.03.2014 Sahit Krasniqi, Vice Court President Ferizaj Basic Court Salih Topalli, Chief of case Management Office Milaim Avdiu, Criminal Case Clerk 09:30 am – 10:45 am Ferizaj Basic Court Tour 10:45 am - 11:00 am Gjilan Basic Court Tour 01:00 pm – 01:30 pm Zydhi Haziri, Court President Gjilane Basic Court 01:30 pm – 02:30 pm Focus Group Discussion (Basic Court and 2 Branch Court Staff) Sabit Shkodra, Public Information Officer Eroll Imami, Court Statistics Shemsijen Kadriu, Court Admin. Basic Court Ferizaj Valbona Dulahu, Registr Officer Basic Couirt Ferizaj Branch Sali Topalli, Head of Registr Basic Couirt Ferizaj Branch Sadik Limani, Civil Registry Clark Hatixhe Shaqiri, Civil Registry Clark Baki Sylejmani, Criminal Reg. Clark Feriza Basic Court Milaim Avdiu, Criminal Reg. Clark Feriza Basic Court Bekim Salihu, Senior Officer Qazim Shabani, Administrative Clerk 02:30 pm – 03:30 pm Wednesday 26.03.2014 Vushtrri Branch Court Kada Bunjaku, Court President 09:00 am – 10:00 am Skenderaj Court Tour Astrit Dibra, Assistant to Administrator Basic Court Mitrovica Branch Skenderaj 10:30 am – 11:00 pm Isa Shala, Supervisory Judge 11:00 am – 12:00 pm Pristina Basic Court Hamdi Ibrahimi, President Judge 01:15 pm – 02:15 pm Focus Group Discussion at EROL Office Arber Jashari, Public Information Officers Court of Appeals Sejdi Sadiku, Court Administrator ferizaj Basic Court Lumnije Surdulli, Court Administrator Mitrovica Basic Court Jna Gashi, Court Administrator Supreme Court Bajram Rama, Assistant Administrator Branch of Gllogovac Astrit Dibra, Assistant Administrator Branch of Skenderaj Nysret Arifi, Head of reg. of Chief of File Reg. Court of Appeals Ismet Berisha, Mitrovica Basic Court Florije Krasnioqi, Civilian registry Clerk Supreme Court Mimosa Paqarada, Civil registry Clerk Court of Appeals Hakif Sfishta, Criminal Registry Clerk Court of Appeals 02:30 pm – 03:30 pm Albert Avdiu, Director of the KJC Secretariat 04:30 pm – 06:00 pm Thursday 27.03.2014 USAID team Tanya Urquieta, Director of Democracy & Governance Office Matthew Carpenter, Program Office Washington DC Gresa Caka, Project Management Specialist DGO office Melita Cacaj, Monitoring & Evaluation Specialist 08:45am - 09:45 am Bedri Bahtiri, Advisor to the MOJ 10:00 am – 11:00 am Anita Mihailova, Key Expert Support to the KJC and the KPC Project Rr Perandori Dioklecian 11:30 am – 01:30 pm Adriana Blanksma Ceta, EULEX Advisors to the KJC Yordan Sirakov 02:30 pm – 04:30 pm Friday 28.03.2014 Edis Erkic, Norway CMIS Director 10:30 am – 12:00 pm Salij Mekaj, President Judge Court of Appeals Arber Jashari, PIO 01:30 pm – 03:00 pm Judge Biljana Rexhiq Prosecutor Laura Pulaj Forum of Women Judges and Prosecutors Court of Appeals 03:30 pm – 04:30 pm David Greer, Chief of Party Contract Law Enforcement Program 05:00 pm – 06:30 pm Monday 31.03.2014 Kelly Gavanagh, Chief of Party of EROL 02:30 pm – 03:00 pm Theobald Volkmar, GIZ Project Director 04:00 pm – 05:00 pm Thursday 03.04.2014 Enrico Visentin, Task Manager Rule of Law/Cooperation Section 09:00 am – 10:00 am Supreme Court Chief Justice 01:30 pm – 2:30 pm SCHEDULE FOR NANCY NTI ASARE & VIGAN QOROLLI MARCH 17 – APRIL 6, 2014 Name, Title and Contact Information Time Monday 17.03.2014 USAID TEAM Tanya Urquieta, Director of Democracy & Governance Office Matthew Carpenter, Program Office Washington DC Gresa Caka, Project Management Specialist DGO office Melita Cacaj, Monitoring & Evaluation Specialist 10:00 am -11:30 am Harold D. Dampier Jr, Chief of Party KLP Petrit, National Local Advisor 03.00 pm -05:30 pm Tuesday 18.03.2014 Kelly Gavanagh, Chief of Party of EROL Andrea Muto, Deputy Chief of Party of EROL 09:00 am -12:30 am 01:30 pm -05:00 pm Wednesday 19.03.2014 Veton Dula, Spokesperson of CC 10:00 am – 11:00 am Bashkim Percuku, Head of CC IT Office 11:00 am – 11:30 am Wednesday 19.03.2014 EROL Staff Chuck Ericksen, Lindita et al, Comp IV 09:00 am – 12:00 am Donika Podrima, ATRC Staff 13:00 pm – 14:30 pm Lavdim Krasniqi, KJI Director 15:00 pm – 17:00 pm Thursday 20.03.2014 Liridona Kozmaqi Spokeperson, Office of special persecutor in Kosovo Ekrem Lutfiu, KPC Spokesperson. 09:00 am – 10:00 am Aishe Qorraj-Berisha, KJC Spokesperson, Public Information Officer 10:00 am – 11:30am Bajram Ukaj, Dean of UPLF, Professor 12:30 pm – 01:30am Mixhait Reqi, President of Illiria College Arber Reqi, Administration Director Illiria College Shpresa Ibrahimi, Lecturer of Law 14:00 pm – 17:00 pm Friday 21.03.2014 Chick Erickson Andrea Muto, Deputy Chief of Party of ERoL Kelly Gavanagh, Chief of Party of ERoL, 045434974 Fatmir Kutllovci, KJI Judicial T raining Specialist 09:00 am – 12:00pm Albulena Sadiku, NGO BIRN, Kaltrina Hoxha, NGO BIRN, Petrit Skenderi, UNDP (past NCSC) 14:00 pm – 15:30pm 16:00 pm – 17:00 pm Monday 24.03.2014 Adem Vokshi, KBA Disciplinary Committee, 11:00 am – 12:00pm Ibrahim Dobruna, KBA President, Yllë Zekaj, KBA Executive Director 12:00 pm – 16:00pm Tuesday 25.03.2014 PEJA Focus Group Discussion with Journalists Halil Gashi, Journalist Marin Kelmendi, Journalist 09:00 am – 11:30 am PRIZREN Focus Group Discussion with Journalists Qazim Thaqi, Journalist Refki Reshitaj, Journalist 13:00 pm – 15:30 pm Wednesday 26.03.2014 Shpresa Rama, Chairman of Committee for Gender at KBA Xhevdet R. Rama, Avokat, Lawyer, KBA 09:00am – 10:00am Patrick Reilly & Jasmin Gojani, OSCE 13:00 pm – 14:00pm Rezearta Reka & Virgjina Dumnica, UNDP 14:00 pm – 15:30pm Thursday 27.03.2014 USAID TEAM Tanya Urquieta, Director of Democracy & Governance Office Matthew Carpenter, Program Office Washington DC Gresa Caka, Project Management Specialist DGO office Melita Cacaj, Monitoring & Evaluation Specialist Focus Group Discussions with UPLF Professors Bajram Ukaj, Dean UPLF and Professor Avdullah Aliu, Professor Civil law Shaban Çareti, Professor French Language Bleta Brovina, Teaching Assistant Qerim Qerimi, Professor Vice Dean 08:45am – 09:45am 10:00am – 12:00 am Friday 28.03.2014 Jehona Lushaku, Advisor to Minister of Justice 10:00 am – 11:30 am Focus Group Discussion with professors at Illyria College Mixhait Reçi, Dean Riza Smaka, Professor Hamdi Podvorica, Professor Evzi Hani, Carrier Center Donikë Dobruna, Assistant Diellza Koliqi, Assistant Shpresa Ibrahimi, Lecturer of Law 03:00 pm – 14:30 pm Monday 31.03.2014 Katya Dormisheva, KJI (Team Leader for IRZ) Jean-Jacques Heintz, Visiting Judicial Expert KJI Tanya Temekjisjam, Short term expert KJI 10:00am-11:00 am Pristina Journalists Rabisha Muhaxhiri, RTK Blerim Sopi, Radio Kosova Mendohije Krelani, Gazeta Tribuna 13:00 pm – 15:00 pm EROL Staff Volkmar Theobald, GIZ, Project Manager-Legal Reform Project Flakron Sylejmani, GIZ, Senior Legal Advisor, Legal Reform 14:00 pm – 16:00 pm 16:00 pm – 17:15 pm ANNEX D: LIST OF DOCUMENTS REVIEWED List of Documents Reviewed PROVIDED BY USAID EROL Annual Report (Y2) (April 2012-Mar 2013) EROL 110729 EROL Q1 PMEP Report FINAL EROL Annual Report (Y1 April 2011- March 2012) EROL Q.1 Quarterly Report (April 1-June 2012) EROL Q.1 Quarterly Report (April-June 2011) EROL Q.2 Quarterly Report (July-Sept 2011) EROL Q.2 Quarterly Report(July-Sept 2012) EROL Q.3 Quarterly Report ( Oct- Dec 2011) EROL Q.3 Quarterly Report (Oct-Dec 2012) EROL Q1 PMEP Report FINAL EROL Quarterly - year 2 quarter 3 - oct to dec 2012 - kg 10 jan with pmp EROL Quarterly - year 3 quarter 1 - April to June 2013 - kg EROL Quarterly Report - Year 2 Quarter 3 - Oct to Dec 2012 EROL Quarterly Report - Year 3 Quarter 2 - July to September 2013 EROL Y1 Workplan EROL Y2 Work Plan EROL Y3 Work Plan EROL Year 2 (April 2012 - March 2013) PMEP Report EROL_Q2_PMEP_Report EROL_Q3_PMP_Report EROL Y4 Work Plan (Draft) NCSC Award- KLP_AID-167-A-12-00001 NCSC KLP Program Year 1 PMP Report (January 30, 2013) NCSC KLP Program Year 2 PMP Report (January 30, 2014) NCSC KLP QR1 January-March 2012 NCSC KLP QR2, April-June 2012 NCSC KLP QR3, July-September 2012 NCSC KLP QR5 January-March 2013 _30 April 2013_ NCSC KLP QR5 January-March 2013 - Annexes (30 April 2013) NCSC KLP QR6 April-June 2013 - Annexes (30 April 2013) NCSC KLP Year 1 Report (January-December 2012) -- January 30, 2013 NCSC KLP Year 1 Work Plan April 27 2012 Final.Approved NCSC KLP Year 2 Report (January-December 2013) -- January 30, 2014 FINAL NCSC KLP Year 2 Report (January-December 2013) -- January 30, 2014 FINAL ANNEXES NCSC KLP Year 2 Work Plan Final Track Changes (080213 approved version) LEGISLATION Law No. 03/L-199 on Courts Law No.03/L –223 on Kosovo Judicial Council Law No.03/L –224 on Kosovo Prosecutorial Council Law No.03/L –225 on State Prosecutor Law No. 04/L-193 Law on the Bar MISCELLANEOUS Kosovo Judiciary Strategy 2014-2018 (Draft) KPC Institutional Strategy 2014-2016 (Draft) National Backlog Reduction Strategy adopted on 16 September 2013 KJC’s ICT Strategy 2012 – 2017 Kosovo Progress Report issued by EU Commission on 16 October 2013 ANNEX E: MINI SURVEY AND FOCUS GROUP DISCUSSION QUESTIONNAIRES MINI-SURVEY AND FOCUS GROUP DISCUSSION with COURT STAFF TRAINED BY EROL March 24-26, 2014 1. In how many EROL trainings have you participated? None____ One_____ Two _____ Three____ 2. In which EROL trainings did you participate? Electronic Case Registries ___ Model Court Standards Court Refurbishment Process _____ Court Management Skills for Court Administrator’s _____ Electronic Time Stamp____ Training of Court Statistic Officers, IT Officers, Case Management Office Heads_____ Training on Database Entry_____ Training on New Case Registry ____ Change Management Leadership Training for Court Administrators___ 3. How do you rate the quality of EROL training provided to you? Very Good ____ Good ____ Satisfactory ____ Poor ____ 4. What were the best aspects of the EROL training? Check the applicable. Interactive teaching methodology ____ Written Materials provided during the training ___ Quality of the Instructors ___ Location of the training ___ Number of trainees in the training ___ 5. Which specific skills did you learn at the EROL trainings? Computer skills ____ Automated case registry and database entry ___ Court statistics and reporting ____ Court standards ____ Leadership skills ____ Other ___ 6. Would you say that you are you applying the skills learned through EROL’s training in your daily job? Strongly disagree____ Disagree___ Agree____ Strongly agree___ Please specify your answer if you wish: 7. What were the deficiencies in the EROL training and technical assistance? Please summarize. 8. What improvements could be made to the EROL training and technical assistance? Please summarize. 9. How have you benefited in your work from the EROL trainings and technical assistance you have received? Check the applicable. Ability to perform responsibilities at a higher level of aptitude___ Increased understanding of the responsibilities of other court staff ____ Increased ability to work as a team with the other court staff _____ Increased knowledge of court operations ___ Increased understanding of the court’s role to provide services to the public___ Increased ability to provide better services to the public ___ 10. How have the courts been strengthened and court operations improved as a result of the model court refurbishment, technical assistance and trainings? Yes___ Somewhat ___ No ___ Please explain: 11. Has there been any emphasis on gender considerations in the training and technical assistance you have received from EROL? Yes ___ No ___ 12. What is EROL’s greatest contribution to the court system? QUESTIONS TO BE ADDRESSED TO THE COURT PIOS: 1. Have you participated in any of the EROL trainings? Yes ___ No ___ If not please specify why? 2. How do you rate the quality of the trainings and support provided to your institution in general by EROL? Excellent ____ Good _____ Weak _____ Very poor____ 3. What new skills have you acquired during these trainings and how have they been implemented at your institution? Please summarize. 4. How have EROL activities enhanced citizens’ awareness of the justice system? Please summarize. 5. How have EROL’s activities increased citizens’ role in ensuring the delivery of justice? Mini Survey for Law Students March 2014 Please use back of sheet if you need additional space for your answers. 1. How has your legal training assisted you in achieving your goals? 2. How would you improve law school effectiveness? 3. How has the Career Development Center (CDC) served you? 4. Have you attended any Legal Clinics and what was your experience? 5. Are you male or Female? Focus Group Discussion for Professors 1. What do you perceive the strengths and weakness of the legal education system? 2. What components of the current activities must be expanded, replicated, or scaled up? Which should be discontinued? What are the gaps and duplications in the overall effort of legal education? 3. How should international support best be provided? 4. Which current (international) organizations/programs are the most effective in the legal education area? Why? 5. What are the available opportunities for expanding clinical education at the law school? FGD QUESTIONS WITH THE PUBLIC RELATIONS OFFICE OF THE KJC, KPC 1. Have you participated in any of the EROL trainings? 2. What new skills and/or impressions you acquired from EROL Objective Four and how should they be implemented at your institution? 3. How do you rate the quality of the EROL Objective Four provided to your institution in general? 4. What are the best methods used during the EROL trainings? 5. How the brochures produced by EROL for the public opinion enriched generally the belief of the public to the judiciary institutions? 6. What lessons learned can be used further from the USAID programs in near future, and what are the biggest needs in your field of work? 7. What are the greatest advantages and weaknesses from the EROL’s implementation of Objective Four? FGD QUESTIONS WITH MEMBERS OF THE KBA What is your assessment of the USAID-KLP’s assistance so far? What type of USAID assistance has been effective or ineffective and why? How has KLP contributed to improving the functioning of the MCLE and the disciplinary system at the KBA? What kind of future support is needed from USAID to ensure efficient service delivery to members of MCLE? How has gender and ethnic diversity improved within the KBA membership as a result of the KLP assistance? In your view, in what ways would USAID assistance to KBA become sustainable in the longer term? ANNEX F: LIST OF REGULATIONS TO BE ADOPTED BY KJC AND THEIR STATUS AS OF MARCH 31, 2014 List of Regulations to be adopted by KJC and their Status as of 31 March 2014 No. Institution building measure Date approved Implementation of the Law on Courts (2011- 2013) 1. Regulation on internal organization of the Basic Court (art.25); Regulation on internal organization of the Court of Appeals (art.20.3, 25); Regulation on internal organization of the Supreme Court (art.21.7, 25); Regulation of the General Sessions of the Supreme Court (art 23, 25); Regulation on assignment of cases to departments of Basic Courts (art.12.5) Regulation on Internal Organization of the Courts adopted on 4 January 2012; 2. Regulation on publication of decision of Court of Appeal and Supreme Court (art.19.2, 24) Not yet adopted. 3. Regulation on determination of qualification for the appointment of judges and lay judges (art.26) Regulation on the Procedure for Proposal for Appointment and Re-Appointment of Judges adopted on 11 May 2013 and amended on 10 June 2013. 4. Regulation on definition of professional associates and judicial trainees (art.33) Regulation on the Procedure for recruitment of Court Interns adopted on 2 November 2012; ( Regulation is not published yet at the KJC website); Implementation of the Law on KJC 5. Regulation and procedure for the discipline of Council members (art. 13.1) Not yet adopted. 6. Regulation on appointment of lay judges (art. 16.2) Not yet adopted. 7. Regulation on appointment of judges (art. 4.1.2) Regulation on the Procedure for Proposal for Appointment and Re-Appointment of Judges adopted on 11 May 2013 and amended on 10 June 2013. 8. Regulation on performance assessment of judges (art. 19.1) Regulation on the Evaluation of Performance of Judges adopted on 22 February 2012 and amended on 2 May 2013 and on 10 June 2013. 9. Regulation on performance assessment of lay judges (art. 19.1) Not yet adopted. 10. Regulation on transfer of judges (art. 4.1.5), including regulation on appeals against transfer decisions (art. 20.6) Not yet adopted. 11. Regulation on the President Judge responsibilities (art. 24.1) Included in the KJC Regulation on Internal Organization of the Courts, adopted on 4 January 2012. 12. Regulation on KJC-Secretariat internal organization and functioning (art. 27.9) Not yet adopted (drafting in process – discussed by the KJC Normative Committee on 2-6 October 2013) 13. Regulation on the organization and functioning of the Office of the Disciplinary Counsel (chapter VII.) Not yet adopted. 14. Rules, regulation and policy directives on the functions of Court administrators (art. 31.3) Included in the Regulation on Internal Organization of the Courts – 4 January 2012; No. Institution building measure Date approved 15. Regulation on disciplinary procedure (chapter VI) Regulation for Disciplinary Committee Performance adopted on 27 December 2013 16. Code of Professional Ethics of Council members (art. 4 1.16) Code of Ethics and Professional Conduct for the KJC members adopted on 8 October 2012. 17. Code of Professional ethics for court staff (art. 4 1.18) Code of Conduct for Judicial Administration Personnel adopted. 18. Regulation on ensuring representation and recruiting of judges from the non-majority communities (art. 4 1.3 and Feasibility Study requirement) Not yet adopted 19. Regulation on determining the number of judges in courts and the number of court staff (art. 4 1.13) Not yet adopted. 20. Regulation on management and disclosure of information retained by the Kosovo judiciary (art. 4 1.23) Not yet adopted. 21. Regulation on proposing of presidents of courts and approving of supervising judges for branches (art. 4 1.6 and 1.7) The Regulations in place were adopted for the purpose of implementation of the new Law on Courts and the reorganization of the judicial system of the Republic of Kosovo. 22. Regulation on removal of court presidents and supervising judges Not yet adopted. 23. Regulation on the preparation and proposal of the budget of the judiciary (art. 15) Included in the KJC Regulation for Internal Organization and Activities of the Kosovo Judicial Council adopted on 2 November 2012. 24. Regulation on conducting preparatory exam (art. 4 1.19) Not yet adopted. 25. Policies and strategies for the efficient and effective functioning of the courts (art. 21) ICT strategy for the years 2012-2017 adopted on 9 March 2012: KJC Strategic Plan and the ne Backlog Reduction Strategy adopted 6 September 2013; KJC Strategic Plan for the Judiciary ( 2013-2018) – Not yet adopted but deliberation close to completion as of March 2014 26. Policies, standards and directives for regulating the training of judges, lay judges and staff (art. 50) Not yet adopted. 27. Regulation on security of Courts (Art 30, and Feasibility Study requirement) SoPs for the Security of Judges and Courts and the Administrative Instruction on the Installation and use of the Security Cameras and maintenance of Video Recordings adopted on 10 June 2013; On 28 June, the KJC has signed an MoU with the Kosovo Police for providing the required protection to judges in cases of threats against them. Implementation of the Criminal and Criminal Procedure Code (KJC obligations) 28. Establishment of the Conditional Release Panel in accordance with the Law on the Execution of Penal Sanctions (art. 94.4 Criminal Code) Regulation for Organization and Function of the Conditional Release adopted on 22 February 2013. 29. Establishment of proceedings for compensation, rehabilitation and the exercise of other rights of persons who have been convicted or arrested Regulation on the Procedure for Compensation of Damage to Persons Convicted or Arrested Without Reason adopted 2 November 2012 (not yet No. Institution building measure Date approved without justification (arts. 526, 527 and 533 Criminal Procedure Code). published at the KJC website) 30. Establish a reliable criminal records database, including on the basis of mutual legal assistance in criminal matters (art. 491 Criminal Code and Visa Liberalization Roadmap) Not yet adopted. 31. Maintenance of Order and Media in the Courtroom (Art. 301 of the Criminal Procedure Code) Guidelines on Recordings during the main trials adopted on 9 September 2013. ANNEX G: DISCLOSURE OF CONFLICT OF INTEREST DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Mary Noel Pepys Title Team Leader Organization ME&A Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) AID-RAN-I-00-09-00018/AID-167-TO-14-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Effective Rule of Law (EROL) Program and Kosovo Legal Profession (KLP) Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 4/14/2014 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Roland Gjoni Title Court Administrator/Justice Sector Analyst Organization ME&A Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-RAN-I-00-09-00018/AID-167-TO-14-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Effective Rule of Law (EROL) Program and Kosovo Legal Profession (KLP) Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Signed Electronically Date 4/17/2014 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Nancy Sharp Nti Asare Title Legal Education Analyst Organization ME&A Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-RAN-I-00-09-00018/AID-167-TO-14-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Effective Rule of Law (EROL) Program and Kosovo Legal Profession (KLP) Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Signed Electronically Date 4/21/2014 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Vigan Qorrolli Title Local Legal Sector Specialist Organization ME&A Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-RAN-I-00-09-00018/AID-167-TO-14-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Effective Rule of Law (EROL) Program and Kosovo Legal Profession (KLP) Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Signed Electronically Date 4/14/2014