EVALUATION OF USAID/GEORGIA’s JUDICIAL ADMINISTRATION AND MANAGEMENT REFORM (JAMR) PROJECT Final Report October 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Howard Fenton and Ekaterine Popkhadze through Social Impact. 2 EVALUATION OF USAID/GEORGIA’s JUDICIAL ADMINISTRATION AND MANAGEMENT REFORM (JAMR) PROJECT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available in printed or online versions. Documents ar e also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc. to the United States Agency for International Development under USAID Contract No. RAN-I-00-09-00019-00. SI’s Evaluation of USAID/Georgia’s JAMR Project i TABLE OF CONTENTS ACRONYMS ................................................................................................................................ II EXECUTIVE SUMMARY ........................................................................................................... 1 Background ................................................................................................................................. 1 Evaluation Methodology ............................................................................................................. 1 Findings ....................................................................................................................................... 2 Conclusions ................................................................................................................................. 2 Recommendations ....................................................................................................................... 3 INTRODUCTION ......................................................................................................................... 1 Country Context and Background of Project .............................................................................. 1 Project Implementation ............................................................................................................... 1 Evaluation Purpose and Methodology ........................................................................................ 2 Limitations of Methodology ....................................................................................................... 3 FINDINGS ..................................................................................................................................... 3 Component 1 ............................................................................................................................... 4 Component 2 ............................................................................................................................... 7 Component 3 ............................................................................................................................... 8 Component 4 ............................................................................................................................... 8 CONCLUSIONS .......................................................................................................................... 10 Component 1 ............................................................................................................................. 11 Component 2 ............................................................................................................................. 12 Component 3 ............................................................................................................................. 12 Component 4 ............................................................................................................................. 13 RECOMMENDATIONS ............................................................................................................ 13 APPENDIX A. SCOPE OF WORK ....................................................................................... I APPENDIX B. PERSONS CONTACTED ........................................................................... V APPENDIX C. INTERVIEW QUESTIONS ...................................................................... IX JAMR Interview Questions: First Meeting ............................................................................... IX JAMR Interview Questions: Second Meeting .......................................................................... XII JAMR Interview Questions: Practitioners ............................................................................. XVI JAMR Interview Questions: International Organizations .................................................... XVIII JAMR Interview Questions: Department of Common Courts ................................................. XX JAMR Interview Questions: High Council of Justice ............................................................ XXI JAMR Interview Questions: Judicial Actors ........................................................................XXIV APPENDIX D. CLEARANCE RATES ACROSS JAMR PROJECT YEARS ......... XXVI APPENDIX E: TIMELINE OF JAMR ACTIVITIES ................................................ XXIX ii ACRONYMS CMS Case Management System DCC Department of Common Courts HCoJ High Council of Justice HSJ High School of Justice JAMR Judicial Administration and Management Reform JILEP Judicial Independence and Legal Empowerment Project MoF Ministry of Finance MoJ Ministry of Justice NGO Non-Governmental Organization PMP Performance Management Plan RoL Rule of Law SC Supreme Court SI Social Impact SOW Scope of Work UNDP United Nations Development Programme USAID United States Agency for International Development SI’s Evaluation of USAID/Georgia’s JAMR Project 1 EXECUTIVE SUMMARY Background The Georgian judicial system has undergone significant reforms since the Rose Revolution in 2003, including the replacement of many older and allegedly corrupt judges, an increase in judicial salaries, a consolidation of multiple regional courts into 20 “unified” courts, the refurbishment or reconstruction of courthouses, and numerous legislative changes in court procedures. However, lingering court management and administration issues continued to create significant case backlogs and lengthy case processing times, leaving litigants frustrated and dissatisfied. The Georgian courts began to address these issues in 2005 and they became the focus of the four-year, $3,702,000 United States Agency for International Development (USAID) Judicial Administration and Management Reform (JAMR) project. Specifically, JAMR addressed case and court management issues through four components: (1) improvement of court operations; (2) establishment of professional court administrators; (3) strengthened budgeting capacity of the judiciary; and (4) public outreach. Purpose of the Evaluation In August 2011, the USAID/Georgia Mission contracted Social Impact (SI) to conduct a final performance evaluation of JAMR to help the Mission take stock of the effectiveness and relevance of project activities, and identify and document lessons learned to enhance the management and substance of current and future Rule of Law (RoL) interventions. Specifically the evaluation had three major components: 1. Assess the effectiveness of JAMR in achieving its initial project objectives; 2. Evaluate the relevance of JAMR methodology for providing assistance and monitoring its effectiveness in relation to the project objectives; and 3. Provide recommendations for improving the management and substance of other USAID￾funded RoL interventions and identify components of JAMR requiring future follow-up.1 Evaluation Methodology SI utilized a mixed-methods approach for this evaluation. This allowed the evaluation team to move past a mere quantitative comparison of results with targets to a more in-depth study that examined how effective stakeholders and beneficiaries perceived JAMR interventions. It also permitted the team to identify challenges faced and lessons learned, and assess the overall sustainability of project results. The evaluation employed several rapid appraisal methods, including document review, key informant interviews, and stakeholder group interviews. Prior to departure for the field, the team performed a detailed document review of JAMR’s original and amended Scope of Work (SOW), Performance Management Plans (PMPs), annual and quarterly reports, and other materials supplied by USAID/Georgia. The evaluation team then spent two weeks in Georgia conducting semi-structured key informant and group interviews utilizing audience-specific interview guides. Sampling methods were purposive as opposed to random due to the limited number of relevant 1 The full Evaluation Statement of Work is provided in Appendix A. 2 project participants and the heightened chance that results from a sampling approach would be skewed given a relatively small population.2 Findings JAMR was beset by a major obstacle at its inception when the key counterpart, the Supreme Court (SC), indicated that its top priority was the adoption and implementation of a sophisticated electronic case management system (CMS) and that all other aspects of the project were much lower in priority. This led to a compromise which involved reallocating $500,000 of the $3.7 million project budget from other components to Component One to allow the Court to support its CMS. Despite this initial difficulty, there was a significant improvement in the management of cases by the Georgia courts over the four years of the project. Case backlogs virtually disappeared and the processing time for all cases was significantly reduced. Surveys indicated that court users’ satisfaction with the procedures and management of the courts somewhat increased, and interviews confirmed similar degrees of satisfaction among lawyers, including those with misgivings about the decisions of the courts. A number of factors contributed to this improvement in case flow management, including significant changes in civil and criminal procedures regarding shorter processing times, telephone and public notice, and plea bargains. JAMR support through better statistical information, trainings and workshops also contributed. The SC’s CMS was developed and became operational with some JAMR assistance and training. It is being implemented at the same time as the Ministry of Justice’s (MoJ) similar system for prosecutors and judges in criminal cases; this simultaneous rollout will create integration issues. JAMR procured new audio recording equipment for 80 courtrooms that had not yet received the systems; this enables the court to provide indexed audio recordings in place of written transcripts. This procurement was delayed, however, due to issues with the tender process. A professional court management program was started with strong and effective JAMR support that led to the creation of a new court manager position for the unified courts that in turn resulted in the transfer of administrative responsibilities from presiding judges to the new court managers. JAMR provided some support to assist the courts’ budgeting process, but this was not a significant element of the project. Improvements in public awareness of the courts and information available for the public at the courts were contributions of JAMR, in conjunction with the courts’ heightened appreciation of the importance of public satisfaction with the courts. An information desk at the very busy Tbilisi City Courts highlighted this component. Conclusions While JAMR activities such as identifying reasons for delays and extensive case management trainings played some role in reducing case backlogs and processing time, the extent of the project’s influence on these factors is impossible to measure. Based on interviews with judges and court personnel, the evaluation team concludes that changes to the procedural rules and the advent of plea bargaining likely played a greater role. The project’s role in the establishment of professional court managers was more significant and is probably its greatest impact. JAMR had little impact on the court budget process beyond training and management suggestions to the 2 A list of persons contacted may be found in Appendix B; interview protocols keyed to various interviewee groups are provided in Appendix C. SI’s Evaluation of USAID/Georgia’s JAMR Project 3 administrative office of the courts. Training, materials, and assistance in presentation of information about the courts were useful and effective, in conjunction with strong court initiatives. JAMR had difficulties with its startup due to serious miscommunication with the Supreme Court and administrative problems within the project, such as an 18-month delay in the procurement of the audio recording systems. The focus of caseflow efforts on numbers of cases processed may have detracted from the quality of the judges’ decision-making, an issue only addressed at the conclusion of the project. Recommendations Based on the evaluation findings, and the conclusions drawn from them, the evaluation team recommends that: 1. The focus of the Georgian judiciary on case numbers should be closely monitored to ascertain its impact on quality decision-making and decisions. The new USAID Judicial Independence and Legal Empowerment Project (JILEP) is in a good position to do this with its qualitative focus on judicial performance. 2. The Georgian judiciary has demonstrated a serious commitment to improving both its administration and its public outreach. JAMR took advantage of this by providing extensive trainings and workshops. This commitment by the courts should be nurtured and supported through ongoing assistance and continued recognition of the importance of their efforts. 3. USAID should closely monitor changing attitudes and circumstances among anticipated counterparts. In-country developments that are moving quickly should be within the purview of USAID staff supporting the project and the initial work plans should reflect the most current disposition of essential counterparts. 4. Efforts should be made to monitor the courts’ use of audio recording equipment for criminal trials and hearings. Questions remain about why the system is not being used for those and administrative violation cases. 5. Integration of the three case management systems is important to achieve a case management system accessible to litigants and lawyers on all sides of the disputes. Neither of the main systems currently allows defense attorneys access, which furthers the perception of a skewed criminal justice system. 6. Projects whose activities include significant numbers of trainings, study trips, round tables, etc. should develop evaluative processes to obtain some measure of feedback from the participants about the events. While such processes may be only crude measures, they provide broad impressions and reactions that can alert projects to major problems in the events. SI’s Evaluation of USAID/Georgia’s JAMR Project 1 INTRODUCTION Country Context and Background of Project The Georgian judicial system has undergone significant reforms since the Rose Revolution in 2003. By 2006, Georgia had witnessed an increase in judicial salaries, the beginning of a consolidation of multiple regional courts into 20 “unified” courts,3 the refurbishment or reconstruction of courthouses, and numerous legislative changes in court procedures, resulting in a markedly different judicial landscape. However, while encouraging, these changes also highlighted the continuing power imbalance between the executive and judiciary. It is in this context that support for promoting judicial independence became an important element of USAID programming in Georgia. While judicial independence was not a direct element in JAMR, the efficiency and public perception of the courts are critical elements of an independent judiciary. One of the key challenges to sustainable judicial reform, based on different surveys and analysis, was the ability of Georgian judges and administrative staff to handle the flow and processing of cases through the courts. There were significant case backlogs and lengthy case processing times that left litigants frustrated and dissatisfied. The courts responded by focusing on many of these issues, collecting data on the backlogs and pursuing changes in procedure laws. Working in partnership with these courts, JAMR was started in September 2007 to address many of these same court administration issues. The four-year, $3,702,000 project was awarded to DPK Consulting, to promote judicial reform in the context of USAID’s Strategic Goal 2, Governing Justly and Democratically, in particular to: Ensure an effective and equitable justice system by: improving professional capacities and administrative and operational systems of actors and institutions; developing and implementing fair procedures; expanding access to justice; and ensuring adequate oversight, advocacy, and accountability.4 The project addressed this mandate through four components: (1) improvement of court operations; (2) establishment of professional court administrators; (3) strengthened budgeting capacity of the judiciary; and (4) public outreach. Project Implementation JAMR was beset by a major obstacle at its inception when the key counterpart, the SC, indicated dissatisfaction with the initial workplan that USAID had approved in January 20085 and components of the project (see Textbox 1).6 The SC had determined that its top priority was the adoption and implementation of a sophisticated electronic CMS and that all other aspects of the project were either of much lower priority or should be eliminated altogether so that funds could be allocated to 3 There are 36 district courts and 5 city courts in Georgia that make up the first instance courts. These are consolidated into 20 unified courts for purposes of judicial administration. There are two Courts of Appeals (Kutaisi and Tbilisi) and a Supreme Court. While the courts operate in chambers for civil, criminal, and administrative cases, there is some movement of judges between chambers as caseloads merit. 4 FY 2007-2012 Department of State and USAID Strategic Plan, May 2007 Report. 5 JAMR 1st Annual Report at 3 (November 14, 2008). Even prior to the development of the first workplan the project dropped the court facilities improvement element from the original proposal and increased somewhat the IT component after discussions with the Supreme Court. The Georgian government had already begun an extensive court facilities improvement project. 6 Please see Appendix D for a detailed timeline graphic of JAMR project activities. 2 developing the CMS. This ultimately led to a compromise with the SC negotiated by the USAID Mission Director and project CTO which involved reallocating $500,000 of the $3.7 million project budget from the other three components to Component One primarily in order to finance the acquisition of approximately 80 audio recording systems for courtrooms, a project already underway by the SC. This freed up other money budgeted by the Supreme Court for these systems to support its CMS. A revised JAMR workplan reflecting these changes was approved by USAID in July 2008. Unfortunately, the issues with the SC over the CMS were not fully resolved, and JAMR efforts to assist the SC in the assessment, design, and development of the system were rejected. The SC decided to proceed with its own project team and the role of JAMR in this function was further diminished. The delay in the development of the CMS has ramifications for the courts that may resonate for some time, as the MoJ extended its competing CMS to prosecutors’ offices and to criminal cases before the courts and the courts of appeals developed their own simplified case￾tracking and notification system. Integration of these systems will be one of the major future challenges for the court system. Operating with somewhat reduced funding for the other components, JAMR continued its slow implementation. In addition to supporting the CMS and procuring the audio recording equipment in Component One, JAMR engaged with its counterparts in the courts to conduct surveys, workshops, and trainings on both case and court management. There were six pilot courts that JAMR worked with on most of these activities, all first instance courts, located in Tbilisi, Gori, Mtskheta, Rustavi, Batumi, and Kutaisi. In Component Two (developing professional court administrators), JAMR focused on the creation of professional court managers with enhanced legal authority. Work in Component Three (strengthening budget capacity for the courts) lacked support from both the Supreme Court and the High Council of Justice (HCoJ). The SC requested that the component be eliminated completely during the negotiations with USAID and JAMR over the priority of the CMS. This component ultimately devolved into providing management and training assistance to the Department of Common Courts (DCC), the existing administrative unit of the HCoJ responsible for budget matters. Component Four (public outreach), involved assisting the courts in providing better information to the public at the courthouses and through publications and presentations, and training judges and staff on effective approaches to dealing with the public. Evaluation Purpose and Methodology With the project drawing to a close in September 2011, the purpose of Social Impact’s final performance evaluation is to assess the progress made within the Georgian judiciary in relation to JAMR objectives, evaluate the impact of JAMR initiatives on this progress, and analyze the effectiveness of JAMR’s monitoring efforts. The evaluation also seeks to identify the need, if any, for follow-on of the JAMR initiatives and to provide recommendations for improving the management and substance of other USAID RoL projects. Appendix A outlines the evaluation SOW; Appendix B provides a list of persons contacted for the evaluation; Appendix C provides a list of interview questions Textbox 1: Obstacles to Implementation during Year One (2007-08)  Oct ’07: Project begins  Dec ’07: SC suggests JAMR drop facilities improvement task and strengthen IT activities.  Dec ’07: Revised work plan approved  Feb ’08: SC requests significant project restructuring; reduce Component 1, eliminate Components 3 & 4 and allocate 75% of budget to SC for CMS.  Jul ’08: Revised plan adds $500,000 to Component 1, reducing funding for Components 2, 3, and 4.  Aug ’08: War with Russia temporarily disrupts project activities. SI’s Evaluation of USAID/Georgia’s JAMR Project 3 specific to each audience member; Appendix D presents data taken from JAMR annual reports on court operations; and Appendix E provides a timeline of JAMR activities. SI’s evaluation team consisted of our Evaluation Team Leader, Howard Fenton, and a local team member, Ekaterine Popkhadze. The team spent two weeks in Georgia conducting key informant interviews and group interviews with JAMR stakeholders, implementing partners, and counterparts. Interviews were held at four of the six pilot courts working with JAMR (Tbilisi, Mstkheta, Gori and Rustavi), with the SC, HCoJ, DCC, High School of Justice (HSJ), prosecutors, representatives of international NGOs involved in justice sector reforms, public interest attorneys representing the major free legal aid providers, leaders of the Georgia Bar Association, JAMR staff, staff of the new USAID Judicial Independence and Legal Empowerment Project (JILEP) and Mission staff. In total, 19 different interview meetings were held, and more than 70 individuals participated in the discussions. Each group was asked a series of similar questions relating to each of the four JAMR components, focusing first on developments in the judiciary over the four-year span of the project in regard to those components, then identifying external factors and the role of JAMR activities in contributing to these developments. Finally, questions were asked about the interaction of the organization or group members with JAMR over the course of the project, their level of satisfaction with that interaction, and any specific issues that arose. The evaluators also had access to all of JAMR’s annual and quarterly reports to USAID, including their PMP and assessment results, consultant reports, survey results, training materials and schedules and a variety of other materials developed by DPK in its performance of the project. Budget information beyond the initial year was unavailable and therefore the evaluation team could not examine the financial impact of the reallocation of the $500,000 to the CMS development in Component One on the other three components. Limitations of Methodology The most significant weakness of this approach is that causation cannot be attributed to project activities as directly as it could be with a more rigorous evaluation approach. However, the circumstances for this evaluation called for a quick, cost-effective design. By focusing on qualitative data, interspersed with quantitative data gleaned from court user surveys and the initial JAMR budget, the evaluators were able to get detail-rich data which they continuously analyzed and verified throughout data collection. The evaluators were able to confirm findings by probing similar topics with various stakeholders to see whether findings converge, and in instances where they did not, were able to adapt and further investigate these discrepancies. FINDINGS This section includes an overall presentation of finding regarding the slow implementation of the project and its change in scope, followed by findings presented for each component. It concludes with a special section focusing on the performance monitoring and evaluation (M&E) tools that JAMR used in reporting on its activities. Significant delays in implementation of the project 4 As detailed below in the discussion of the individual components, there were significant revisions and delays in the implementation of the JAMR project.7 The SC indicated that it was not in agreement with the overall focus and scope of the project from the outset and began requesting major changes in the funding and emphasis of the original components and workplan. The Court rejected one of the most significant projects of the second year, the CMS plan and request for proposals generated by the JAMR consultants. Thus during the first two years of the project while this disagreement was ongoing between JAMR and the SC over the development of its CMS, the activity largely consisted of commissioned surveys, workshops and training programs. The key activities among all of the components did not occur until the last two years of the project.8 This includes the procurement of the audio recording equipment, the initiation of the court manager training programs, the assistance to the Department of Common Courts (DCC) on budgeting, and the securing of information resources for the pilot courts such as the Tbilisi Court information desk and public information boards at the others. In March of 2010 JAMR proposed revisions to its agreement with USAID that reflected the changes in the project over the first two years, which presumably were accepted. Component 1: Improvement of Court Operations Case processing time in the pilot courts (and throughout Georgia) has been reduced and case backlogs have been substantially reduced. JAMR’s draft final report indicated that the backlog in pending cases was reduced from 450 cases per judge to 60, and that the case processing times declined by an average of 45 percent. Statistics kept by the SC and HCoJ and interviews with judges and court management staff confirmed these significant reductions. According to the judges and court personnel interviewed the reasons for these changes are varied. Beginning in 2005 a number of changes in the procedural laws were made that reduced the allowable time for processing civil,9 administrative,10 and criminal cases.11 Changes were also made in notice procedures allowing publication and telephone notice to parties of hearings and preliminary rulings.12 In civil and administrative cases, detailed case application and reply forms were mandated by courts that provided relevant information about the parties and about the legal basis for the claim.13 These forms both provided contact information for the parties that had been previously difficult to find and maintain, and outlined the legal theory for the judge in a concise and consistent manner. On the criminal side, the 7 A senior USAID officer indicated that the project began under the supervision of the Regional Legal Advisor rather than a CTO. It is not clear if this contributed to communication difficulties between the Supreme Court, JAMR and the Mission. 8 The second Chief of Party for the project, David Magradze, began work at the start of the third year. Magradze had experience in the government including service as a deputy foreign minister but did not have a court administration background. 9 There were significant reforms in the Civil Procedure Code adopted December 28, 2007. 10 Administrative case procedures follow the civil procedure rules. 11 Plea bargaining became a part of Georgia criminal procedure in March 2005. A new Criminal Procedure Code became effective October 2010. 12 Telephone and public notice of hearings and preliminary rulings were part of the December 2007 reforms, while conference calls for preliminary hearings and publication of other notices were permitted through the December 2010 amendments. The JAMR Survey Report “Reasons for Case Delay in Trial and Appellate Courts” (2010) indicated that the liberalized notice rules contributed to t he reduction in case delays. See JAMR Draft Final Report at 9 (September 2011). 13 The law requiring case application and reply forms was adopted December 28, 2007. SI’s Evaluation of USAID/Georgia’s JAMR Project 5 introduction and immediate embrace of plea bargaining by prosecutors and courts substantially reduced the number of cases that the criminal courts had to process.14 Prior to the initiation of JAMR, the HCoJ and SC had begun to collect data on case backlogs and processing time for cases within the courts of Georgia and assemble data on individual courts and judges. JAMR assisted in refining this data collection and analysis. The HCoJ and SC began to disseminate this information to the courts and to use it in evaluating the performance of judges. The project collected data on case clearance rates from the six pilot courts without any breakdown as to the reasons for the improved performance15. Interestingly, according to the data collected by JAMR (presented below), it appears that the clearance rate actually dropped significantly after Year 1 of the project but improved somewhat between Years 2 and 3. One possible explanation for this result could be the refinement in data collection. However, as JAMR project materials do not discuss this issue, this explanation could not be confirmed by the evaluation team. Table 1: Clearance Rates As Reported by JAMR Annual Reports Year # of Cases Filed # of Cases Disposed Clearance Rate Year 1 15884 19018 1.197 Year 2 12365 13171 1.065 Year 3 16521 18161 1.099 JAMR conducted a caseflow survey in the pilot courts that identified causes for the delay in the processing of cases. The survey, for example, identified delays or inability to serve parties as the most significant cause of the delays in case processing.16 Over the course of the project extensive case management trainings and workshops for judges and administrative staff were held and materials on case management techniques were translated or prepared for the courts, focusing to a large extent on the logjams that the surveys had identified. The automated CMS has been initiated by the Supreme Court and is being implemented in stages in the first instance courts. There was disagreement between JAMR and the SC about the significance and approach to the development of an automated CMS. The SC wanted the CMS to be the highest priority for JAMR’s support of the judiciary, at the expense of other components in the project. Negotiations between the SC, USAID and JAMR resulted in a revision to the project SOW in July 2008 that increased the funding for Component One and reduced it for the other three by approximately $500,000 of the overall $3.7 million project budget. JAMR agreed to procure the audio recording equipment for the approximately 80 unequipped courtrooms, allowing the SC to utilize the funds it had budgeted for that project for the CMS, and to provide support through consultants, reports, and recommendation to the SC on the CMS. The SC largely disregarded JAMR consultants and the case management report and recommendations, including the results of a study trip to Bosnia and Herzegovina, and proceeded with its own contractor in the development of the CMS. Interviews with the Supreme Court and the Department of Common Courts personnel involved with CMS development and implementation confirmed their dissatisfaction with the efforts of 14 Reports of the number of criminal cases resolved nationally with plea bargains range from 80% to 90%. The Tbilisi City Court indicated that 85% of its criminal cases are resolved through plea bargains. 15 See Appendix D which presents data collected on clearance rates for civil, criminal, and administrative cases across the six pilot courts across JAMR’s project life. 16 Changes in the law regarding publication and telephone notice were already enacted when the survey results were published. 6 JAMR. The project staff also indicated that after this disagreement JAMR was largely limited to a training and support role in CMS implementation.17 Thus over the course of the project JAMR provided some consultation with the SC and its working group on the CMS and then provided training for court staff on the new system. The SC’s CMS is running in the pilot courts and most others throughout the country. It is still being developed and has not reached its full capacity18 but court staff and judges interviewed are satisfied with the system. There are two other case management systems operating as well. The courts of appeals have a simple system that tracks the status of cases and provides notices to parties and counsel about pending hearings and deadlines. This was created relatively quickly in part in response to the delay in the creations of the CMS. There is also a well-developed MoJ/prosecutorial system that is operating for all of the criminal court proceedings, which currently has greater capacity than the CMS system. The systems are not compatible at present, and while neither allows private attorneys to access the system presently, both systems anticipate expanding to accommodate more users. Audio recording systems are in place in approximately 100 courtrooms and are gradually replacing written transcripts with indexed audio recordings. JAMR agreed to purchase and install audio recording systems in the balance of the courtrooms (80) that had not received the systems through the SC and DCC program. While not an original part of the project, JAMR and USAID agreed to relieve the SC of this budget item and allow more money for CMS. The software for the audio recording system was already developed by DCC and 14 systems had already been installed when the project was transferred to JAMR. There was an 18 month delay in acquisition of the equipment by JAMR due to deficiencies in the original tender documents prepared by JAMR, and the acquisition did not begin until December 2009. By the conclusion of the project the equipment was installed in the 80 courtrooms however. According to the Department of Common Courts (which has responsibility for maintaining the systems) the equipment is operational throughout the country in almost all courtrooms. In the four pilot courts visited the equipment is being used for hearings in civil and administrative cases. In Mtskheta and Rustavi the system is also being used in criminal and administrative violations19 cases, but in Gori and Tbilisi the recording system is not in use for these cases. The Tbilisi City Court criminal courtrooms are not even equipped with the audio recording devices, and the court shows little interest in acquiring or utilizing the equipment. A Bench-Bar Committee was created for Tbilisi City Courts that made substantive contributions to case handling by local courts. A committee was established including judges and lawyers to discuss general issues confronting the courts. One judge and one attorney were identified by the project to select the other members by consensus. The members were limited to lawyers and judges dealing with civil and administrative matters only. Attorney members did not include lawyers from NGOs or legal aid organizations. JAMR provided a moderator for the 17 The staff member hired to work with the SC on CMS implementation was released by JAMR because he had little or nothing to do after these changes according to JAMR management. 18 There must be changes to the Presidential Order on certain court rules to fully implement it as well, which process is underway. 19 Administrative violations cases are civil proceedings that may result in \administrative detention for the parties, i.e. jail time. SI’s Evaluation of USAID/Georgia’s JAMR Project 7 meetings to assure the discussions stayed focused on general or procedural topics and did not include discussion of specific cases. The committee made substantive, constructive recommendations on procedural matters, resulting in changes to case management by judges and proposed changes to civil procedure rules ultimately adopted by Parliament20 Court users are satisfied with the improvements in case handling by court staff and judges. Court user surveys commissioned by JAMR confirm the satisfaction of both lay and professional court users with the improvements in the processing of cases by the courts. Surveys were conducted in 2008 and 2010 in the six pilot courts and found a general increase in the level of satisfaction with the courts.21 Interestingly, the 2010 Survey showed the greatest increase was in satisfaction with the building location and comfort while the most insignificant increase was with satisfaction over the simplicity of court procedures and provision of information. Interviews with lawyers from both legal aid organizations and the Bar Association and representatives of Georgian NGOs engaged in legal reforms groups confirm lawyer satisfaction with the improved case management. This included lawyers generally dissatisfied with case outcomes or distrustful of courts. Surveys sponsored by the United Nations Development Programme (UNDP)22 and European Union23 of the general public’s attitudes towards the court system reflect a lack of confidence or trust in the decisions of the courts, questions not addressed directly by the JAMR court user surveys. Component 2: Establishment of Court Administrators as Recognized and Effective specialists within the Georgia Judicial System The position of court manager has been created through changes in the law, although final implementing regulations are still being drafted. The position of professional court manager was created by the HCoJ, with most of the necessary changes in the law to empower the manager to perform their designated duties. JAMR conducted an assessment of court management practices during 2008 and presented a report to the HCOJ and SC recommending the creation of professional court manager positions in the courts. JAMR worked with the HCOJ to develop the position description and with the HSOJ to develop the training program and certification exam for the court manager position. JAMR paid for the first two groups of current court staff to receive the training, although future trainings will only be available for a fee. Twenty￾four people completed the training and were certified. There were eighteen court managers hired for the unified courts,24 including three women. One of the women was dismissed when a new presiding judge was appointed. The new court managers all served as staff directors or held other senior positions in the courts prior to their certification and employment. The civil service status of these managers is unclear, as they were not selected through a competitive process, and in two courts they are designated as “acting” court managers. 20 Judges and lawyers interviewed confirmed the substance of the recommendations that are reflected in the JAMR Draft Final Report of September 2011. 21 Court User Survey prepared by the Institute for Polling and Marketing, Tbilisi, August 2010 at 43. 22 “Basic Knowledge and Perception Survey about the Judicial System of Georgia” (United Nations Development Programme and Social Research Institute, 2010) 23 2010 Crime Survey of Georgia (Ministry of Justice and GORBI) 24 The court manager for the Tbilisi City Court did not complete the training and was not certified. 8 According to JAMR staff there was no support within the SC or the HCoJ for the creation of a professional court managers association in Georgia and this part of the component was dropped.25 The trained and certified court managers are in frequent communication with each other, sharing questions, problems and solutions, a function that the association would have performed. Neither the judges nor the court managers interviewed at the four pilot courts visited reported difficulties with the transition to the enhanced court manager system. According to the judges interviewed at the pilot courts there has been some increase in the number of cases handled by the presiding judges in some of the courts, but not a significant number. Component 3: Establishment of a Budget Office to Strengthen the Capacity of the Judiciary to Formulate and Manage the Court’s Budget Decentralization of budget and procurement responsibilities to local courts has not occurred due to court policy determinations. The original SOW contemplated enhancing the capacity of the local courts to assume more responsibility for their budgets and procurement. This part was dropped due to the determination at policy levels of the SC and government to maintain centralized budgeting authority for the courts. The SC did not support this component of the project and indicated its interest in this component being dropped from JAMR.26 Both the SC and the HCoJ representatives interviewed indicated that decentralization was never an option. In its Third Annual Report JAMR recommended revising Component Three to focus on strengthening the Department of Common Courts to improve its budgeting capacity.27 Its 2010 work plan reflected this change.28 Training on judicial budget preparation has been provided to the staff of the DCC and court staffs. JAMR worked with the DCC, the existing administrative office for the courts under the HCoJ, to improve its management and enhance its ability to more effectively perform its budgetary duties. Assessments of the office’s capacity to deal with budget and IT issues, trainings and study trips were provided to DCC staff. High turnover in the DCC office has impeded the effectiveness of the training to a certain extent. Budget training was included as a part of the court administration training by the HSJ. None of the court administrative staff indicated that budgeting was an important or particularly difficult part of their duties. “Program budgeting” expertise is being provided to the DCC to assist it in complying with the Ministry of Finance (MoF) mandate. The MoF is adopting program budgeting (or performance budgeting) for all government entities and the courts budget is subject to this new requirement. JAMR is working with the DCC to assist them in adapting to the new requirements. This process further reinforces the continued centralization of the judicial budgeting process. Component 4: Public Outreach 25 See JAMR 3rd Annual Report, p. 21 (October 30, 2010). 26 See JAMR 1st Annual Report at 4. 27 JAMR 3rd Annual Report at 23 (October 30, 2010). 28 JAMR 2010 Work Plan at v. SI’s Evaluation of USAID/Georgia’s JAMR Project 9 Public information facilities, publications and trainings have been provided for the courts. JAMR provided substantial assistance to the courts to improve their public outreach activities. This included installation of an information desk in the lobby of the Tbilisi City Court (this court hears 47 percent of the cases in Georgia) and information boards in unified courts outside Tbilisi. The courts were also provided with suggestion boxes to enable members of the public to provide comments, suggestions and concerns. These are not generating a significant number of comments, nor many relevant ones according to the court administrators and judges interviewed at the four pilot courts. JAMR conducted training for court personnel on providing public service, sponsored numerous publications for the courts, and developed a system for monitoring the public service approach of court personnel. The court managers interviewed at the four pilot courts indicated that they and their colleagues have adopted some of these techniques, such as calling staff anonymously to evaluate the way they answer the phones and respond to questions. The project commissioned a “Public Communication and Outreach Strategy” for the courts at the end of the project for which the courts are currently seeking funding. Results of court user surveys indicate increased satisfaction with court facilities and service. The two court user surveys conducted by JAMR indicated somewhat increased satisfaction with the courts.29 Some of this improvement was driven by higher satisfaction with facilities, reflecting the court renovations that have taken place throughout the country, as well as by more efficient case processing and improved public service. These surveys are discussed in more detail in Component One above. No identifiable improvement in the ability of pro se litigants to participate in court proceedings, and some evidence of greater difficulty due to case filing reforms. The judges and court administrators at the four pilot courts visited do not regard pro se litigants30 as a critical issue, although there are a significant number of parties without lawyers in civil and administrative cases. The recent adoption of the new case application and reply forms requires knowledge of the law applying to the dispute that may be making the pro se process more difficult, as even some lawyers find the forms confusing. Court personnel may help with administrative details of the forms but are unable to assist with legal theories or statutes implicated in the matter, information required by the forms. Pro se cases cannot be disaggregated from case statistics in civil and administrative cases to determine their time of disposition but court personnel impressions were that the cases took longer to process. 31Because the procedures in administrative cases are still partially inquisitorial, administrative court judges were better able to assist pro se litigants during the hearing. JAMR held a roundtable on issues relating to pro se litigants with judges and lawyers to consider possible approaches to addressing problems raised by this process. JAMR and 29 Court User Survey, Institute for Polling and Marketing, August 2008; Court User Survey prepared by the Institute for Polling and Marketing, Tbilisi, August 2010. 30 A pro se litigant is one appearing in a court proceeding without an attorney as their representative. 31 JAMR originally included “Decrease in processing time of pro se cases” as the second objective under its Component 4 Indicators. In its 3rd Annual report (at 35) it recommended that the indicator be dropped as too difficult to measure and suggested that “number of court staff trained in customer service” be substituted. 10 the HCoJ collected over 100 frequently asked questions for pro se litigants, and a draft publication of these questions is currently under review. JAMR Monitoring and Evaluation The performance objectives and indicators in JAMR’s measurement and evaluation plan tended to be gross measures of activity with little qualitative assessment. For example JAMR tracked the following as indicators to measure the project’s progress:  Number of USG-assisted courts with improved case management  Mean case disposition times in courts assisted by USG in the area of case management  Ratio of new case filings to case dispositions in courts assisted by USG in the area of case management  Number of justice sector personnel that received USG training. Indicators for the other components were similar, in that they focused on numbers trained or actions completed (such as the number of audio recording systems installed or the number of information boards placed in courts). The JAMR reports indicate the number of assisted courts and personnel trained, as well as study trips conducted and workshops and conferences sponsored or facilitated across all four components. The reports do not suggest that the project conducted any assessments of these trainings or activities, and the JAMR staff indicated that they had not. Thus there is no participant feedback on the events. The primary statistical data generated by JAMR for its reports are the numbers demonstrating the reduction of the case backlogs and shortening of case disposition time, reflecting very substantial improvement in both areas. The data in these reports are not broken out by courts, but numbers across the country show similar reductions in backlogs and case processing time. The reports do not attempt to show any correlation between specific activities of JAMR or procedural reforms and the reduction in the numbers however. Only in Component Four were there detailed surveys conducted of court user attitudes by the JAMR subcontractor, the Institute for Polling and Marketing in Tbilisi. While these surveys appear well crafted and thorough, they do not measure the activity of JAMR and its results. CONCLUSIONS Across its four components, JAMR had a significant impact on the role of court administrators through the creation of professional court managers, some immeasurable impact on the first, case management, and the fourth, public outreach, and little impact on the third component, court budget processes. Delays during the initiation of the project and poor communication and understanding between the project and one of its principal partners, the SC, created complications for the implementation of the CMS, deprived the project of resources for use in its other components, and significantly slowed the overall implementation of the project. The project provided widespread trainings and written materials across a range of issues and conducted several useful surveys relative to two of its components. The quality of the trainings and materials and the impact on the project’s objectives remain unmeasured, although participants interviewed during this evaluation SI’s Evaluation of USAID/Georgia’s JAMR Project 11 gave mostly positive responses.32 The conclusions articulated below do not attempt to address all of JAMR’s project activities but instead focus on those efforts that either had the greatest impact or were directed towards the most significant objectives of the project. Each will be addressed by component. Component 1: Improvement of Court Operations 1. The case backlog and time of case processing has been significantly reduced. This is an important development for the Georgia courts and one that will very likely be a permanent development due to the changes in court procedures that were critical in making it happen. The impetus for these changes came from the HCoJ and the Georgian courts and the movement was underway before the inception of JAMR. Rule changes in the delivery of notices of court hearings and rulings, and the widespread use of plea bargains in criminal cases may have had the greatest impact on the backlogs and timing of case processing according to judges, prosecutors and defense attorneys interviewed, although the reduction of statutory times for handling cases also contributed. The data collected by JAMR in support of this component reflects the improvement but does not permit any analysis of the factors that made the greatest contribution. 2. JAMR contributed to the improvement of court operations through its surveys which identified the reasons for case delays and allowed for targeted follow-up, as well as its collection, translation, and distribution of case management materials and training sessions according to court personnel who participated in the sessions. Its assistance in developing more useful and comprehensive statistics also aided in locating logjams in the case flow. 3. There is a risk that the emphasis on caseloads and case processing, especially their role as a predominant factor in evaluating the performance of judges, is having and will continue to have an adverse impact on the quality of the decision￾making, and particularly the ability of the judges to prepare decisions including the legal basis for their judgment.33Efforts by the HCoJ, with JAMR support, to develop a case weighting system based on international models may help address this concern. Project and court attention directed to issues of quality decision￾making at the end of the program may also help to ameliorate the negative impact of the heavy emphasis on counting cases, but should continue to hold the attention of the court. 4. The CMS developed by the SC is a powerful and valuable device to the management of cases and caseflow. It will facilitate case filings and dispositions and have the capacity to generate a wide range of statistical information about cases and track them at every stage of the process. The SC was prepared to begin development of the CMS at the outset of JAMR, and disagreed with the direction of the project as it was initiated. There were serious misunderstandings between JAMR, USAID, and the Court, resulting in both a significant amount of lost time and the wasting of project resources on consultants and reports that the Court was either not interested in or chose to ignore. While the loss of resources for other components of the project was substantial, the loss of time was even more critical. 32 JAMR trainings reached hundreds of court staff and judges. During the interviews almost all of the participants from the courts and bar had participated in one or more JAMR workshops or training sessions. 33 As an example, criminal defense attorneys interviewed indicated that decisions in pre-trial detention hearings included virtually identical findings as to why the defendant should be denied release on bond. 12 The court of appeals quickly developed a rudimentary case handling and notice system in that period of time, and the MoJ extended its case management system through the prosecutors’ office to include criminal proceedings with access by judges hearing criminal cases. Coordinating or integrating these systems will not be simple, both from a technological and political standpoint. According to DCC staff and other court personnel familiar with both systems, the technological basis of the two systems is not compatible. Perhaps more problematic, the MOJ proponents of its system see it as a comprehensive electronic data management system that will address all parts of the criminal justice system from investigation through incarceration. 5. The audio recording equipment acquisition activity that JAMR took over as part of the agreement with the SC has largely been completed with the installation of the equipment in almost all Georgia court rooms. However, its implementation was delayed by irregularities in the initial tender that set back the project at least 18 months. The equipment was identified and the software developed and running when JAMR took over the project, which should have been a quick and simple acquisition project. Somewhat more problematic, but arguably outside the scope of the JAMR role, is the lack of use of this technology for criminal and administrative violations cases. This is particularly troubling as those cases result in the loss of personal liberty of the parties, and have been the subject of complaints about alteration of the written transcripts.34 Component 2: Establishment of Court Administrators as Recognized and Effective Specialists within the Georgia Judicial System 1. It was in this component that JAMR made the greatest contribution to a successful outcome. At the end of the project the position of court manager had been created for the unified courts, with legal authority, a successfully developed training and certification program, and the majority of positions in the courts filled. JAMR guided and assisted this process both in its conceptualization and implementation, and effectively participated in the creation of the profession of court administrator in Georgia. 2. The successful creation of the positions and the training and certification of the new managers must include one caveat. All of those selected for the training, and funded by JAMR, were current senior court staff. The selection process for the training was closed, and only those certified were eligible for the position of court manager. The civil service status of these positions, with the little job security that such status entails, is unclear since the selection was not competitive. Component 3: Establishment of a Budget Office to Strengthen the Capacity of the Judiciary to Formulate and Manage the Court’s Budget This component was the weakest of the project. As originally envisioned it was to address the decentralization of the budget and procurement functions of the courts. Decentralization was never the agenda of the leadership of the courts. The SC wanted to eliminate the component (continuing to express that view to this day) and thought it of little value. The outcome of the efforts under this 34 Defense lawyers interviewed indicated that on occasion motions made during criminal trials that might provide a basis for appeal were not reflected in the written transcripts. Audio recordings would prevent this practice from occurring. SI’s Evaluation of USAID/Georgia’s JAMR Project 13 component was to strengthen the existing administrative office of the courts, the DCC, and to assist them with development of their budget skills for dealing with the MoF’s program budgeting. While not insignificant, this increased capacity does not address the underlying issues of political or government control over court budgets. Component 4: Public Outreach 1. Similar to Component 2, JAMR made a substantial contribution through this component. The materials provided, such as the information desk at Tbilisi City Court and information boards at the others courts throughout Georgia, helped tell the story of the court and its procedures. Similar to Component 1, the commitment of the courts to improving their public image was central, while JAMR played a role in supporting that initiative. The focus on customer service has apparently improved the image of the courts according to the court user surveys and the courts’ commitment to ongoing surveys reflects their desire to continue to improve their image.35 2. One aspect of this component, however, showed no progress, and that is the ability of pro se litigants to utilize the court system. Parties without lawyers continue to comprise a significant percentage of litigants and little was done to assist them. The advent of the required application and reply forms, with their legal basis requirements, has made it even more difficult for these parties to participate. Limitations on legal aid services will continue to make this a problem that should be addressed. RECOMMENDATIONS 1. The focus of the Georgian judiciary on case numbers should be monitored closely to ascertain its impact on quality decision-making and decisions. The new Judicial Independence, Legal Education and Professionalism Project is in a good position to do this with its qualitative focus on judicial performance. 2. The Georgian judiciary has demonstrated a serious commitment to improving both its administration and its public outreach. JAMR took advantage of this in providing its extensive trainings and workshops. This commitment by the courts should be nurtured and supported through ongoing assistance to them and continued recognition of the importance of their efforts. 3. USAID should closely monitor changing attitudes and circumstances among anticipated counterparts. In-country developments that are moving quickly should be within the purview of USAID staff supporting the project and the initial work plans should reflect the most current disposition of essential counterparts. 4. Efforts should be made to monitor the courts’ use of audio recording equipment for criminal trials and hearings. Questions remain about why the system is not being used for those and administrative violation cases. 5. Integration of the three case management systems is important to achieve a case management system accessible to litigants and lawyers on all sides of the 35 For example, a 2010 Court Users Survey commissioned by the Institute for Polling and Marketing found that of the 2,000 court users surveyed, 71.2% reported being satisfied with the service of the court and 77% believed that court personnel were competent., 14 disputes. Neither of the main systems currently allows defense attorneys access, which furthers the perception of a skewed criminal justice system. 6. Projects whose activities include significant numbers of trainings, study trips, round tables, etc. should develop evaluative processes to obtain some measure of feedback from the participants. While such processes may be only crude measures, they provide broad impressions and reactions that can alert projects to major problems in the events. SI’s Evaluation of USAID/Georgia’s JAMR Project I APPENDIX A. SCOPE OF WORK I. Objective USAID/Caucasus-Georgia seeks to carry out an evaluation of the full interventions implemented under the Judicial Administration and Management Reform (JAMR) project activity in Georgia since September 2007. The current JAMR activity ends in September 2011. The evaluation will help the mission (a) take stock of the effectiveness and relevance of the current activity (b); use lessons learned to enhance management and substance of current and future Rule of Law (ROL) interventions; The main components of the evaluation will be as follows: 1. Evaluation of results in relation to the initial project objectives made in the Georgian Judiciary institutions that JAMR has provided support for. 2. Evaluation of the relevance of JAMR methodology of providing assistance and interventions and monitoring their effectiveness (including the JAMR PMP) in relation to the project objectives. 3. Recommendations on improving the management and substance of other USAID-funded Rule of Law (RoL) interventions as well as identify the areas of the JAMR project which would need a follow up in the future in the framework of interventions under the same Assistance Objective. II. Background Although after the Rose Revolution in 2003 the judiciary in Georgia had undergone significant reforms – “corrupt” judges were removed from the bench; judges salaries were increased on average to 300% per court level; the courts were restructured and consolidated from 75 trial courts to approximately 20 courts; Court houses were renovated and equipped in 2007 many problems related with internal management of the system. Courts in Georgia were inefficient: Most cases were not decided within a reasonable time - the significant case delay was attributed not only to the large numbers of judicial vacancies caused by the reorganization of the courts, but also to the out-dated case management procedures. Judges received little administrative support within their courts and needed broad training in case management. Court hearings were constantly being rescheduled as attorneys, parties to the case, and witnesses, freely ignored hearing dates. Implementation of major reforms designed during earlier donor programs, in particular, the World Bank project to improve court administration and case management, foundered on lack of commitment and resources. The need for an activity to address court administration and case management was identified in two assessments conducted in 2006. In March, the Criminal Justice Sector Assessment led by EUR/ACE in the State Department was conducted by an interagency team of State Department, the Department of Justice and USAID. Although the focus of the assessment was primarily the criminal justice sector, the report identified the areas for USAID interventions as judicial strengthening, judicial education, legal education reform and legal profession reform. In July, USAID conducted, through Management Systems International (MSI), a comprehensive Rule of Law/Anticorruption Assessment. The final recommendations for USAID assistance along with other issues focused on judicial strengthening; court administration and case management. The USAID Judicial Administration and Management Reform Project (JAMR) was designed to support and move the judiciary to become an independent yet equal branch of government through a comprehensive program for improved court administration and case management throughout all the courts of Georgia. It is an assistance program structured to play a major role in promoting judicial reform as reflected by the Department of State’s F Framework under the SI’s Evaluation of USAID/Georgia’s JAMR Project II Governing Justly and Democratically Program Objective, Rule of Law and Human Rights Program Area, and Program Element Justice System. The geographic scope of this four year, $3,702,000 project was not limited to any particular region of the country. Start and end dates of the contract 114-C-00-07-00068-00 are 09/28/2007 and 09/28/2011 respectively. Based on the Georgian judicial leadership’s commitment to and provision of resources for judicial reform, the overwhelming need for improved court administration and case management is being addressed by JAMR to achieve: - Improved court operations: improve the quality and timeliness of judicial decision making since court administrative and management issues were delegated to court administrative staff. Establishment of court administrators as recognized and effective specialists within Georgia’s judicial system: improve court practices by providing the technical assistance necessary for the court administrative staff to process cases in a timely and efficient manner Establishment of a Budget Office to strengthen the capacity of the judiciary to formulate and manage the courts’ budget: - Improved public awareness: improve public access, services, uses and understanding of the courts and their facilities. The Major result of JAMR activities to date is increased efficiency of the Georgian court system by improving case management and court administration. JAMR helped trial and appellate courts throughout the country by providing case flow management training to judges, court staff and the local bar, assessing the causes of case delay and installing audio recording equipment and training court secretaries in audio recording. These and other activities carried out by the project reportedly resulted in a decrease of the mean case disposition time from 111 days in 2009 to 79 days in 2010. Another achievement of the project in case management was the creation of the court manager position and an accompanying professional certification program. Fourteen court managers completed a three-week court management course at the High School of Justice (HSOJ) and successfully passed the certification exams. JAMR performance information sources are project annual work plans, quarterly and annual reports, monitoring and evaluation plan. The project tracks the following for indicators to measure progress:  Number of USG-assisted courts with improved case management  Mean case disposition times in courts assisted by USG in the area of case management  Ratio of new case filings to case dispositions in courts assisted by USG in the area of case management  Number of justice sector personnel that received USG training These documents including baseline information will be provided to contractor. III. Evaluation tasks The evaluation team shall conduct the following tasks: 1. Evaluate progress, in relation to the initial project objectives, made in the Georgian judicial system with the JAMR project support  Describe the present level of institutional performance of the target institutions; provide an overview of the degree to which the target institutions use the newly developed and established work processes, systems, structures and policies  Indicate the key internal and external factors responsible for either success or failure with a description of respective lessons to be learnt SI’s Evaluation of USAID/Georgia’s JAMR Project III  Where appropriate, identify any barriers that have hindered the project from achieving the initial objectives  Assess the degree and likelihood of sustainability of JAMR assistance results  Review the extent to which the JAMR interventions are perceived as effective by the target institutions/stakeholders 2. Evaluate relevance of JAMR method of providing assistance and interventions and monitoring their effectiveness (including the JAMR PMP) in relation to the project objectives.  Assess relevance of the methodology used by JAMR for achieving the planned objectives in theory. Given the objectives of the JAMR project, did the project apply best practices according to the literature on Court administration to achieve their objectives?  Assess practical relevance of the methodology used by JAMR for achieving planned objectives. Does the evidence suggest that the JAMR methods helped in achieving the results?  Were there any failures, and if so, were they due to inadequate implementation of the methodology or the methodology itself?  Assess the relevance of monitoring efforts of the JAMR project. Do the monitoring efforts make sense logically in relation to project objectives and do they provide sufficient data for determining if project objectives were achieved? 3. Provide recommendations on improving the management and substance of other USAID￾funded Rule of Law (RoL) interventions as well as identify the areas of the JAMR project which would need a follow up in future in the framework of interventions under the same AO.  Identify those critical factors of success or failure of the evaluated interventions (institutional, individual, financial, etc.) that need to be taken into account in other RoL programs;  Indicate how some of the best practices established in JAMR’s successful interventions can be drawn on in future/current RoL programs;  Provide recommendations for a better developed and a more practical use of the PMP as an effective performance measurement system;  Provide ideas for ensuring sustainability of JAMR interventions;  Indicate any possible risk factors or critical assumptions to be taken into account in future interventions related to the same AO. IV. Methodology The evaluation will examine all interventions to date. At a minimum, the evaluator will: - Review and analyze the relevant project documentation (Project Statement of Work, Monitoring and Evaluation Plan, Annual Work Plans, quarterly and annual performance reports). These will be provided in advance to the start of the fieldwork as part of the preparation stage; - Meet and interview a) project key management staff, b) project beneficiaries c) other stakeholders (donor agencies, CSOs, etc) as deemed relevant; - Conduct workplace visits and in-country field trips to Mtskheta, Rustavi and Gori. These trips will provide an opportunity to engage regional courts that benefited from USAID JAMR project and assessing the outcomes of JAMR’s work with them. SI’s Evaluation of USAID/Georgia’s JAMR Project IV - Prepare an Evaluation Report that will fully document findings and conclusions of the evaluation and provide feedback on the JAMR project strengths, weaknesses and results achievements to date, within the context of the DG Strategy, activities and programs of other donors in this field. The applicant needs to propose the most relevant methods and tools for data collection and analysis that will reduce evaluator-specific judgments. Findings should be specific, concise and supported by quantitative and qualitative information that is reliable and valid. Strengths and limitations of the proposed methodologies should be discussed in the application. The evaluator, in collaboration with USAID/Georgia, will finalize the overall evaluation methodology as well as initial work plan and submit it to USAID/Georgia prior to their arrival in Georgia. The evaluator will also work with the mission to develop a list of illustrative questions referring to respective areas for exploration under the Evaluation Tasks section. V. Deliverables The outputs associated with this contract must be completed and accepted by USAID/Georgia no later than September 23, 2011. The expert will debrief USAID at least once (midway through the analysis or prior to departure). The deliverables of this award are: 1. Written design, including identification of key question(s), methods, work plan, main features of data collection instruments, data analysis plans, and dissemination plan 2. All records from the evaluation (e.g., focus group transcripts), and all quantitative data in an organized fashion and fully documented for use by those not familiar with the project or evaluation. 3. Draft evaluation report: A preliminary draft evaluation report should be presented prior to departure from Georgia for Mission/DG review and comment. 4. In-brief and de-brief sessions will be organized by the mission leadership. 5. The final evaluation report. The final report will be submitted no later than 3 weeks from the date of USAID comments. The final report shall include:  An Executive Summary – A 3-5 page summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations, and lessons learned (if applicable)  Table of contents  An evaluation report (no more than 15 pages) including major findings and related issues and questions.  Statements of differences of opinion on the part of funders, implementers, and/or members of the evaluation team (if these are received by the evaluator)  Appendices (standardized data collection tools such as interview forms, focus group records, etc., the statement of work; the list of documents reviewed; the list of meetings held) SI’s Evaluation of USAID/Georgia’s JAMR Project V APPENDIX B. PERSONS CONTACTED Monday, September 5 Monday, September 5 at 3 pm JAMR staff Former Chief of Party: David Magradze Component leaders: Eka Oniani Shorena Gigauri Natia Kukuladze Tuesday, September 6 NGOs Tuesday , September 6 at 10:30  Georgian Young Lawyers’ Association Tamar Chugoshvili  Article 42 Nazi Janezashvili  Human Rights Center Beqa Jiqia  Transparency international Zurab Gvelesiani USAID Judicial Independence and Legal Empowerment Project; JILEP Tuesday, September 6 at 2 pm Chief of Party Herb Bowman Deputy Chief of Party Giorgi Chkheidze Judicial Reform Adviser Inga Todria Beneficiaries Practitioner lawyers ( pro bono) Tuesday, September 6 at 5 pm  Director of Legal Aid Center of GYLA(civil and administrative lawyer) Kakha Kojoridze  Criminal lawyer (GYLA) Maia Khucishvili  Criminal Lawyer (Article 42) Archil Chofikashvili  Civil and administrative Lawyer (Article 42) Sofiko Aleqsidze  Criminal lawyer (State Legal Aid Service) Irakli Mirtskhulava SI’s Evaluation of USAID/Georgia’s JAMR Project VI Wednesday, September 7 International Organizations Wednesday , September 7 at 11 am  United Nations Development Programme (UNDP) Natia Natsvlishvili  European Union (EU) Tamar Khulordava  GesellschaftfürInternationaleZusammenarbeit (GIZ) Volker Stampe  Council of Europe (CoE) Leila Marshania US Embassy/USAID Wednesday, September 7 from 3 pm  Regional Legal Advisors Jared Kimbal and David DeVillers  UDAID/Georgia Director  UDAID/Georgia Deputy Director Jock Conly Joakim Parker  DG Office Giorgi Vashakidze, Rusudan Tabatadze Thursday, September 8 High Council of Justice Thursday, September 8, 11 am  Assistant to Secretary UchaTodua  Head of the HR Department Tamar Sulakvelidze  Head of the Analytical department Giorgi Pavladze  Deputy Head of Disciplinary Department, former Tbilisi City Court Manager, VakoGavasheli High School of Justice Thursday, September 8, 2 pm  Deputy Director of the School Shota Ruxadze Department of Common Courts Thursday, September 8, 4 pm  Head of the Department  Deputy Head of the Department of Common Courts Dima Gegelia Kale Mkervalisvhili Friday, September 9 Supreme Court of Georgia 13 pm  Deputy Chief Justice Zaza Meishvili SI’s Evaluation of USAID/Georgia’s JAMR Project VII  Court Manager Giorgi Berdzuli  PR person of the Supreme Court  Assistant to the Chief Justice Nana Vasadze Oliko Panchulidze577 14 99 33 Monday, September 12 JAMR Staff Monday, September 12 at 11 am  Chief of Party  Deputy Chief of Party  Former Chief of Party  Component Leaders Jason Schwarz KakhaTsikarishvili David Magradze Eka Oniani Shorena Gigauri Prosecutors and Prosecutors’ Office Monday, September 12 at 4 pm (Gorgasali str # 24)  Prosecutors and Prosecutors’ Office Qeti Chomaxashvili - Supervising Prosecutor from Mtskheta Prosecutor from Qvemo Qartli Prosecutor from Tbilisi Bar Association Monday, September 12 at 6 pm  Head of the Ethics Commission of Bar Irakli Kordzaxia  Bar member, Private lawyer David Asatian Tuesday, September 13 Mtskheta Regional Court 10 am  Head of Mtskheta Regional Court  Manager of Mtskheta Regional Court  Judge  Assistant to Judge  Head of Chancellery Vasil Mshvenieraze Dimitri Niazashvili Teimuraz Jervalidze Natia Togonidze Meri Tatarishvili Wednesday, September 14 Gori Regional Court 11 am  Head of Gori Regional Court  Manager of Gori Regional Court  Judges Nikoloz Marsagishvili Diogen Dolidze Eka Zarnadze Shorena Tciqaridze David Papuashvili Davit Mgelashvili SI’s Evaluation of USAID/Georgia’s JAMR Project VIII Thursday, September 15 Tbilisi City Court 12 am  Head of Tbilisi City Court  Head of Bureau  Judges Mamia Fkhakadze Giorgi Guguchia Soso Gurtckaia Nino Sharadze Lela Nozadze Rustavi Regional Court 3 pm  Judges  Assistants to Judges Larisa Liparteliani Lasha Chkhikvadze Natia Kanteladze Natia Qavtaradze Nato Zlierishvili Ekaterine Beridze Tamar Fkhakadze Friday, September 16 USAID Debrief 5 pm Chief of Party (JILEP) Deputy chief of Party(JILEP) DG Office DG Office DG Office DG Office Herb Bowman Giorgi Chkheidze Susan Cowley Giorgi Vashakidze Rusudan Tabatadze Kirsten Michener SI’s Evaluation of USAID/Georgia’s JAMR Project IX APPENDIX C. INTERVIEW QUESTIONS JAMR Interview Questions: First Meeting TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. What was the relationship of JAMR with the Supreme Court in the development and implementation of the Case Management System? a. In which courts has this been implemented? b. What are the plans for future courts? c. What is the process for monitoring the effectiveness and results of the use of the CMS? d. What problems have been identified with the system and what steps are planned to address these? 2. What factors do you think have contributed to the reduction of the case backlog and processing time? Which have had the greatest impact? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. More cooperative attitude of practitioners, prosecutors, government agencies? e. More judges? f. Fewer cases being filed? g. Other reasons? 3. Which courts have been the most receptive and cooperative to your initiatives? Which courts the least? Why? 4. Have you observed an improvement in the quality of judicial decision-making over the past 4 years? a. Fairness of decision? b. Quality of written decisions? c. Other aspects of decisions? d. To what do you attribute this increase in quality of decision-making? 5. Were there any problems with the installation of the new audio recording equipment for the courts or the training of court staff? Component 2: Court Administrators 1. Was the institutional framework (including normative acts), training institutions, position description (basic structure) established? 2. What is the background of new administrators and how were they recruited? 3. Have you observed circumstances where they have played a role in improving the administration of the court? Please discuss. SI’s Evaluation of USAID/Georgia’s JAMR Project X Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) 1. To what extent has department been given budgeting authority of the courts? Been empowered to make budgetary determinations? 2. How has the department exercised its authority? 3. Has the role of the lower courts increased in the budgeting process? How? 4. Compare budgets before and after – what were the changes? 5. What is the status and prospects for delegation of procurement authority to local courts? What plans for monitoring and accountability? Component 4: Public Outreach 1. What physical changes and publications produced (number of information desks set up; publications; programs, etc.)? 2. What are the on-going monitoring mechanisms? What has been done, are they being used? 3. To what extent are trainings about being more customer service oriented effective/taken place? Are they being institutionalized/sustainable? 4. What is the process for reviewing and acting upon citizen suggestions? Can you provide specific examples? 5. Did JAMR review the courts responses to individual requests for information under the General Administrative Code, and the Supreme Court’s annual reports required by that Code? 6. Are there any indications that pro se litigants are better informed and prepared to represent themselves? KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE Component 1: Improvement of Court Operations 1. What are the factors affecting caseload in the Tbilisi city courts vs. other city courts? 2. Were differences in civil, criminal and administrative proceedings taken in to account? 3. Is the CMS software being effectively utilized? 4. What external factors facilitated JAMR’s successes? 5. What external factors made JAMR’s efforts more difficult? Component 2: Court Administrators 1. How well have court administrators been accepted by court staff in their new roles? Component 3: Capacity to Develop and Manage Budget 1. What is the biggest obstacle to greater budget independence for the courts? 2. Are there anticipated difficulties in the DCC adopting Ministry of Finance budgeting procedures? 3. Does the statutory framework exist and does the budgeting department have independent ability to request and allocate funds? 4. To what extent does the High Council of Justice, the Ministry of Finance or other executive branch actors change the budget requests by the Department of Common Courts? SI’s Evaluation of USAID/Georgia’s JAMR Project XI 5. To what extent does the Ministry of Finance or other executive branch actors change the budget requests by the Supreme Court? Component 4: Public Outreach 1. What external factors {other than JAMR activities) may have affected public perceptions of the courts? 2. Has there been greater usage of the courts as a result of improved public confidence? DEGREE AND LIKELIHOOD OF SUSTAINABILITY OF JAMR ASSISTED IMPROVEMENTS Component 1: Improvement of Court Operations 1. Do you anticipate that the improvements in timely processing of cases will continue? Why or why not? 2. What is the maintenance of software/hardware and training of personnel in utilizing the system? 3. What is the extent of court personnel utilization and satisfaction with the systems and their benefits? Component 2: Court Administrators 1. What is the continued professionalization of the positions, continuing education programs, etc.? 2. How much money is allocated for training, salary, etc.? 3. The extent to which Court Manager is utilizing powers bestowed upon him? Support and recognition of the Chief Judge of the authority of the Court Manager? Are they comfortable relinquishing power to Court Manager? 4. Are there plans for any kind of civil service protections? Is there a way of protecting Court Managers from being fired in retaliation, etc.? Component 3: Capacity to Develop and Manage Budget No questions Component 4: Public Outreach 1. To what extent is public information/education/satisfaction built into regular training of court staff? Needs to be fully integrated into training programs. 2. What continuing efforts to provide information and support to enhance the ability of pro se litigants to successfully represent themselves? EXTENT TO WHICH TARGET STAKEHOLDERS/INSTITUTIONS ARE AWARE OF JAMR INTERVENTIONS NO QUESTIONS TASK 2: EVALUATING RELEVANCE OF JAMR METHOLODY AND MONITORING SYSTEMS NO QUESTIONS TASK 3: LESSONS LEARNED 1. What difficulties did you encounter in implementation that future projects might avoid because of your experiences? 2. What external factors made your implementation and successes more possible? SI’s Evaluation of USAID/Georgia’s JAMR Project XII JAMR Interview Questions: Second Meeting TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. What was the relationship of JAMR with the Supreme Court in the development and implementation of the Case Management System? a. Is the CMS software being effectively utilized? b. Is JAMR engaged in the efforts to integrate the Prosecutors and Court of Appeals case systems software with the CMS? c. What was the impact of the budget reallocation on the remaining components d. What were the reasons for the delay in the acquisition of the audio recording equipment for the courts? e. Did JAMR do any work in the development of the software for the system? 2. What factors do you think have contributed to the reduction of the case backlog and processing time? Which have had the greatest impact? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. Increased number of plea bargains? e. More judges? (What was the increase in number of judges for the pilot courts during the project?) f. Fewer cases being filed? g. Other reasons? 3. Which courts have been the most receptive and cooperative to your initiatives? Which courts the least? Why? Component 2: Court Administrators 1. What is the status of the court management literature being developed? Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) 1. What happened to the decentralization focus of the original RFP? 2. What has JAMR’s role been in supporting the DCC? 3. What is the status of the program budgeting support for the DCC? Component 4: Public Outreach 1. What is the status of the IPM Public Relations Strategic plan? 2. How did JAMR go about developing the 100 FAQs for pro se litigants? KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE 1. What are the factors affecting caseload in the Tbilisi city courts vs. other city courts? 2. What external factors facilitated JAMR’s successes? 3. What external factors {other than JAMR activities) may have affected public perceptions of the courts? SI’s Evaluation of USAID/Georgia’s JAMR Project XIII 4. What external factors made JAMR’s efforts more difficult? 5. How has the JAMR project contributed to the independence of the courts? 6. What difficulties did you encounter in implementation that future projects might avoid because of your experiences? BENEFICIARY INTERVIEW QUESTIONS: NGOS TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. Have you had any experience with the new Case Management System being implemented in local courts? Please describe. 2. Did you find the CMS to be effective and useful? An improvement or prior systems? Please explain. 3. Data indicates that the case backlog for individual judges has sharply declined since the beginning of the JAMR project, and that the time it takes to resolve cases has decreased as well. Is this consistent with your experience with the courts? a. Does it vary from court to court? b. Which courts have shown the greatest improvement, which the least? 4. What factors do you think have contributed to these improvements? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. More cooperative attitude of practitioners, prosecutors, government agencies? e. More judges? f. Fewer cases being filed? g. More service-oriented staff and judges? h. Other reasons? 5. Have you observed an improvement in the quality of judicial decision-making over the past 4 years? a. Fairness of decision? b. Quality of written decisions? c. Other aspects of decisions? d. To what do you attribute this increase in quality of decision-making? 6. Have you had experience with the new audio recording capacity of the courts? a. What was the reason for your experience with it? b. Did the system function properly? c. Was the court staff knowledgeable in its use? Sustainability SI’s Evaluation of USAID/Georgia’s JAMR Project XIV 7. Do you anticipate that the improvements in timely processing of cases will continue? Why or why not? Component 2: Court Administrators 1. Are you familiar with the new position of Court Manager? 2. What, if any, experiences have you had with them? 3. In your dealings with them, did you find them: a. Knowledgeable? b. Cooperative? c. Effective? 4. Have you observed circumstances where they have played a role in improving the administration of the court? Please discuss. Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) 1. Are you familiar with efforts to enhance the budget authority and competence of the Department of Common Courts? 2. Will this enhanced authority contribute to judicial independence? If so, how? Component 4: Public Outreach 1. Have you observed an improvement in the public perception of the courts? 2. Have you observed an improvement in the NGO community’s perception of the courts? 3. To what do you attribute this improved perception? 4. Are you familiar with JAMR efforts to provide better information for pro se litigants? Describe. 5. Are pro se litigants better informed and prepared to represent themselves than they were 4 years ago? SUSTAINABILITY 1. Have the courts experienced a permanent change to being more open and transparent? 2. Will the courts continue to be proactive in providing public information? 3. Are continuing efforts to enhance the ability of pro se litigants to successfully represent themselves necessary? What steps could be taken to assist them? KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE? Responses to these questions were solicited above. EXTENT TO WHICH TARGET STAKEHOLDERS/INSTITUTIONS ARE AWARE OF JAMR INTERVENTIONS Same for all four components 1. To what extent did JAMR communicate with you during their implementation of this project? A. During assessment of problems phase? B. Development of solutions and approaches phase? C. Preparation and implementation of training? D. Preparation of new or revised normative acts? SI’s Evaluation of USAID/Georgia’s JAMR Project XV 2. To what extent are you satisfied with what JAMR has done? 3. To what extent are you satisfied with the JAMR’s overall communication with you during implementation and follow-up? TASK 2: EVALUATING RELEVANCE OF JAMR METHOLODY AND MONITORING SYSTEMS 1. What is the most effective way to measure public confidence in the courts? 2. What is the most effective way to measure the quality of judicial decision-making? 3. Were there things JAMR could have done to make further improvements that it did not do? A. Case handling procedures? B. Improved public information? SI’s Evaluation of USAID/Georgia’s JAMR Project XVI JAMR Interview Questions: Practitioners TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. Have you had any experience with the new Case Management System being implemented in local courts? Please describe. 2. Did you find the CMS to be effective and useful? An improvement or prior systems? Please explain. 3. Data indicates that the case backlog for individual judges has sharply declined since the beginning of the JAMR project, and that the time it takes to resolve cases has decreased as well. Is this consistent with your experience with the courts? a. Does it vary from court to court? b. Which courts have shown the greatest improvement, which the least? 4. What factors do you think have contributed to these improvements? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. More cooperative attitude of practitioners, prosecutors, government agencies? e. More judges? f. Fewer cases being filed? g. More service-oriented staff and judges? h. Other reasons? 5. Do you anticipate that the improvements in timely processing of cases will continue? Why or why not? (Sustainability). 6. Have you observed an improvement in the quality of judicial decision-making over the past 4 years? a. Fairness of decision? b. Quality of written decisions? c. Other aspects of decisions? d. To what do you attribute this increase in quality of decision-making? 7. Have you had experience with the new audio recording capacity of the courts? What was the reason for your experience with it? a. Did the system function properly? b. Was the court staff knowledgeable in its use? Component 2: Court Administrators 1. Are you familiar with the new position of Court Manager? 2. What, if any, experiences have you had with them? 3. In your dealings with them, did you find them: SI’s Evaluation of USAID/Georgia’s JAMR Project XVII a. Knowledgeable? b. Cooperative? c. Effective? 4. Have you observed circumstances where they have played a role in improving the administration of the court? Please discuss. Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) No questions for practitioners Component 4: Public Outreach 1. Have you observed an improvement in the public perception of the courts? 2. Have you observed an improvement in the practicing bars perception of the courts? 3. To what do you attribute this improved perception? 4. Are pro se litigants better informed and prepared to represent themselves? Sustainability 5. Have the courts experienced a permanent change to being more open and transparent? 6. Will the courts continue to be proactive in providing public information? 7. Are continuing efforts to enhance the ability of pro se litigants to successfully represent themselves necessary? KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE (NEED TO INCLUDE MAJOR Responses to these questions were solicited above. EXTENT TO WHICH TARGET STAKEHOLDERS/INSTITUTIONS ARE AWARE OF JAMR INTERVENTIONS 1. To what extent did JAMR communicate with you about their perception of the problems, needs of the judicial system? 2. To what extent are you satisfied with what JAMR has done? 3. To what extent were you satisfied with the JAMR’s communication with you during implementation and follow-up? TASK 2: EVALUATING RELEVANCE OF JAMR METHOLODY AND MONITORING SYSTEMS 1. What is the most effective way to measure public confidence in the courts? 2. What is the most effective way to measure the quality of judicial decision-making? 3. Were there things JAMR could have done to make further improvements that it did not do? A. Case handling procedures? B. Improved public information? SI’s Evaluation of USAID/Georgia’s JAMR Project XVIII JAMR Interview Questions: International Organizations TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. Have you had any experience with the new Case Management System being implemented in local courts? Please describe. 2. Data indicates that the case backlog for individual judges has sharply declined since the beginning of the JAMR project, and that the time it takes to resolve cases has decreased as well. Is this consistent with your observations of the courts? 3. What factors do you think have contributed to these improvements? 4. Have you observed an improvement in the quality of judicial decision-making over the past 4 years? 5. Have you had experience with the new audio recording capacity of the courts? Sustainability 6. Do you anticipate that the improvements in timely processing of cases will continue? Why or why not? Component 2: Court Administrators 1. Are you familiar with the new position of Court Manager? 2. What, if any, experiences have you had with them? 3. In your dealings with them, did you find them: a. Knowledgeable? b. Cooperative? c. Effective? 4. Have you observed circumstances where they have played a role in improving the administration of the court? Please discuss. Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) 1. Are you familiar with efforts to enhance the budget authority and competence of the Department of Common Courts? 2. Will this enhanced authority contribute to judicial independence? If so, how? Component 4: Public Outreach 1. Have you observed an improvement in the public perception of the courts? 2. To what do you attribute this improved perception? 3. Are you familiar with JAMR efforts to provide better information for pro se litigants? Describe. SUSTAINABILITY 1. Have the courts experienced a permanent change to being more open and transparent? 2. Will the courts continue to be proactive in providing public information? 3. Are continuing efforts to enhance the ability of pro se litigants to successfully represent themselves necessary? What steps could be taken to assist them? SI’s Evaluation of USAID/Georgia’s JAMR Project XIX KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE? Responses to these questions were solicited above. EXTENT TO WHICH TARGET STAKEHOLDERS/INSTITUTIONS ARE AWARE OF JAMR INTERVENTIONS Same for all four components 1. To what extent did JAMR communicate with you during their implementation of this project? A. During assessment of problems phase? B. Development of solutions and approaches phase? C. Preparation and implementation of training? D. Preparation of new or revised normative acts? 2. To what extent are you satisfied with what JAMR has done? 3. To what extent are you satisfied with the JAMR’s overall communication with you during implementation and follow-up? TASK 2: EVALUATING RELEVANCE OF JAMR METHOLODY AND MONITORING SYSTEMS 1. What is the most effective way to measure public confidence in the courts? 2. What is the most effective way to measure the quality of judicial decision-making? 3. Were there things JAMR could have done to make further improvements that it did not do? A. Case handling procedures? B. Improved public information? SI’s Evaluation of USAID/Georgia’s JAMR Project XX JAMR Interview Questions: Department of Common Courts Component 3: Capacity to Develop and Manage Budget 1. What is the role of the DCC in the budgetary process? How has the DCC exercised its budgetary authority? 2. Has the role of the lower courts increased in the budgeting process? How? What happened to the plans to devolve the budget authority to the lower courts? 3. How the budgeting process has changed after JAMR’s involvement? a. Describe your working relationship with the JAMR team. 4. What is the biggest obstacle to greater budget independence for the courts? 5. Are there anticipated difficulties in the DCC/HCOJ adopting Ministry of Finance program budgeting procedures? 6. To what extent does the HCOJ, Ministry of Finance or Government change the budget requests by the DCC? 7. What is the status and prospects for delegation of procurement authority to local courts? What plans for monitoring and accountability? SI’s Evaluation of USAID/Georgia’s JAMR Project XXI JAMR Interview Questions: High Council of Justice TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. What was the involvement of HCOJ in the development and implementation of the Case Management System? a. In which courts has this been implemented? b. What are the plans for future courts? c. What is the process for monitoring the effectiveness and results of the use of the CMS? d. What problems have been identified with the system and what steps are planned to address these? 2. Is the CMS software being effectively utilized? 3. What is the maintenance of software/hardware and training of personnel in utilizing the system? 4. What factors do you think have contributed to the reduction of the case backlog and processing time? Which have had the greatest impact? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. More cooperative attitude of practitioners, prosecutors, government agencies? e. More judges? f. Fewer cases being filed? g. Other reasons? 5. What steps were taken for improvement in the quality of judicial decision-making over the past 4 years? How quality of judicial decisions was affected by decrees of case disposition time. a. Fairness of decision? b. Quality of written decisions? c. Other aspects of decisions? d. To what do you attribute this increase in quality of decision-making? 6. Does the performance of individual judge in relation to timely proceeding the case analyzed and what measures are imposed for delay, does weight of the case considered during this process and how. 7. What criteria are used for case weighting? 8. Are new audio recording equipments utilized in all 80 courtrooms and do they substitute the written transcript? 9. How HR management system has changed after JAMR’s involvement. Component 2: Court Administrators SI’s Evaluation of USAID/Georgia’s JAMR Project XXII 1. Was the institutional framework (including normative acts), training institutions, position description (basic structure) established? 2. What is the background of new administrators and how were they recruited? 3. What role court managers have played in improving the administration of the court? Please discuss. 4. How well have court administrators been accepted by court staff in their new roles? 5. The extent to which Court Manager is utilizing powers bestowed upon him? Support and recognition of the Chief Judge of the authority of the Court Manager? Are they comfortable relinquishing power to Court Manager? 6. What is the continued professionalization of the positions, continuing education programs, etc.? 7. Are there plans for any kind of civil service protections? Is there a way of protecting Court Managers from being fired in retaliation, etc.? Component 3: Capacity to Develop and Manage Budget 1. How has the HCOJ exercised its budgetary authority? 2. Has the role of the lower courts increased in the budgeting process? How? 3. How budgeting process has changed after JAMR’s involvement? 4. What is the biggest obstacle to greater budget independence for the courts? 5. Are there anticipated difficulties in the DCC/HCOJ adopting Ministry of Finance budgeting procedures? 6. To what extent does the Ministry of Finance or Government change the budget requests by the High Council of Justice? 7. What is the status and prospects for delegation of procurement authority to local courts? What plans for monitoring and accountability? Component 4: Public Outreach 1. What is the process for reviewing and acting upon citizen suggestions (public comment box)? Can you provide specific examples? 2. Are there any indications that pro se litigants are better informed and prepared to represent themselves? 3. What continuing efforts to provide information and support to enhance the ability of pro se litigants to successfully represent themselves? SI’s Evaluation of USAID/Georgia’s JAMR Project XXIII Questions for High School of Justice 1. Which training projects have you work on with JAMR? a. Court Manager training and certification? b. DCC Budget training? c. Public service training? i. For court staff? ii. For judges? d. Case Management System training? e. HR training? f. Others? 2. When did you start working with JAMR? 3. What has your working relationship been with JAMR staff over the 4 years of the project? SI’s Evaluation of USAID/Georgia’s JAMR Project XXIV JAMR Interview Questions: Judicial Actors VISIT TO THE COURTS WILL INCLUDE REVIEW OF SOFTWARE OPERATIONS AND AUDIO RECORDING SYSTEMS, PUBLIC ACCESS AND INFORMATION FACILITIES AND OFFICE ARRANGEMENTS (FOR COURT MANAGER) TASK 1: EVALUATION OF PROGRESS TOWARDS PROJECT OBJECTIVES INSTITUTIONAL PERFORMANCE: Component 1: Improvement of Court Operations 1. What factors are making the greatest contribution to the reduction of case backlogs and case processing times? a. Better, faster service of process? b. Decline in continuances requested/granted? c. Better court administration? d. More cooperative attitude of practitioners, prosecutors, government agencies? e. More judges? f. Fewer cases being filed? g. Other reasons? 2. Are hardware/software systems (CMS, audio recording, etc.) in place and utilized? a. Case Management System? b. Audio recording system? c. Are people properly trained? d. What were the “bugs”? How were they fixed? Component 2: Court Administrators 1. Does your court have a certified court manager? 2. What was t is the background of the new manager and how were they were recruited? 3. How has the court allocated responsibility between the manager and the presiding judge? 4. Is the presiding judge taking more cases? 5. What has been the reaction of court staff to the new arrangement? 6. What has been the reaction of the other judges to the new arrangement? Component 3: Capacity to Develop and Manage Budget (Department of Common Courts) 1. Has your court become more involved in the budgeting process with the DCC? 2. Does this involvement give you a better voice for your court’s needs? 3. Would you like more flexibility in the procurement process for your court? Component 4: Public Outreach 1. What steps has your court taken to improve public perception? 2. To what extent have trainings about being more customer service oriented been effective? 3. Who at the court is responsible for public information? SI’s Evaluation of USAID/Georgia’s JAMR Project XXV 4. How does our court handle information requests under the Administrative Code’s access to information provisions? 5. Has your court taken any steps to assist pro se litigants? KEY INTERNAL AND EXTERNAL FACTORS RESPONSIBLE FOR SUCCESS/FAILURE (NEED TO INCLUDE MAJOR BARRIERS TO SUCCESS)? QUESTIONS ADDRESSED ABOVE DEGREE AND LIKELIHOOD OF SUSTAINABILITY OF JAMR ASSISTED IMPROVEMENTS Questions addressed above. EXTENT TO WHICH TARGET STAKEHOLDERS/INSTITUTIONS ARE AWARE OF JAMR INTERVENTIONS Same for all four components 1. To what extent did JAMR communicate with about their perception of the problems, needs? 2. To what extent were they satisfied with what JAMR has done? 3. To what extent were they satisfied with the JAMR’s communication with them during implementation and follow-up? TASK 2: EVALUATING RELEVANCE OF JAMR METHOLODY AND MONITORING SYSTEMS NO QUESTIONS FOR COURTS TASK 3: LESSONS LEARNED Questions for courts as they arise during conversations SI’s Evaluation of USAID/Georgia’s JAMR Project XXVI APPENDIX D. CLEARANCE RATES ACROSS JAMR PROJECT YEARS Year 1 Civil Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 3588 6161 6165 3592 1.72 Kutaisi 488 516 276 248 1.06 Mtskheta 247 304 177 120 1.23 Gori 521 600 226 147 1.15 Rustavi 261 281 88 68 1.08 Batumi 710 884 478 304 1.25 TOTAL 5815 8746 7410 4479 1.50 Administrative Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 2365 1508 1268 2125 0.64 Kutaisi 148 155 96 89 1.05 Mtskheta 62 80 92 74 1.29 Gori 132 187 133 78 1.42 Rustavi 73 79 34 28 1.08 Batumi 278 402 356 232 1.45 TOTAL 3058 2411 1979 2626 0.79 Criminal Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 4574 4896 2580 2258 1.07 Kutaisi 576 689 339 226 1.20 Mtskheta 216 291 149 74 1.35 Gori 513 630 543 426 1.23 Rustavi 395 377 54 72 0.95 Batumi 737 978 853 612 1.33 TOTAL 7011 7861 4518 3668 1.12 SI’s Evaluation of USAID/Georgia’s JAMR Project XXVII Year 2 Civil Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 3769 4250 3189 2708 1.13 Kutaisi 472 440 250 282 0.93 Mtskheta 195 179 74 90 0.92 Gori 113 138 91 66 1.22 Rustavi 331 314 166 183 0.95 Batumi 421 367 288 342 0.87 TOTAL 5301 5688 4058 3671 1.07 Administrative Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 1779 1821 1038 996 1.02 Kutaisi 146 142 98 102 0.97 Mtskheta 56 82 70 44 1.46 Gori 91 98 55 48 1.08 Rustavi 106 94 27 39 0.89 Batumi 184 205 177 156 1.11 TOTAL 2362 2442 1465 1385 1.03 Criminal Court Number of Cases Filed Number of Cases Disposed Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 3136 3176 1631 1591 1.01 Kutaisi 391 397 216 210 1.02 Mtskheta 138 155 60 43 1.12 Gori 322 448 335 209 1.39 Rustavi 304 331 73 46 1.09 Batumi 411 534 439 316 1.30 TOTAL 4702 5041 2754 2415 1.07 SI’s Evaluation of USAID/Georgia’s JAMR Project XXVIII * Source: JAMR Annual Reports (Years 1-3) Year 3 Civil Cases Court Number of Cases Filed Number of Cases Disposed   Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 5252 5654 2766 3048 1.07 Kutaisi 608 613 215 210 1 Mtskheta 192 187 68 69 0.97 Gori 397 391 138 144 0.98 Rustavi 312 333 177 156 1.06 Batumi 674 745 385 308 1.1 TOTAL 7435 7923 3749 3935 1.06 Administrative Cases Court # of Cases Filed # of Cases Disposed   Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 3076 3335 1289 1011 1.08 Kutaisi 175 201 79 52 1.14 Mtskheta 74 81 32 25 1.09 Gori 158 153 57 62 0.96 Rustavi 136 150 86 72 1.1 Batumi 361 381 186 160 1.05 TOTAL 3980 4301 1729 1382 1.08 Criminal Cases Court Number of Cases Filed Number of Cases Disposed   Number of Pending Cases at the Start of Reporting Period Number of Pending Cases at the End of Reporting Period Clearance Rate Tbilisi 3583 4037 1524 1007 1.12 Kutaisi 363 458 129 34 1.26 Mtskheta 126 144 43 25 1.14 Gori 264 319 106 51 1.2 Rustavi 257 294 71 34 1.14 Batumi 513 685 308 133 1.33 TOTAL 5106 5937 2181 1284 1.16 SI’s Evaluation of USAID/Georgia’s JAMR Project XXIX APPENDIX E: TIMELINE OF JAMR ACTIVITIES 2. The SC asked that component 1 be reduced, components 3 & 4 eliminated, and 75% of the budget allocated to the Court to develop its case management system. 3. The revised plan added $500,000 to component 1 and reduced funding for components 2, 3, & 4. Oct-07 Dec-07 Jan-08 Feb-08 Nov-07 1. The proposed changes included dropping the facilities improvement task and strengthening IT activities. The SC Working Group reflected the concerns of the leadership of the SC. Mar-08 Apr-08 May-08 Jun-08 Jul-08 Aug-08 Sep-08 JAMR Timeline - Year One (2007-2008) JAMR & USAID approve revised workplan Project begins Supreme Court (SC) WorkingGroup suggests changes to JAMR workplan1 JAMR provides initial IT assessment to SC SC requests significant restructuring of project2 War with Russia temporarily disrupts project JAMR conducts trainings and workshops for components 1, 2, & 3 througout Year 1 JAMR & USAID agree on revised plan3 SI’s Evaluation of USAID/Georgia’s JAMR Project XXX 1. The SC rejected the plan and RFP. Oct-08 Nov-08 Dec-08 Jan-09 Feb-09 Mar-09 Apr-09 May-09 Jun-09 Jul-09 Aug-09 Sep-09 JAMR Timeline - Year Two (2008-2009) JAMR and HCOJ jointly release JAMR's assessment of court administrative and management systems CourtUsers Survey ofsix pilots courts is released JAMR announces tender for purchase of audio recording systems JAMR presents automated case management system plan and RFP to SC1 JAMR presents case delay reduction strategy for pilot courts JAMR conducts trainings and workshops for components 2, 3, & 4 througout Year 2 SI’s Evaluation of USAID/Georgia’s JAMR Project XXXI 1. JAMR played a key role in drafting the legislation and developing position descriptions. 2. JAMR recommended dropping the pro se litigant case processing time reduction objective (Component 4) and the creation of Association of Court Administrators objective (Component 2) from the PMP. Oct-09 Nov-09 Dec-09 Jan-10 Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 JAMR Timeline - Year Three (2009-2010) USAID approves tender for audio recording equipment and contracts are signed Audio recording equipmentis installed in 80 courtrooms JAMR revises component 3 and drops strategic budgeting planning system from PMP JAMR initiates Bench‐Bar Committee meetings Trainings on audio recording systems are held for court staff JAMR presents conceptpaper on courtmanagers to HCOJ The second Court User Survey of pilot courses is completed. JAMR and High School of Justice prepare and present courtmanager training and certification A new law is enacted creating the court manager position1 JAMR conducts trainings and workshops across all 4 components throughout Year 3 JAMR recommends changesto PMP2 Information stands are installed in pilot courts SI’s Evaluation of USAID/Georgia’s JAMR Project XXXII * Source: Evaluation team analysis of JAMR documents and interviews. Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 JAMR Timeline - Year Four (2010-2011) Georgia Parliament adopts civil procedure rules change initiated by JAMR‐organized Bench Bar Committee 18 Court Managers are certified and hired JAMR consults with Department of Common Courts on Ministry of Finance program budgeting Comment boxes are placed in pilot courts JAMR organizes pro se litigation roundtable JAMR conducts trainings and workshops across all 4 components throughout Year 4 JAMR presentsjudicial public relations strategy plan to courts for future implementation For more information, please visit http://www.socialimpact.com Social Impact, Inc. 2300 Clarendon Boulevard Suite 300 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com