FINAL PERFORMANCE EVALUATION OF KAZAKHSTAN JUDICIAL PROGRAM (KJP) January 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc. FINAL PERFORMANCE EVALUATION OF KAZAKHSTAN JUDICIAL PROGRAM (KJP) This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number 06: 72011518D00003/72011519F00012 Learning from Evaluations & Assessments for Development (LEAD) It was prepared independently by: Dr. Roland Gjoni, Team Leader (ME&A) Nazgul Yergalieva, Justice Expert Alfira Senkina, Logistics Coordinator/Interpreter Contractor: ME&A, Inc. 4350 East-West Highway, Suite 210 Bethesda, MD 20814 Tel: 301-652-4334 www.MEandAHQ.com Cover photo: International Conference on “Further Improvement of Investment Proceedings,” held in Nursultan on May 31, 2019. Photo Courtesy of US Embassy in Kazakhstan/Facebook Profile. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The final evaluation of the United States Agency for International Development Central Asia (USAID/Central Asia) Kazakhstan Judicial Program (KJP), implemented by the American Bar Association/Rule of Law Initiative (ABA/ROLI), aimed to: 1) measure project success and identify any remaining gaps, 2) analyze monitoring and evaluation (M&E) systems and identify any issues with progress reporting, and 3) provide recommendations on future USAID interventions to increase public trust in Kazakhstan’s judicial system. The evaluation team (ET) used a mixed-methods approach including desk review of project reports and sector literature, key informant interviews and focus group discussions, and quantitative and qualitative analysis of KJP’s M&E system. KJP met its objectives by improving judicial decision writing capacity and enhancing court efficiency and accessibility. Facing limited support for judicial independence and accountability, KJP focused its assistance and resources on other areas. KJP’s interventions to increase Academy of Justice (AOJ) capacity had limited success due to institutional weaknesses and leadership changes. KJP worked to improve its M&E system to capture and report progress; however, most progress indicators relied on quantitative aspects (i.e., number of events, participants, trainings, materials distributed) rather than measuring the project’s effects and contribution. USAID should continue to support: 1) public trust in the judicial system, by working with stakeholders to strengthen judicial independence and accountability systems, further modernize the courts, and improve judicial education; and 2) public demand for a transparent, independent, and accountable judiciary, by helping civil society organizations increase monitoring and reporting capacity through evidence-based policy research and advocacy efforts. CONTENTS EXECUTIVE SUMMARY .............................................................................................................................................. i 1.0 EVALUATION BACKGROUND, PURPOSE, AND QUESTIONS .......................................................... 1 1.1 Evaluation Purpose............................................................................................................ 1 1.2 Evaluation Questions......................................................................................................... 1 2.0 PROJECT BACKGROUND............................................................................................................................... 2 3.0 EVALUATION METHODS AND LIMITATIONS........................................................................................ 3 3.1 Evaluation Methodology.................................................................................................... 3 3.2 Evaluation Limitations ....................................................................................................... 6 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS................................................................. 8 4.1 EQ1: To what degree has the ABA/ROLI met the two objectives as stated in the cooperative agreement and what gaps in addressing the program objectives still exist? 8 4.1.1 Objective 1 .............................................................................................................................. 8 4.1.2 Objective 2............................................................................................................................. 12 4.1.3 Remaining programmatic gaps........................................................................................... 15 4.2 EQ2: What issues exist in the monitoring system established by ABA/ROLI which might have affected its effectiveness in reporting on KJP progress?.........................................19 4.2.1 Strengths................................................................................................................................. 20 4.2.2 Weaknesses........................................................................................................................... 21 4.3 EQ3: Going forward, how, and in which areas, should USAID work to increase public trust in the judicial system of Kazakhstan? .....................................................................24 4.3.1 Political developments and their impact on judicial reforms..................................... 24 4.3.2 Working methods................................................................................................................ 28 4.3.3 Priority areas of justice sector TA................................................................................... 30 4.3.4 CSO engagement in judicial programs............................................................................ 32 ANNEXES ………………………………………………………………………………………………..35 Annex 1: KJP Evaluation Statement of Work .................................................................................36 Annex 2: Matrix of Findings, Conclusions, and Recommendations................................................39 Annex 3: Evaluation Instruments/Illustrative Questions for KIIs and FGDs...................................44 Annex 4: Data Sources....................................................................................................................47 Annex 5: List of Documents Reviewed...........................................................................................49 Annex 6: Field Work Schedule .......................................................................................................51 LIST OF TABLES Table 1: Key Informants Interviewed............................................................................................................................ 4 Table 2: FGD with Justice System Participants........................................................................................................... 5 Table 3: Respondents by Data Collection Instruments............................................................................................ 5 Table 4: Respondents by Location and Gender......................................................................................................... 5 Table 5: Judge Respondents by Position and Gender............................................................................................... 6 Table 6: KJP Decision Writing Manuals........................................................................................................................ 9 Table 7: KJP Judicial Exchange Programs................................................................................................................... 11 Table 8: Kazakhstan in World Ratings........................................................................................................................ 26 ACRONYMS Acronym Description ABA/ROLI American Bar Association/Rule of Law Initiative ADR Alternative Dispute Resolution AIFC Astana International Financial Center AOJ Academy of Justice AMEP Activity Monitoring and Evaluation Plan AOR Agreement Officer Representative CLE Continuing Legal Education COP Chief of Party CSO Civil Society Organization CVE Countering Violent Extremism EQ Evaluation Question ET Evaluation Team EU European Union FCE For Cost Extension FGD Focus Group Discussion FY Fiscal Year GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GOK Government of Kazakhstan HJC High Judicial Council IP Implementing Partner IT Information Technology KJP Kazakhstan Judicial Program KII Key Informant Interview M&E Monitoring and Evaluation ME&A ME&A, Inc. MOJ Ministry of Justice MOU Memorandum of Understanding NCE No-Cost Extension NGO Non-Governmental Organization OECD Organization of Economic Cooperation and Development OSCE Organization for Security and Cooperation in Europe PMP Performance Management Plan PMEP Performance Management and Evaluation Plan SC Supreme Court SOW Statement of Work TA Technical Assistance TOT Training-of-Trainers U.S. United States UNDP United Nations Development Program Acronym Description UOJ Union of Judges USAID United States Agency for International Development USG United States Government WB World Bank i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the final evaluation of the Kazakhstan Judicial Program (KJP), funded by the United States Agency for International Development Central Asia (USAID/Central Asia) and implemented by the American Bar Association/Rule of Law Initiative (ABA/ROLI), was to: 1) determine the extent to which the KJP achieved the two objectives set forth in the Cooperative Agreement with USAID and identify any remaining gaps; 2) analyze the internal monitoring and evaluation (M&E) systems of ABA/ROLI and identify any issues that may have affected the effectiveness of KJP’s progress reporting; and 3) provide recommendations on future USAID assistance to increase the public trust in the judicial system of Kazakhstan. The primary audience for the KJP evaluation is the USAID/Central Asia Mission, which may use the results to determine the methods and priorities for further assistance to Kazakhstan’s judiciary. PROJECT BACKGROUND On April 16, 2012, USAID awarded ABA/ROLI a grant of $1,950,000 to implement a four-year initiative called the Kazakhstan Judicial Independence Program (KJIP) (Cooperative Agreement AID-176-A-12- 00011) until April 15, 2016. The KJIP would focus on increasing the independence and accountability of Kazakhstan’s judicial system by pursuing two objectives: 1) Objective 1: Improving administrative systems within Supreme Court (SC); and 2) Objective II: Strengthening managerial capacity at the SC. Following discussions with the SC leadership and USAID, a year later, ABA/ROLI submitted a program and budget modification to USAID. Upon USAID’s approval on November 1, 2013, the project name changed to KJP and project components dealing with court administration and other aspects of judiciary (i.e., judicial selection, appointment, and discipline) were removed from the scope. The KJP would focus on modernizing the judicial system and improving public trust in Kazakhstan’s courts through two revised objectives: 1) Objective 1: Improving the capacity of judges; and 2) Objective 2: Enhanced efficiency and accessibility of the judicial system. USAID awarded ABA/ROLI a No-Cost Extension (NCE) extending KJP’s period of implementation from April through July 30, 2016 and, in July 2016, an additional $2,100,000 For Cost Extension (FCE) for 38- months through September 30, 2019. KJP now focused on equipping judges with skills and knowledge to better adjudicate cases based on the newly-adopted civil and criminal legislation. After consulting USAID, in September 2016, ABA/ROLI revised KJP’s project objectives and Performance Management and Evaluation Plan (PMEP) to report on the following four objectives: Objective 1: Improve capacity of judges to consider cases in civil law; Objective 2: Improve capacity of judges to consider criminal cases; Objective 3: Improve capacity of the judiciary to handle investment disputes; and Objective 4: Improve public awareness of judicial modernization and court operation. In May 2019, USAID granted another 12-month FCE (through September 30, 2020) to focus primarily on improving judicial capacity to handle and adjudicate intellectual property rights. The cumulative amount of the KJP award is $4,800,000 (inclusive of all extensions). As per the Statement of Work (SOW), the evaluation assessed KJP’s performance from April 16, 2012 through September 30, 2019 to determine its progress—and any gaps—in achieving the approved objectives (as of November 2013), namely: 1) improved capacity of judges; and 2) enhanced efficiency and accessibility of the judicial system. ii EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation team (ET) used a mixed-methods approach with complementary qualitative and quantitative data collection methods, which included: desk review of project documents and literature on Kazakhstan’s judicial system, key informant interviews (KIIs) conducted in person and via written questionnaires and Skype calls, focus group discussions (FGDs), and quantitative data analysis of Performance Management Plan (PMP) reports. The ET conducted field work in Kazakhstan (Almaty and Nursultan) from December 1 through December 13, 2019. The ET held FGDs with city and district court judges in Almaty and Nursultan, SC justices, the Academy of Justice (AOJ), and the SC’s Department of Administration. Overall, the ET gathered opinions on the KJP’s performance from 71 respondents. The ET triangulated and analyzed evidence collected from the KIIs, FGDs, and desk review to reach conclusions and make recommendations. The team focused the analysis on findings that: a) recurred with relatively greater frequency; b) were common across data gathered through different methods; and c) came from multiple partners, beneficiaries, and stakeholders. The ET also used different forms of frequency, trend, theme, and pattern analysis to compare results across respondent groups. This allowed findings to reveal who benefitted from what types of the KJP activities as well as which elements of an activity were contributing to the achievement of outputs and outcomes, which elements were not, and, most importantly, why. Evaluation limitations included: selection bias, recall bias, social desirability bias, and difficulty of attribution. Furthermore, the ET faced challenges in conclusively assessing the extent to which objectives have been met due to the lack of baselines and the fact that a large number of progress indicators were designed to measure the output and delivery of technical assistance (TA). MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Evaluation Question (EQ) 1: To what degree has ABA/ROLI met the two objectives as stated in the Cooperative Agreement and what gaps in addressing the program objectives still exist? Findings • Results vary across objectives, but KJP has generally delivered the technical outputs it was asked to deliver. KJP met Objective 1 as its TA contributed to higher-quality written decisions and improved application of new legislation by judges. • KJP has largely been successful in achieving Objective 2. Despite weak public trust in the judicial system, there is evidence to support KJP’s positive contribution to the gradual improvement in efficiency and accessibility of courts. • ABA/ROLI has a strong reputation for delivering timely and high-quality assistance to the local beneficiaries based on agreed needs. • Despite the extent of KJP’s accomplishments, significant gaps remain in the initial objective (strengthening judicial independence and accountability) due to the lack of willingness of the SC to work with KJP in those areas at the start of the program. • KJP has provided TA to the AOJ, according to the approved work plans; however, due to institutional weaknesses which plagued the AOJ from 2012-2018 (i.e., lack of strong leadership, strategic vision, and effective management practices), the results and its sustainability have been limited. • Most activities at the AOJ were perceived as one-off initiatives by the beneficiary. For example, courses supported by KJP have yet to be formally integrated in the training curriculum for sitting judges or Master’s students and evidence is weak to suggest that training-of-trainers (TOT) programs improved teaching practices at AOJ courses offered to judges. iii Conclusions • KJP has been largely successful in achieving Objective 1 as evidenced by the higher quality of judicial decisions and improved judicial capacity to apply new laws. • KJP’s TA to improve the capacity of the AOJ to provide high-quality judicial education has been less successful due to the weak capacity of the beneficiary. • KJP’s contribution to expansions of alternative dispute resolution (ADR) mechanisms—mediation and arbitration—are showing positive results. KJP’s role as a synergy provider and enabler for dialogue between courts and media has contributed to enhancing court openness and transparency under Objective 2. • Under both objectives, due to the predominantly on-demand nature of the TA and the high number of output-oriented indicators, it is difficult to reliably determine the effect of KJP; however, partners confirm the positive contribution of KJP’s deliverables. • Remaining gaps in addressing the original objectives (i.e., improving judicial selection, judicial independence, impartiality, evaluation of judges, etc.) exist but were not a result of KJP’s technical capacity. • KJP’s ability to address key challenges of the judicial system, such as independence and accountability, was limited by the lack of willingness on the part of the SC at project start. • KJP’s TA to improve the capacity of the AOJ to provide higher-quality judicial education has been less successful despite the delivery of technical outputs by the KJP. The effect and sustainability of KJP’s assistance are limited due to the weak capacity of the beneficiary and frequent leadership changes at the AOJ. Recommendations • USAID should continue to work on strengthening judicial independence and accountability procedures and systems in accordance with national strategies and action plans of the Government of Kazakhstan (GOK) with the aim of increasing public trust in the judiciary. • USAID should conduct further analysis on the current needs of the AOJ and provide further support to the AOJ, taking into account the ongoing United Nations Development Program (UNDP) interventions and future plans. EQ 2: What issues exist in the monitoring system established by ABA/ROLI which might have affected its effectiveness in reporting on KJP progress? Findings Strengths • ABA/ROLI’s M&E system has evolved over program lifespan, due to both KJP team and home office efforts. ABA/ROLI diligently collects, processes, reports, and stores M&E data. • ABA/ROLI’s home office has provided important support to incrementally strengthen the qualitative aspects of monitoring for results. • Following internal assessments by the ABA/ROLI Evaluation Unit in Washington, DC, KJP has a designated M&E Officer to cover tasks previously shared between the Chief of Party (COP), staff attorneys, and program managers. Weaknesses • Almost all indicators under Objective 1 are output-oriented focused on reporting the activities and the delivery of TA in the form of a number—e.g., number of decision-writing manuals developed, seminars conducted, resource material provided, etc.—which are not designed to capture change in the capacity of judges. • The only indicator intended to measure the increase in the quality of decisions (Indicator 1.2.6) is conceptualized as a form of “before and after” comparative analysis to measure change; however, its implementation raises questions of the validity of the findings. iv • Objective 2 indicators are appropriate to report output but less useful to demonstrate how KJP activities have led to change in capacity—either through demonstrable use of the introduced practices or discernible change in the capacity or behavior of beneficiaries. • Although the annual report indicates that ABA/ROLI coordinated an evaluation of the program by an independent evaluator, project documents also indicate that the same expert has been contracted by ABA/ROLI for various assignments related to the activity she was supposed to evaluate. Conclusions • ABA/ROLI has diligently worked to improve its M&E system by dedicating more time, home office involvement, and in-country human resources to refine the progress indicators, diversify the data collection methods, and track progress towards meeting program objectives. • Overall, the qualitative aspects of progress monitoring remained weak until 2016 as various factors constrained the ability of ABA/ROLI to effectuate, track, and verify impact beyond output delivery. • The introduction of qualitative elements—allowing for “before and after” comparison based on the baseline data generated in 2016—is positive. • The validity and reliability of the findings for the first comparative study in July-August 2019 could have been strengthened by engaging outside experts more removed from the research activity. Recommendations • USAID should continue to encourage the integration of qualitative element in ABA/ROLI’s M&E plans, e.g., collect thematic case studies to identify successful results in certain areas of assistance (countering violent extremism [CVE], ADR, investment justice) to better describe KJP’s contribution and added-value. • USAID should consider asking implementing partners (IPs) to utilize Outcome Harvest and Complexity Aware Monitoring Approaches to better document the impact of their interventions. EQ 3: Going forward, how, and in which areas, should USAID work to increase public trust in the judicial system of Kazakhstan? Findings Working Methods • There is a growing understanding in the GOK on the need to increase public trust in state institutions and the judicial system, but most respondents questioned the credibility of political leadership’s commitment to reforms and the extent to which reforms will genuinely lead to the enhanced independence and accountability of the judicial system. • Although judicial independence and accountability systems remain incomplete, there are positive trends which may facilitate further programmatic work to increase public trust. • Currently, conditions for civil society organizations (CSOs) inclusion in programming appear more favorable than five years ago as the GOK has declared its intention to engage civil society in its efforts to respond to public dissatisfaction through the Council of Public Trust. However, the GOK controls the agenda, process, timing, and sequencing of planned CSO engagement in implementing reforms aimed at increasing the public trust in state institutions. • Reformers within the justice system are few and not univocal, and resistance to comprehensive reforms is high among influential actors within and outside the judicial system. • USAID’s reputation and KJP’s diligent performance constitute a valuable asset for future programming in the judiciary. • On-demand TA in the areas where United States Government (USG) and GOK priorities converge facilitates results for the implementer but may constrain USAID’s ability for more impactful programming to increase public trust. v Areas of Work • In accordance with GOK’s multi-annual justice sector-strategies and action plans, more work is needed to upgrade and improve the quality of procedural/substantive law in both criminal and civil justice. Justice system stakeholders welcome USAID support for further modernization of court processes/administration. • Despite KJP’s work, more targeted assistance to strengthen the judicial education and professional competence of judges and AOJ’s capacity is needed. • Judges need more training on topical areas of law (i.e., international human rights law and their protection, cybercrimes, intellectual property rights, advanced programs on ADR, CVE, intellectual property, consumer protection law, etc.). • Exposure to international best practices, peer-to-peer professional exchange, and reform support is seen as beneficial by informants, within and outside the justice sector. • Union of Judges’ (UOJ) capacity to demand more independence and accountability for judges (i.e., constructive advocacy efforts, training for judicial ethics, strengthening committees to conduct comparative studies) is currently lacking and would benefit from USAID’s assistance. • More work is needed in strengthening CSO court monitoring and reporting capacities and using evidence-driven research to educate public demand for more open and independent courts. This would require a significant upgrade in CSOs’ capacity to plan, execute, and disseminate evidence￾based research and implement CSO public advocacy campaigns on judicial competence, integrity, independence, financial asset disclosure, and avoidance of conflict of interest by judges. Recommendations Working Methods • Due to lack of political will for broad rule of reforms, USAID should monitor and reassess political developments in the design and implementation stage of the next judicial sector program. • Any future judicial program should attempt to be more ambitious in working with the High Judicial Council (HJC), SC, AOJ, and UOJ, and go beyond providing TA in the areas agreed with justice institutions, despite the fact that on-demand assistance appears to be more of a “safe instrument.” • To increase public engagement and trust in judicial system, USAID should consider combining two streams of work in the next program: 1) working with HCJ and SC on strengthening judicial independence and accountability; and 2) supporting CSO research and advocacy for a more transparent and accountable justice system. Areas of Work • Support the HJC and SC to upgrade and streamline laws and regulations to strengthen the independence and accountability of judges (i.e., selection, appointment, promotion, evaluation, and dismissal of judges). • Continue the support to the SC multi-annual plans to reform the civil and criminal procedure legislation and court modernization initiatives. • Offer further TA to the AOJ on improving formative and continuous judicial education programs. This component may focus on institutional strengthening and curricula development in topical areas of law (i.e., human rights protection, cybercrime, CVE, investment disputes, consumer protection). In particular, USAID should consider providing more training in international human rights law and the case law/jurisprudence of international human rights protection bodies (European Union, Inter-American, and African courts on human rights, United Nation committees). • Continue supporting SC plans for expansion of ADR use, both in court and outside court, through public awareness campaigns and introduction of legal changes with smart incentives/penalties aimed at making ADR the default mechanism for a larger category of cases. vi • USAID should consider CSO engagement in the future judicial program and work to enhance local non-governmental organization capacities to conduct research on topics related to judicial independence, judicial competence, impartiality, and integrity. Evidence-driven research and advocacy efforts should be designed to raise public awareness on the functionality of the judicial system and educate public demand on judicial independence and accountability. • Support citizen information campaigns on the court system and judicial reforms—particularly targeting youth—in the Kazakh language. 1 1.0 EVALUATION BACKGROUND, PURPOSE, AND QUESTIONS 1.1 EVALUATION PURPOSE The United States Agency for International Development Central Asia (USAID/Central Asia) contracted with ME&A, Inc. (ME&A) to conduct the final performance evaluation of the Kazakhstan Judicial Program (KJP), a justice sector program implemented by the American Bar Association/Rule of Law Initiative (ABA/ROLI) through a Cooperative Agreement since April 2012. Initially, USAID awarded ABA/ROLI a grant of $1,950,000 under the exceptional capability clause to implement a four-year project called “Kazakhstan Judicial Independence Program” (Cooperative Agreement AID-176-A-12-00011). Following discussions with the Supreme Court (SC) leadership on the potential areas of cooperation, ABA/ROLI submitted a program and budget modification, which became effective November 1, 2013 upon USAID’s approval. After this change, the KJP worked with the SC, Academy of Justice (AOJ), and Union of Judges (UOJ) on modernizing the judicial system and improving public trust in Kazakhstan’s courts on the basis of two revised objectives as follows: Objective 1: Improving the capacity of judges; and Objective 2: Enhancing the efficiency and accessibility of the judicial system. ABA/ROLI received a 38-month For Cost Extension (FCE) in July 2016 for a total amount of $2,100,000 through September 30, 2019, and modified the program objectives, from two to four, without modification of the initial Cooperative Agreement. In May 2019, USAID granted ABA/ROLI another 12-month cost extension of an additional $750,000; KJP’s expected end date is now September 30, 2020. The purpose of the final evaluation of KJP is to: 1) determine the extent to which ABA/ROLI has met the two stated objectives and identify any remaining gaps in addressing the program objectives; 2) examine any issues in the monitoring system established by ABA/ROLI that might have affected its effectiveness in reporting on KJP’s progress; and 3) provide recommendations on the areas and methods of work for future programmatic assistance of USAID to increase public trust in the judicial system of Kazakhstan. The evaluation was conducted from October 15, 2019 through January 2020 by an evaluation team (ET) consisting of an international expert, Dr. Roland Gjoni (Team Leader), and a local expert, Ms. Nazgul Yergalieva (Justice Expert). During the field research, the ET was ably assisted with translation, interpretation, and logistics by Ms. Alfira Senkina. 1.2 EVALUATION QUESTIONS The evaluation answered three evaluation questions (EQs) set out by USAID/Central Asia: 1. EQ 1: To what degree has ABA/ROLI met the two objectives as stated in the Cooperative Agreement and what gaps in addressing the program objectives still exist? 2. EQ 2: What issues exist in the monitoring system established by ABA/ROLI which might have affected its effectiveness in reporting of KJP progress? 3. EQ 3: Going forward, how, and in which areas, should USAID work to increase public trust in the judicial system of Kazakhstan? The ET used the above questions to organize the entire evaluation, from determining the research methods, categories of respondents, and data collection tools to structuring the findings, conclusions, and recommendations of this evaluation report. 2 2.0 PROJECT BACKGROUND Since gaining independence in 1991, Kazakhstan has made great efforts to reform and modernize its justice system. The United States Government (USG), through USAID, has supported Kazakhstan’s justice and rule of law reforms since 1993. Almost three decades later, USAID still remains vitally involved in Kazakhstan on issues touching upon economic growth, social development, and good governance. Prior to the KJP, rule of law programs provided technical assistance (TA) to legislative reforms, strengthening judicial training and professional competence of judges, and a court recording program launched under the previous USAID program, which ended in September 2011.1 To continue its support in the sector, USAID developed a new judicial sector program in 2011. The program initially aimed to enhance judicial independence and accountability by working with the SC, UOJ, and Institute of Justice under the Academy of Public Administration (since 2016, renamed the AOJ). From April 2012 through November 1, 2013, the two objectives of the program were: • Objective 1: Improved administrative systems within the SC; and • Objective 2: Strengthened managerial capacity at the SC. On December 15, 2012, Kazakhstan launched a new strategy setting the course for modernization of the judicial system and increasing court efficiency through introduction of alternative dispute resolution (ADR) and other measures to improve public trust. In the light of these developments and the expressed unwillingness of the SC to cooperate with KJP in the implementation of project components related to judicial independence and administration, ABA/ROLI proposed a modification of the program description and budget allocation which USAID approved on November 1, 2013. The main goal after the change in program description was to support the modernization of and increase the public trust in Kazakhstan’s judicial system. The objectives from November 1, 2013 through September 30, 2016 were aligned with the SC’s priorities and modified as follows: • Objective 1: Improved capacity of judges; and • Objective 2: Enhanced efficiency and accessibility of the judicial system. Strengthening the judicial system of Kazakhstan was one of the strategic objectives of the USAID Regional Development Cooperation Strategy for Fiscal Year (FY) 2015-2019 launched on May 29, 2014. Future modifications of the KJP have slightly changed the project priorities, although they have been in line with USG objectives of: governing justly and enhancing the rule of law and human rights in Kazakhstan; and strengthening the judiciary’s capacity to meet international standards of accountability, professionalism, and impartiality to better protect citizens’ rights and uphold the rule of law in the courtroom. In July 2016, USAID awarded a 38-month extension until September 30, 2019. The extension was necessitated by the rapid developments in the judicial system and responded to the SC’s request for assistance in the development of legislation and regulations to promote economic and investment courts, streamline judicial decision-writing, and promote the use of mediation/arbitration in commercial disputes. In particular, during the extension period, KJP was asked to focus on the priorities set out in cooperation between the United States and Kazakhstan in the field of two strategic aspects—countering violent extremism (CVE) and improving the investment climate for international investors. These priorities were identified as among the main directions of an enhanced strategic partnership between the United States and Kazakhstan when President Nazarbayev visited the White House in January 2018. During the extension period October 1, 2016 to September 30, 2019, ABA/ROLI partnered with the SC, AOJ, and 1 This was done through a Firm Fixed Price Request for Proposals (RFP) and through a Request for Applications (RFA) over a period of eight months. In addition, the Democracy and Governance Office (DGO) explored the possibility of engaging a local non-governmental organization (NGO) as an implementing partner (IP) but concluded that this was not feasible due to lack of capacity to process USAID funding and lack of willingness by the SC to work with local NGOs. 3 UOJ in pursuing four objectives: • Objective 1: Improved capacity of judges to consider cases in civil law; • Objective 2: Improved capacity of judges to consider criminal cases; • Objective 3: Improved capacity of the judiciary to handle investment disputes; and • Objective 4: Improve public awareness of judicial modernization and court operation. In 2019, USAID granted another 12-month FCE until September 2020 during which ABA/ROLI will pay special attention to improving judicial capacities to adjudicate intellectual property rights. It will also continue to work on the areas of investment disputes, violent extremism, and court transparency. While at the time USAID developed the KJP, justice system beneficiaries were not ready to address the key challenges of judicial independence and accountability; however, over the life of the program, modernizing the court system and increasing the public trust in the judiciary have been high on the agenda of the Government of Kazakhstan (GOK). Legislative and institutional reforms have accelerated since December 2012 after the announcement of a new national development strategy, titled “Strategy Kazakhstan 2050: A New Political Direction for a Successful Country.” In May 2015, a set of measures called “100 Concrete Steps” was adopted to further implement the goals set out in the “Strategy 2050.” Chapter II of the 100 Concrete Steps is dedicated to “Ensuring Rule of Law” and Steps 16 to 34 include multiple initiatives related to rule of law and judicial system calling, inter alia, for further modernization of the court system and enhancing public confidence in judiciary. In January 2018, the new Chairman of the SC announced several new initiatives the judiciary to implement the Kazakhstan 2050 Strategy, in the “100 Concrete Steps” and the “Rukhani Zhangyru,” a GOK Program for Modernization of Public Consciousness. The package of reforms, called “Seven Pillars of Justice,” aims to enhance the rule of law and confidence of the public in the judiciary. In 2018, a working group on judicial system modernization under the leadership of the Head of State launched new initiatives to make judicial selection more competitive and increase judicial salaries to attract competent lawyers to judicial office. Judicial reforms have continued swiftly with the introduction of more adversarial elements in the criminal justice sector, including: introducing jury trials, investigative judges and summary proceedings; expanding ADR; and making courts more accessible and efficient through application of e-Justice and other measures. Despite the notable progress, Kazakhstan’s political transformation has lagged behind its economic development. Kazakhstan’s ranking in the international indices on rule of law have only slightly improved over the life of the KJP. The Judiciary still suffers from low public trust and widespread perceptions of lack of independence, impartiality, and judicial competence. As the KJP nears its completion, USAID commissioned this evaluation to assess the work of KJP and consider how and in which areas should potential assistance focus in order to increase public trust in the judicial system. 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY The ET used a mixed-methods approach for this evaluation consisting of four main data collection methods, namely: 1) desk review; 2) key informant interviews (KIIs); 3) focus group discussions (FGDs); and 4) analysis of quantitative data extracted from KJP progress reports and documents of justice system stakeholders. The ET designed data collection instruments to answer the three EQs put forward by the USAID which, in accordance with the Statement of Work (SOW), covered the KJP’s performance from April 2012 through September 2019. 4 After the effective date of the contract on October 15, 2019, the ET started the desk research and submitted the draft work plan and evaluation methodology on November 9, 2019. Desk review included three elements: 1) project documents provided by USAID and KJP; 2) national strategies, action plans, laws, and regulations and other documents from GOK and justice institutions; and 3) literature review of existing reports on the state of judiciary in Kazakhstan (see more details under EQ 3). The ET conducted field work in Kazakhstan in Almaty (December 1-6, 2019) and Nursultan (December 6-13, 2019). In addition to meeting with relevant USAID and ABA/ROLI staff members, the ET conducted KIIs and FGDs in Almaty and Nursultan—meeting with a total of 71 respondents—including SC justices, Almaty City Court and district court judges, Nursultan city judges, AOJ senior management, UOJ, and a number of lawyers, mediators, journalists, and representatives of local civil society organizations (CSOs), international organizations and donors working in the rule of law sector. Quantitative Research Approaches and Analysis The ET used quantitative data from KJP project reporting from April 2012 to September 2019 to examine KJP’s successes in meeting its objectives as well as to identify remaining gaps in areas where programmatic activities were less successful to meet the anticipated objectives. The ET used the indicators developed in the project’s original Performance Management and Evaluation Plan (PMEP) in 2012, as revised in November 2013 and October 2016, to gauge the extent of the accomplishment of the KJP’s objectives for each objective. Although targets and indicators have undergone several changes in response to the modified program description and objectives, the ET made efforts to analyze the indicators which most closely related to the objectives set out in the SOW of this evaluation mission, namely: 1) improved capacity of judges; and 2) enhanced efficiency and accessibility of courts. While the length of the KJP and the technical nature of activities made it difficult to collect quantitative data from beneficiaries through KIIs and FGDs, the ET tried to collect and analyze pertinent data from official statistics of the SC or other justice institutions to triangulate or corroborate findings emerging from quantitative analysis of the KJP progress indicators and qualitative statements from informants. Qualitative Research and Analysis As per the SOW, this evaluation was primarily of a qualitative nature but involved analysis of relevant quantitative data to assess the Monitoring and Evaluation (M&E) System based on which KJP reported its progress. The ET collected documents or literature relevant to the work of KJP with the judicial sector made available to the ET by USAID, ABA/ROLI, and other organizations for the purpose of generating qualitative or quantitative data to answer the EQs. The data collected were analyzed through different forms of frequency, trend, theme, and pattern analysis to compare results across respondent groups. This process enabled the ET to qualitatively determine who is benefiting from what types of KJP activities, which elements of an activity are contributing to the achievement of outputs and outcomes, what programmatic gaps may still exist, and, most importantly, why such gaps remain. Through this, qualitative data collection was used to assess the extent to which the project is addressing the USAID objectives, responding to the needs of the project beneficiaries, and achieving the expected impact in improving judicial capacity, transparence, and accessibility. Table 1: Key Informants Interviewed Category Women Men Total Justice Institutions Judge Coordinator/SC 1 0 1 UOJ 0 2 2 Press Secretaries 1 1 2 CSOs/Think Tanks 2 0 3 Lawyers 1 2 3 5 Category Women Men Total Mediators 2 0 2 Arbitrators 1 2 3 Media Organizations 1 1 2 International Consultants on ADR 1 2 3 Journalists 1 0 1 International Donors and Partners 6 5 11 ABA/ROLI 3 0 3 USAID 3 1 4 Total 23 17 40 Due to the large number of judges to be interviewed in a short time and their busy trial schedule, the ET held several FGDs in Almaty and Nursultan with judges of city and district courts and justices of three panels of the SC. FGDs were also held with AOJ senior management, representatives, and the Court Administration Department of the SC. The Chief Justice of the SC and the chair of the civil law panel of the SC who initially confirmed the time for an interview were unavailable to meet in person due to schedule conflicts. Table 2: FGD with Justice System Participants Category Women Men Total Civil Panel of SC 2 1 3 Criminal Panel of SC 2 2 4 Investment Panel of SC 2 3 5 Department of Administration 2 2 4 Almaty City Judges 3 2 5 Nursultan City Judges 5 0 5 AOJ 2 3 5 Total 18 13 31 The ET also sent written questionnaires and organized several conference calls with several informants who were unavailable for face-to-face interviews during the field work. The team interviewed a total of 71 data sources, inclusive of KIIs, FGD, written questionnaires, and telephone interviews: Table 3: Respondents by Data Collection Instruments Instrument Respondents KIIs 34 Written Questionnaires 3 Telephone Interview 3 FGD 31 Total 71 Despite minor changes in the interview schedule, the ET was able to meet a variety of respondents, of both genders, including judges, lawyers, mediators, arbitrators, journalists, and informants from local and international organizations in two locations. Table 1: Respondents by Location and Gender Location Women Men Total Almaty 13 5 18 Nursultan 34 15 49 Other 3 1 4 Total 50 21 71 6 Table 5: Judge Respondents by Position and Gender Position Women Men Total SC 8 6 14 City Courts 6 2 8 District Courts 2 0 2 Total 16 8 24 The ET triangulated evidence from different qualitative and quantitative methods and from different data sources to strengthen the credibility of its findings. The ET also focused on findings that: 1) recurred with relatively greater frequency; 2) were common from across data gathered through different methods; and 3) were generated from a variety of KJP beneficiaries, non-beneficiaries, and local and international partners and stakeholders. The ET used these findings and conclusions to make recommendations to USAID. Data Analysis The ET used a variety of data analysis techniques to support the development of evaluation findings, conclusions, and recommendations. As instructed by USAID in the SOW, the ET also reviewed international reports on the legal and political context which influenced KJP’s objectives and outcomes and Kazakhstan’s ratings by the international indexes on the rule of law. Data collected during the field work were used to identify emergent themes or hypotheses and refine evaluation research tools and strategies. Over the course of the evaluation, the ET held regular debriefs and brainstorming sessions to analyze emerging findings on KJP’s performance. Preliminary findings and conclusions were captured in a matrix that categorized findings, conclusions, and recommendations by EQ and was used as living document during the field work. A summary of preliminary findings, conclusions, and recommendations was shared with USAID on December 13, 2019. 3.2 EVALUATION LIMITATIONS The ET anticipated several limitations that are common in the design of most evaluations and took steps to manage and mitigate them over the course of the evaluation. Limitations encountered and mitigated by the ET included: • Selection bias: The fact that the KJP has been working for over seven years posed a respondent selection challenge because changes in the judicial administration personnel made it difficult to identify people who were in a position to provide relevant information. To identify key stakeholders and beneficiaries, the ET used the project reports and worked with USAID and KJP to obtain contact information. The team frequently shared the schedule of meetings with USAID. To increase the diversity of data sources, the ET worked with KJP to facilitate introductions and outreach to partners, beneficiaries, and stakeholders for KIIs and FGDs. Although there was the risk of selection bias, ET tried to minimize the potential for it by expanding the pool of respondents to individuals and organizations who were not beneficiaries of the KJP but had relevant knowledge on the state of rule of law and ongoing reforms. • Availability of Respondents: The ET had only nine working days in Almaty and Nursultan for the field work and could not reschedule meetings with a few key informants who initially agreed to meet but had to cancel due to higher priority engagements (i.e., Chief Justice of the SC, Chair of the Civil Law Panel). Similarly, four city and district court judges from Nursultan could not attend the FGDs due to their trial schedules. The ET would have benefited from discussions with the Ministry of Justice (MOJ) about the topics related to the EQ 3. Due to lack of response from MOJ and field work time constraints it was not possible to organize a meeting with senior-level officials of the MOJ. Notwithstanding the time limitations, the ET managed to collect opinions from a sufficiently large pool of respondents that allows a “within sample triangulation” of diverse 7 perspectives and enable the ET to minimize selection bias through “out of sample triangulation” of individuals and organizations that did not directly benefit from KJP activities. • Recall bias: Since several questions raised during the interviews dealt with activities that took place in the past, recall bias cannot be excluded. KJP is an unusually long project with various transformations since April 2012 and project counterparts and recipient institutions often found it difficult to recall activities from the earlier years. Additionally, due to the presence of various TA projects by the World Bank (WB), European Union (EU), Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), United Nations Development Program (UNDP), and others, many respondents could not distinguish KJP activities from those of other organizations. This was further exacerbated by the fact that with the exception of a limited number of direct beneficiaries, many stakeholders are not interested or able to always discern various international projects and tend to use general descriptors like the “internationals” or sometimes refer to USAID projects as “Americans” and EU projects as “Europeans” without specific names. The ET tried to provide the interviewees with advance notice and details of the project being assessed to mitigate recall bias within USAID projects and across donor projects. • Attribution difficulties: Due to recall bias mentioned above, it was difficult for some respondents to clearly distinguish the contribution of KJP. During the KII, the ET mitigated this limitation as well as the recall bias limitation by describing the KJP activities more extensively so that informants could help the ET clarify the activity under discussion and avoid confusing it with those of other donors. The ET observed that, due to the purely technical level of the assistance and a certain level of co-creation of project deliverables between the SC and KJP, only a small number of respondents could attribute the success to the KJP. A large number of the non-judge respondents viewed the activities organized by the KJP as SC activities and could not attribute the effect or lack thereof to the KJP. The SC also appeared to work with various donors on different areas without necessarily explaining to judges which donor is working on what project. Donors also felt that sometimes the SC was managing rather than coordinating international projects, which was evident in the lack of detailed knowledge among donors on each other’s activities. To mitigate this limitation, ET used specific prompts to ask some interviewees about their interaction with KJP and their overall impressions of the delivery of the TA before getting into more specific questions about KJP’s objectives and accomplishments. • Social Desirability Bias/Halo Effect: The ET also faced “halo effect” bias, the known tendency among respondents to under-report socially undesirable answers and alter their responses to what they perceive as the social norm. The extent to which respondents were prepared to reveal their true opinions may also have varied for some questions that called upon the respondents to assess the performance of their colleagues or people on whom they depend upon for the provision of services. In this evaluation, the need to be cautious was also imminent as various respondents expressed their unease to say anything critical about a foreign organization like USAID which is offering important support to the people of Kazakhstan and the state institutions (see EQ 2). The extent to which respondents were prepared to reveal their true opinions varied, depending upon the question asked, which, on a few occasions, denied the ET insight into the assessment of KJP’s work particularly as it relates to discernible effect of their activities. The vague answers offered could have been the result of respondents not wanting to provide information that appears to assess the performance of their colleagues or, despite promises to the contrary by the ET, perceptions that their statements would not be kept confidential. When interviewees were less forthcoming or seemed to only provide overwhelmingly positive responses, the ET asked more direct questions about areas where progress may have been less evident or whether there were any remaining gaps. 8 • Difficulty assessing the impact of the KJP: The ET observed that many of the respondents were not fully aware of what the KJP’s objectives were, what the situation was when KJP began, and what was intended or achieved over the course of program so KIIs were of limited value to properly assess the extent of KJP’s impact. The ET relied mostly on the KJP M&E data to assess progress towards meeting objectives given that most informants did not have specific, detailed knowledge about what KJP objectives, definitions, and measurements of success. In addition, the objectives upon which KJP reported progress changed in 2013 and 2016. Many of the indicators and targets determined had to change to respond to the modified objectives. Particularly since 2016, progress towards meeting objectives was largely framed in terms of successful outputs and activities based on the demands of the SC. This can also be discerned in the way KJP reported progress after shifting from two to four objectives when ABA/ROLI more clearly tracked and reported progress on a set of technical deliverables requested by the SC and agreed by USAID (see EQ 2). 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 EQ1: TO WHAT DEGREE HAS THE ABA/ROLI MET THE TWO OBJECTIVES AS STATED IN THE COOPERATIVE AGREEMENT AND WHAT GAPS IN ADDRESSING THE PROGRAM OBJECTIVES STILL EXIST? FINDINGS 4.1.1 Objective 1 For the purpose of evaluating the KJP’s work to improve the quality of judicial decisions, the ET assessed the activities related to this objective over the years taking into account various changes to the objectives and outcomes made in November 2013 and September 2016. The following presents the findings for the KJP Objective 1 by examining KJP activities related to increased judicial capacity to write better decisions, improved application of new legislation, and increased educational capacity of the AOJ. Drawing on the available evidence collected primarily by KIIs with judges of all levels (24), ABA/ROLI staff, and document review, the ET found that the KJP has largely met the Objective 1 and its activities on increasing the quality of judicial decisions and improved application of the newly introduced legislation have achieved positive results. KJP’s assistance on improving judicial capacity has achieved positive results. KJP’s activities in the area of improving judicial capacity are numerous, but the bulk of the TA consists of: 1) developing manuals and handbooks on writing judicial decisions for regional and lower court judges; 2) developing course curricula and conducting training through in-person and distance learning training formats; and 3) conducting training-of-trainers (TOT) programs to prepare sitting judges to deliver peer￾to-peer trainings.2 The ET found that, beginning in FY 2013, KJP has dedicated considerable resources to developing decision writing manuals to help judges improve the quality of judicial decisions. To develop these manuals, KJP generally followed the following preparatory steps: 1) conducted needs assessments through consultations with regional courts; 2) coordinated with the SC on the topic of the manual; 3) commissioned an expert to develop the manual and incorporated feedback from the SC justices; 4) submitted the manual for the 2 From FY 2013 until FY 2016, progress indicators regarding improving the quality of written decisions included: number of judges trained, number of courses developed, number of TOT trainees qualified to conduct peer-to-peer trainings, number of materials on improving judicial decisions quality, and increased knowledge among among judges about judicial decision writing. 9 approval at the scientific board of the AOJ; and 5) distributed 500 copies of each manual to regional courts and made electronic copies available through the websites of ABA/ROLI and the SC. After the July 2016 extension, the expected results included development of writing manuals, development of course curricula and other resource materials for the AOJ, and organizing training and seminars on civil law, criminal law, CVE, and investment disputes. The KJP was also required to measure the increased capacity to write judicial decisions by assessing the percentage of decisions from all courts (not just in regions with training) for consistency with decision writing methodology that the KJP developed with input from senior SC and retired judges. Over the years, KJP completed and exceed the targets related to the number of judges trained and number of materials distributed on improving judicial decisions quality. Table 6: KJP Decision Writing Manuals # Title3 Year Author Copies4 1 Methodology of composing the judgment in a civil case 2013 Schnitger & Kuanova 1,161 2 Review of civil cases upon newly discovered circumstances 2014 Mamontov 500 3 Methodology of judicial decision writing in labor disputes 1 2014 Kuanova 500 4 Protection of personal non-property goods and rights 2015 Mamontov 500 5 Methodology of judicial acts in disputes on maintenance obligations 2016 Kuanova 500 6 Inheritance disputes: methodology of judicial decision writing 2017 IKuanova 500 7 Methodology of judicial decision writing in labor disputes 2 2019 Kuanova 500 Total 4,161 In total, KJP developed seven decision writing manuals and distributed over 4,100 copies (usually 500 copies for each manual) for judges of regional court (oblast) and lower level courts. KJP exceeded its targets for training judges by training over 700 judges (instead of the targeted 400) in judicial decision writing, in-person and online. It also surpassed targets for the TOT program by training 16 instead of 10 trainers for peer-to-peer training. Based on the qualitative evidence collected primarily through KIIs and FGDs with judges, KJP’s TA contributed to higher-quality written decisions and improved application of new legislation by judges. A large majority of judges (15 out of 24) considered the judicial decisions manuals particularly useful to junior judges. For example, all judge respondents in Almaty (5) and Nursultan (5) thought that the KJP manuals on decision writing are useful particularly for new judges and lower-level judges. Four (4) judges pointed to the manual on labor disputes compiled in 2014 and revised in 2019 due to legislative changes, as an example of a widely acclaimed publication supported by KJP. The ET also heard from a number of respondents (6) that, while the manuals have been distributed and are available in the regional courts’ training centers, they are considered as “recommended readings” so it is difficult to gauge if and how often they are consulted by sitting judges in their daily work. A few judges (4 out of 24) stated that despite notable improvements in the quality of manuals over the years, some of them could have been less theoretical and could be more useful if they included more case-based and practice-oriented material. The same respondents also suggested that it is necessary to continue refining their content and expanding their use among junior judges in the regions. Although judges were positive about KJP’s support to achieve a higher quality of written decisions, the ET did not have the time and resources to explore the degree to which judges have acquired new knowledge and are applying it in their daily jobs. Determining the effectiveness of KJP’s manuals and pilot training programs in improving the judicial quality to write better decisions has also been a challenge for the 3 Original titles may be longer and have been shortened due to space limitations. 4 The number of copies is approximate because KJP upon request of the AOJ did print a small number of copies of some manuals after FY 2016. 10 project. Beginning in FY 2013, KJP utilized a user survey to determine the usefulness of the manuals to judges, reporting to USAID a high level of self-declared satisfaction and application of knowledge by judges. Starting in FY 2016, KJP also introduced a baseline study and conducted a follow-up comparative analysis in 2019 to track the increased knowledge on judicial decision writing. Because improving judicial capacity is an open-ended process, measuring KJP’s effect is complicated by a number of factors. First, improving the quality of written decisions is an ongoing initiative directed by the SC, which has codified rules for writing judicial decisions by adopting normative regulations on how to draft judicial decisions and amending them periodically.5 For example, the “Seven Pillars of Justice” package of reforms announced in 2018 identifies the improved quality of judicial decisions as one of the key priorities of judicial development. Higher quality of written decisions is part of the Project 6: “Perfect Result,” Subproject “Exemplary/Ideal Judicial Decision” and progress in this area has been considered an important but ongoing process since the early 2000s. The ET learned from KIIs with the SC justices (11 out of 13) that improvement of the quality of decisions is a multi-annual and multi-stakeholder process coordinated by the SC; KJP is one of the main international projects contributing in this regard. Four (4) SC justices considered KJP’s contribution through manuals and associated training on judicial decisions as very important in reinforcing the results of the SC’s work and in increasing the overall quality of written decisions in recent years. Second, as a long-term process, improving judicial capacity has considerable “lag effects” accumulated over time, which can hardly be captured with a snapshot approach by assessing the effectiveness of one or more manuals. For example, the ET noticed that a majority of judges were familiar with only a few of the manuals and could not reliably comment on how much they are used by judges despite their overall opinion that resource materials are always helpful. On the other hand, five (5) SC justices pointed out that some elements of the ABA/ROLI manuals (i.e., manual of civil cases, labor disputes, etc.) have been incorporated and codified in normative regulations of the SC (last amended in 2018) and their cumulative effects will outlast the KJP’s project life, which points to the sustainability of the KJP’s contribution in this area. Third, KJP initially lacked baseline data through which to measure the impact of its activities and relied on pre and post self-evaluation forms of training participants to gauge the level of satisfaction with trainings and determine the percentage of judges who declared they have increased their knowledge through participation in training programs. Later, starting in 2016, KJP introduced qualitative elements to analyze the increased knowledge among judges about judicial decision writing by undertaking a baseline analysis and then a comparative study in 2019. The ET found the introduction of qualitative elements as a commendable effort on the part of KJP but also identified several limitations, which question the validity of research procedures and reliability of its results (see EQ 2). Another important outcome under Objective 1 was to improve the application of new legislation by judges which KJP pursued through: 1) organizing judicial exchange on international best practices; 2) conducting videoconferences after the judicial exchange with judges in the regional and city courts; 3) developing and conducting seminars for judges on application of new legislation through in-person and distance-learning formats; and 4) organizing large national and international conferences on newly introduced laws and new legal concepts. The ET found that KJP’s TA has facilitated the transition to new legal codes and improved the application of new legislation by judges. Overall, KJP has organized six judicial exchange programs (3 in the United States, 2 in Estonia, and 1 Germany) with 45 participants (judges of various levels and justice system personnel) on criminal infraction, plea bargaining and investigating judges, CVE-related cases, investment 5 Normative Regulation of SC No. 3 of March 16, 2018. “On introducing amendments and additions to the Normative Regulation No. 5 of the Supreme Court of the Republic on the Court Decision of Kazakhstan as of July 11, 2003.” 11 disputes, the adversarial features of criminal justice system, and, more recently, on intellectual property rights. 6 Table 7: KJP Judicial Exchange Programs # Judicial Exchange/Study Tour Topic Time Country Participants 1 Defining and charging “criminal infractions” 2014 Germany 6 2 Applying plea bargaining and the role of investigating judge 2014 Estonia 9 3 United States (U.S.) experience in investment disputes 2017 USA 8 4 U.S. experience in criminal justice, including CVE and cybercrimes 2018 USA 8 5 Estonian experience with adversarial procedure 2019 Estonia 3 6 U.S. experience on intellectual property rights 2019 USA 10 Eight (8) judges praised KJP’s contribution in transitioning to the new legal codes and, in particular, the assistance with regard to introduction of adversarial elements in the criminal justice system, investigating judges, criminal infractions, and CVE. The ET also learned that the SC has used the lessons learned from judicial exchange programs, conferences, and TA of the KJP engaged experts to adopt normative regulations on adjudicating violent extremism cases. A number of judges (3) noted that peer-to-peer professional exchanges have also been very useful in informing the legal reform process, reviewing the implementation of law, and proposing amendments in areas where practice suggested improvements. While not all the proposals of the KJP may have been taken into account, the ideas and concepts explored during the international exchange programs have been used by the SC in proposing legal revisions and amendments to the Parliament. Following the change of objectives in FY 2016, KJP’s activities focused on the improved capacity of the judiciary to apply the newly introduced legislation and better adjudicate cases in civil law, criminal law, and investment disputes. Given the amount of resources and expertise dedicated to the judicial exchange programs and sharing of international best practices, the ET probed the judicial respondents on the usefulness and impact of such activities over the years. Judges (7) who participated in study tours events were very satisfied with the content and format of the judicial exchange visits and quality of experts involved in KJP-supported events. The SC justices (4) who commented on the work of KJP in improving the capacity to adjudicate investment disputes pointed out the positive contribution of KJP through co-organizing international conferences on investment justice, exposing judges to international practices on investment disputes, and providing international experts on investment disputes. Although work on improving the investment climate is ongoing, KJP has contributed to formal recommendations made about definition of “investment disputes,” delineating the jurisdiction of specialized panel of the Nursultan city court and specialized panel of the SC. 7 A judge who participated in various investment dispute activities of the KJP noted: “I think ABA/ROLI has helped a lot on defining the jurisdiction of the specialized panel, clarifying concepts based on the American and international experience and bringing together institutional actors in various events on investment disputes which have enhanced cross-institutional understanding and dialogue on a new such as investment justice. They have generally been very attentive to our needs and their events are well organized.” Participants of the CVE study singled out the lessons learned from the U.S. experience in dealing with terrorism cases. All judges (4 out of 4) who chose to comment on the KJP’s work in this regard mentioned that exposure to international expertise, particularly American expertise, has been very useful in advancing the thinking of judges on issues related to admissibility of evidence from conflict zones, sentencing and 6 The intellectual property rights program took place in November 2019 and is outside the scope of this evaluation. 7 Recommendations of the international conference “Mechanisms of Settlement of Investment and Commercial Disputes in Kazakhstan and OECD Countries” organized October 11, 2018 in Astana. 12 punishment issues, international practices in revoking citizenship of terrorism, and other topics related to CVE which are becoming increasingly relevant to the Kazakhstan’s criminal justice system. Although the number of cases on violent extremism in the Kazakhstan judicial system is low (currently less than 20), Kazakhstan is widely considered a global leader in repatriation efforts of individuals who have joined the Islamic State of Iraq and Syria (ISIS) and other terrorist organizations in Syria and Iraq so criminal prosecutions and rehabilitation programs are expected to increase. Judges (3) of the criminal panel of the SC confirmed that various suggestions and recommendations provided by several ABA/ROLI experts on the issue of CVE have been taken into account by the SC when drafting the normative regulation on terrorism related cases.8 One of the judges who took part in the judicial exchange program in the United States noted that: “The CVE study tour was a great opportunity to learn from U.S. counter-terrorism institutions on evidence gathering, judicial orders limiting human rights of suspected violent extremists and the type of rehabilitation programs available. One of the follow-ups from this tour was to recommend expansion of investigating judges to review covert law enforcement measures infringing upon the human rights of suspected terrorists. These qualitative statements are also supported by data from the KJP surveys, which indicate very high self-reported levels of satisfaction (between 93 percent and 100 percent) of participants of judicial exchange visits. Additionally, 96 percent of study tour participants indicated that as a consequence of the study tour their knowledge increased.9 A few judges (3) mentioned that the exchange visits are very intense and require participants to digest a large amount of information in a short time. They suggested advance planning could be improved through more thorough “know before you go sessions,” which enable judges to learn more about the legal system and institutions they are visiting in order to prepare better to ask questions that are pertinent to the Kazakhstan’s judicial practice. On the question about general quality, timeliness, and responsiveness of the KJP team, the ET observed a general consensus among judge respondents (18 out of 24) that ABA/ROLI has a strong reputation for delivering timely and high-quality assistance through a dedicated, responsive, and professional team which has diligently accomplished project objectives. Judges of all levels who commented on the overall reputation of the project (18 out of 24), emphasized that the KJP team is very responsive, professionally competent, and diligent in providing the expected technical deliverables. In particular, justices of the SC (10 out of 14) viewed the KJP as one of the most trusted partners of the SC. A large number of judges (16 out of 24) stated that one of the main strengths of the KJP has been the readiness and ability to engage high quality international and local experts on civil and criminal law, investment disputes, violent extremism, ADR, and media and court relations. This was also corroborated by non-judge respondents (6) who commented on the importance of the KJP for exposing judicial leaders to international best practices by either bringing short-term international advisors to Kazakhstan or by organizing judicial exchange programs abroad. A number of international partners (5 out of 11) also stated that through study trips and large international conferences organized in cooperation with the SC, the KJP have exposed the judicial system to high-level international expertise from U.S. and European counterparts and must be continued in the future. 4.1.2 Objective 2 Despite weak public trust in the judicial system, KJP has been successful in achieving Objective 2 and there is evidence to support KJP’s positive contribution to the gradual improvement of the efficiency and accessibility of courts. From FY 2013-2016, KJP pursued this objective through activities aimed at increasing the use of ADR mechanisms, developing ADR resource materials, and increasing public awareness on modernization of the judicial system. For FY 2016-2019, activities related to ADR were 8 Normative Regulation No. 11 of the Supreme Court of Kazakhstan “On Some Issues of Judicial Practice on Application of Laws on Terrorist and Extremist Crimes” adopted on December 8, 2017. 9 See Annual PMP Report covering the period October 1, 2018 through September 30, 2019. 13 reported under Objective 1 on improving judicial capacity in civil cases. However, for the purpose of this evaluation, the ET assessed activities related to mediation, arbitration, court openness, and transparency taking into account modification of the objectives over the life of the project. KJP’s contribution to ADR’s expansion in court is recognized by partners The ET found that KJP and USAID considered the expansion of mediation and arbitration as a way of reducing judicial caseloads and enhancing the overall efficiency of the court system. KJP started working on mediation and arbitration in FY 2013, designing several activities to raise the professional competence of judges and mediators and boost public awareness on the availability and advantages of the ADR mechanisms. Starting in 2015, the Civil Procedure Code introduced the concept of a “judge-mediator” as the responsible judge to direct parties toward in-court mediation for the settlement of disputes. During FY 2016-2019, KJP offered seminar/training sessions for about 70 judges and 60 Master’s students on the applicable law on mediation in Kazakhstan, in court and off-court mediation procedures, mediation techniques and skills, procedural acts, and agreement, including case study and role play exercises. The percentage of surveyed judges who reported that their skills in mediation have improved is 94.3 percent. More recently, KJP developed a course curriculum on interaction between state courts and arbitration proceedings for students in the Master’s Program and other educational resources (handbook, guidelines, manuals) for sitting judges in all regional courts. Based on the annual and quarterly reports, all targets have been met and exceeded by the KJP. KIIs with judges, mediators, and KJP staff (16) commented that, although a separate law on mediation was adopted in 2011, the use of ADR in court procedures took time to expand. The SC statistics made available to the ET suggest that the use of ADR got new momentum after the SC and UNDP implemented a pilot project with regional courts called “Introduction of Mediation Institute in Kazakhstan” to raise professional competence and public awareness on ADR. 10 For example, the SC reports that in 2013 only 1,276 cases were resolved through mediation. After the implementation of the pilot program in 2014, the SC reported that 9,029 cases or 3.72 percent of the cases were resolved through mediation, of which 3,939 cases have been settled by means of in-court mediation. KIIs with mediators and judges revealed various aspects of the added value of the assistance of the KJP. Nine (9) respondents who commented on the KJP’s activities on ADR stated that KJP has contributed to the expansion of ADR in three main ways: 1) providing training material and educational resources for mediators and arbitrators and creating a cadre of professionals trained to use ADR; 2) raising public awareness on the advantages of mediation and arbitration; and 3) providing international expertise from the United States and EU countries on the recent trends and development on the ADR. First, five (5) KIIs stated that, although it was not a catalyst for the introduction of ADR, KJP’s work over the years has helped to professionalize the supply and increase demand for ADR. After the changes to the law on mediation in 2017, KJP was important in keeping the momentum for the increased use of mediation by developing resource materials for training mediators and judges. Second, KJP has also served as platform for dialogue and cooperation between various ADR stakeholders—including the SC and regional courts, representatives of state bodies, investors, arbitration courts, and international law firms—to foster dialogue among judges and arbitrators and develop recommendations on how to further improve practice of interaction between courts and arbitration in resolving commercial disputes. This has been done by providing them with comparative studies and in-depth assessments of the Kazakhstan system, which have been important to increase the understanding of ADR mechanisms and readjust the goals and aims of ADR. Third, KJP has engaged U.S., European, and Asian experts to offer local stakeholders’ various perspectives on the use of the ADR and help them identify and contextualize ADR best practices. KJP has also conducted a roundtable on interaction between courts and arbitration in resolving commercial 10 The project organized a total 285 training events, from them 111 seminar lessons, 156 round tables, nine trainings, seven lectures, and two conferences. 14 disputes for 89 participants, Asked about the impact of KJP’s activities in ADR, KIIs who chose to comment on the topic (7) stated that it is hard to discern the effect of KJP because most of its activities have been co-organized or implemented under the direction of the SC. In the last four years, mediation and arbitration are progressively being utilized to settle cases in Kazakhstan both before case filing and after. In-court and off￾court mediation have marked a notable increase in the cases of debt collection, utility payments, recovery of alimony for minor children, divorce, and other family disputes. The ET learned from SC justices that in 2018, while over 94 percent of the disputes were resolved by the court, over 50,000 civil and criminal cases (5.5 percent) were resolved through mediation.11 Enhancing the availability of the in-court and off￾court mediation is one of the priorities of the current Chief Justice. The SC is also cooperating with mayors’ offices (Akimats)12 to offer free of charge premises in municipalities to support off-court mediation activities. The ET also learned that several questions regarding the strategic goals of mediation still remain unresolved. While initially mediation was introduced to relieve the courts of cases, the introduction of the judge-mediators has increased public demand for in-court mediation thereby creating additional work for judges. Private mediators consider this as unfair competition to the off-court mediation because in￾court mediation fees are considerably lower than private mediation. However, KJP and other international partners largely viewed the expansion of in-court mediation as a welcome development pointing to considerably lower resolution times for cases resolved through mediation and the contribution of amicable settlement for lowering social conflict. Judges thought that, aside from the lower fees, this is because citizens consider judges to be more effective mediators and expect them to be more formal and professional in conducting their duties. Notwithstanding the progress in the expansion of ADR, the low level of referrals to voluntary mediation before and after case filing remains a challenge. However, this depends on the clarity of strategic vision and the need for comprehensive and coordinated approach within the GOK—and goes beyond the mandate of KJP. ADR policy is a shared responsibility of all three branches—the executive, legislative, and judicial—and more work is needed to articulate a longer-term vision on mediation. Ideally, the GOK should collect feedback from mediation centers, legal institutes, non-governmental organizations (NGOs), bar associations, and other partner organizations on the incentives and smart punishments needed to further increase the use of mediation and make it the default mechanism vis-à-vis contentious court proceedings. Court transparency is increasing also due to KJP’s work with press secretaries and judge￾coordinators. The other important aspect of KJP’s work under this component has been to increase the public accessibility of the court system. Through KIIs with KJP staff and review of project documents, the ET learned that the work of KJP in this regard has consisted of enhancing the use of mediation and arbitration, raising public awareness on new legislation, and, more recently, improving the capacity of courts to handle public relations, enhancing court transparency and outreach efforts. During FY 2016-2019, KJP revised its objectives and focused on increasing public awareness of judicial modernization reforms and court operation. A separate Objective 4 on “Improved public awareness of judicial modernization and court operation” focused on working with judicial personnel, primarily the newly created position of Judge￾Coordinator responsible for dealing with mass-media and press secretaries of regional and city courts. The ET observed that a large majority of respondents was not aware of KJP’s objectives in enhancing court efficiency and accessibility and could provide little information on the degree to which KJP has 11 Statistics provided by the Supreme Court during the field work in December 2019. 12 Akimats are local government units. 15 accomplished its tasks for FY 2012-2016. In contrast, the ET was able to collect more useful information on the work of KJP in improving court openness and transparency for the period covering FY 2016-2019. The vast majority of respondents attributed the positive trend of court transparency and openness to the SC efforts and could not provide details about the contribution of KJP. Court openness and transparency activities have been part of a larger project initiated by the SC in 2014 and expanded under the new Chief Justice. For example, in 2018, the SC developed a new Communication Strategy of the Judiciary aiming to establish effective communication between the public and courts. Furthermore, in the last three to four years, new front offices are being set up in various courts and court sessions and judicial decisions are becoming more accessible. The SC has asked courts to improve their capacity to handle public relations including explaining their judicial decision in user-friendly language to be understood by lay persons and journalists. However, journalists, judge coordinators, and press officers who chose to comment (5) on this topic, while attributing the positive trend to court openness mainly to the SC leadership, also emphasized the role of KJP in two directions: 1) as a neutral convener and provider of a platform for constructive dialogue among various stakeholders from judiciary, media, and international projects; and 2) as a provider of high quality U.S. and European expertise in the court and media relations and training opportunities for press secretaries. KJP is not the only actor in this field as the Organization for Security and Cooperation in Europe (OSCE), InterNews Central Asia, and Friedrich Erbert Foundation have also been working in enhancing court transparency and increasing public trust in the court system. The close connection between the KJP and SC and the contribution of other actors in this area make it difficult to specify the degree of the KJP’s effect too. However, nine (9) KIIs from a variety of sources stated that the KJP is contributing to the positive trend in the evolution of court relations with media and the public. Press secretaries and judge coordinators, owing also to the KJP’s contribution, are better trained to handle the relations with media and the public better. A senior judge encapsulated this in the following statement: “ABA/ROLI provides a multiplicity of ingredients in the activities of the SC aimed at improving court capacity to handle public relations but it is very hard to assess the impact of each actor in activities where co￾organization and co-creation are crucial.” Two (2) judge coordinators and two (2) press secretaries praised ABA/ROLI for engaging U.S. and European (a Romanian judge) experts to offer their own expertise as particularly useful to understand where the Kazakhstan courts stand on issues such as transparency of the judiciary, publicity of court hearings and commentaries of judicial acts through social media, defamation, work of courts’ public information offices, and coverage of courts’ activities by journalists and bloggers. Among the contributions of the KJP in improving court and public relations during 2018 and 2019, KIIs mentioned the following as most important: • Algorithm on responding to high profile cases; • Rules of procedures for judge coordinators defining the roles and responsibilities as the key interlocutors between the courts and media; • Draft normative regulation on libel and defamation; • Expertise on international best practice sharing through U.S. and European judges how to communicate with media and the public; • Training for press secretaries and journalists; and • Development of a manual on how to organize and manage public relations campaign. 4.1.3 Remaining programmatic gaps While only a few informants had useful information on the extent of KJP’s accomplishments, almost all informants provided some information on the remaining gaps. Most informants did not simply refer to the 16 KJP’s work, but conceptualized gaps as existing challenges of the judicial system. Others identified areas where more work is needed by KJP or USAID’s future judicial programs. Based on the available evidence, the ET identified two main gaps: 1) gaps on fundamental issues of judicial independence and accountability left unattended as a result of the KJP’s modified objectives; and 2) gaps related to the capacity of the AOJ to offer high quality judicial education. Gaps related to the original goal of the KJP to strengthen judicial independence The ET found that although since 2013 KJP excluded issues of judicial independence, appointment, or improved court administration from its scope and focused its TA and resources on the appropriate areas that had a reasonable chance of success, the GOK and justice institutions have undertaken various initiatives to improve public trust in judiciary (see EQ 3). The ET noted that the overwhelming opinion of the informants is that many challenges remain in strengthening judicial independence, accountability, judicial integrity, as well as reducing judicial corruption due to the political will and not the KJP’s performance. Referring to the work of KJP, a senior legal professional observed: “I would say that KJP is doing a good job in the areas they are allowed to work. The question that we need to ask remains ‘Are the areas that KJP is working the most pressing ones in the eyes of the general public?’ The answer is clearly not and this is partly the reason for the low public trust in the judicial system.” A large number of judicial and non-judicial respondents (19) stated that public trust is low because reforms have not addressed the fundamental issues but have focused instead on providing TA in the areas agreed upon by the GOK and SC. This is not unique to ABA/ROLI but applies to the UNDP, EU, WB, OSCE, and other foreign assistance projects but was also expressed clearly in the words of the Chief of Party (COP) of the KJP: “I consider KJP as a purely technical project and our job is to provide highest quality of expertise and advice based on the identified needs of the Supreme Court and approval by USAID.” Nine (9) informants mentioned that hierarchies of power are well established in the judicial system and a lot of control is exerted by the chairmen of regional courts who are generally less interested in opening up the system. The few judges who were more forthcoming regarding the strengths and weakness of the judicial system mentioned that important gaps requiring further assistance include: • Strengthening the judicial integrity to insulate the judiciary from undue pressures from other branches of government; • Developing merit-based judicial selection, evaluation, and promotion procedures that increase judges’ integrity and impartiality; • Enhancing judicial competence and the professional ethics of judges; and • Tackling judicial corruption. Gaps regarding the AOJ One of the important aspects of KJP’s work according to the modified program description in 2013 under Objective 1 was to “Increase the ability of the Institute of Justice to offer courses to judges on relevant legislation” initially reported under a separate Outcome 3. KJP was expected to strengthen the Institute’s capacity by developing course curricula for its training programs; improving its educational capacity and equipping faculty members with the most up-to-date continuing legal education (CLE) teaching methods for sitting judges; developing and conducting TOT programs to qualify judges for peer-to-peer professional training; and providing institutional development and capacity building at the time of transformation of the Institute into the AOJ after 2016. Under the Performance Management Plan (PMP) of Years 2 and 3, KJP reported on the following indicators regarding Outcome 3 as follows: 1) number of materials developed and provided to executive branch personnel with USG assistance; and 2) number of materials developed 17 and provided to executive branch personnel with USG assistance. Baselines did not exist for both indicators and ABA/ROLI did not specify targets for any of the indicators in work plans for Years 2 and 3. Through KIIs with justice system stakeholders, KJP, and USAID as well as document review, the ET learned that when KJP was designed in 2011, what is now the AOJ was a relatively new institution having been created in 2001. Since then, the AOJ has experienced several transitions from the SC to the Presidential Administration, which have been unconducive to achieving institutional stability and long-term institutional development. In the framework of “100 Concrete Steps” announced in 2015, Step 18 separated the AOJ from the Academy of Public Administration transferring under the SC and the last transition was completed on February 18, 2016.13 The AOJ now has three constituent departments: a CLE institute that provides regular professional development for currently serving judges; a Masters’ Program for those who want to become judges in the future; and a scientific-research department. It trains over 600 judges and judicial personnel annually and its first class of Master’s students admitted in 2016 graduated in 2018. In Year 5, as the AOJ was being transferred to the SC, KJP engaged an international expert who consulted with the AOJ’s management and submitted written recommendations for institutional development in the areas of scientific research, practical skills development, CLE for judges, and initial training for Master’s students. According to KJP’s internal documents, the SC considered the KJP’s recommendations in the Coordination Council and incorporated many of them in a new “Strategy for Judicial Education of the Republic of Kazakhstan for 2017-2020.”14 In Year 5, KJP set targets for its assistance to the AOJ and completed them, although the ET found the targets to be more adequate in measuring the KJP’s outputs and less useful in capturing the results of AOJ’s capacity building efforts. After changing the Objectives in 2016, KJP continued to work with the AOJ to develop course curricula and course materials on decision writing manuals, mediation, investment disputes, and CVE; however, these activities fell under other objectives aimed at improving the capacity of judges and were not dedicated to improving the capacity of the AOJ. Overall, KJP has developed eight course curricula for the AOJ on decision writing in civil cases, mediation, criminal law, violent extremism, investment disputes, and arbitration, and has trained more than 10 faculty members on interactive teaching methodologies, including TOT and distance learning. During field work, the ET learned that KJP has provided the AOJ with TA in curricula development and institutional development according to the approved work plans. However, its effectiveness and sustainability has been limited due to institutional weaknesses (e.g., lack of strong leadership, strategic vision, and effective management practices), which plagued the AOJ from 2012-2018. KIIs with 11 judges and seven (7) representatives of international organizations revealed a general agreement that the AOJ’s capacity to provide judicial education remains weak. Judicial education is decentralized in Kazakhstan with multiple players covering various aspect of judges’ education. For example, while the AOJ is under the authority of the SC, some of its programs are approved by the Ministry of Education. Furthermore, while the AOJ manages a retraining program for sitting judges and a specialized formative course for Master’s students, regional courts have their own training centers offering in-person and distance learning course formats which are not managed centrally by the AOJ. In addition, the SC also organizes training programs—either through the AOJ or the training centers—and approves the Annual Training Plan for the retraining program for sitting judges (prepared by the AOJ). Last but not 13 In 2001, the Institute for Professional Development of Judges and Judiciary Employees was established under the SC. In 2003, the Institute was renamed as the “Judicial Academy under the Supreme Court of the Republic of Kazakhstan.” A presidential decree No. 1583 "On Measures for Further Improvement of Training, Retraining and Professional Development of Personnel of State Bodies of the Republic of Kazakhstan,” dated May 31, 2005, ordered the reorganization of the judicial academy and renamed it into Institute of Justice of the Academy of Public Administration under the President of the Republic of Kazakhstan. A new presidential decree of February 18, 2016, transformed the Institute into the AOJ and placed it again under the authority of the SC. 14 The full text of the Strategy is available in Russian at http://sud.gov.kz/rus/con tent/strategiya-sudebnogo-obrazovaniya￾respublikikazahstan-na-2017-2020-gody. 18 least, the UOJ delivers mandatory and optional training programs on judicial ethics and rules of professional conduct. Two (2) international representatives stated that the AOJ is considered a weak institution and the quality of judicial education is not given adequate importance by judicial leadership. Specialized judicial education is not a requirement for accessing judicial office nor is CLE considered seriously as part of judicial performance evaluation. Five (5) managers of international projects, three (3) judges, and two (2) practicing lawyers converged around the idea that the AOJ has generally been slow to embrace interactive teaching techniques and case-based instructional methodologies, both for sitting judges and Master’s students. The same project managers expressed their frustration with the continued institutional weaknesses of the AOJ pointing to a recently implemented UNDP project, “Training and Infrastructure Needs Assessment of the AOJ,” which was less successful than expected. 15 Discussions with the AOJ’s new leadership demonstrated that the course curricula and course materials developed by the KJP are not part of the Annual Training Program for the CLE program. The Annual Plan is largely developed on the input provided by regional courts based on a questionnaire that the AOJ sends to solicit information on judges’ training needs. Following analysis of the responses, the AOJ compiles a training plan, usually with 12 topics, which is submitted for SC approval. The AOJ’s staff was generally aware of the course materials provided by KJP and confirmed that copies of manuals and training materials are kept in the library of the AOJ but they are not used for training purposes during the year. It was also not clear that the KJP-developed manual “How to Train the Judge” is used by the AOJ trainers/professors. The ET noticed that senior management of the AOJ considered the course curricula, TOT programs, and course materials developed by KJP as stand-alone initiatives not integrated into annual plans. This was clearly expressed in the statements of the AOJ senior managers: “ABA/ROLI has offered one- or two-day course with international and local guest lectures but we do not have a Memorandum of Understanding or long-term plans for curricula development or training offerings with ABA/ROLI.” Representatives of the AOJ compared the assistance of KJP with other organizations and expressed their interest to have more consultation and coordination with ABA/ROLI in the future. They considered the Memoranda of Understanding (MOUs) with other organizations as valuable instruments to agree on a framework of cooperation: “We have more long-term cooperation plans with Friedrich Erbert Stiftung, GIZ about training programs for sitting judges and we have [Memoranda of Understanding] (MOUs) for long term cooperation with both organizations. In contrast, we have no plans with ABA/ROLI even for 2020 and we would like to meet soon to discuss our cooperation for next year.” CONCLUSIONS Accomplishments • KJP has been largely successful in achieving Objective 1 on higher quality of judicial decisions and improved judicial capacity to apply new laws. • KJP’s contribution to expansions of ADR mechanisms (mediation and arbitration) are showing positive results. KJP’s role as a synergy provider and enabler for dialogue between the courts and media has contributed to enhancing court openness and transparency under Objective 2. • Under both objectives, the predominantly on-demand nature of the assistance and some of the performance indicators make it difficult to reliably attribute the effects to KJP, but partners confirm the positive contribution of KJP’s deliverables. In particular, ABA/ROLI has a strong 15 The UNDP project description is available in English at https://info.undp.org/docs/pdc/Documents/KAZ/AWP_2018_Training%20Needs_Assessment.pdf. 19 reputation for delivering timely and high-quality assistance to the local beneficiaries based on agreed upon needs. Remaining gaps • The remaining gaps in addressing the original program objectives (i.e., improving judicial selection, judicial independence, impartiality, evaluation of judges, etc.) exist but this is not because of the KJP’s lack of technical capacity. KJP’s ability to address the key challenges of the judicial system, i.e., independence and accountability, was limited by the lack of willingness on the part of the SC at the onset of the project. • KJP’s assistance to improve the capacity of the AOJ to provide higher-quality judicial education has been less successful despite the delivery of technical outputs by KJP. The effectiveness and sustainability of KJP’s interventions are limited due to AOJ’s weak capacity and frequent leadership changes. RECOMMENDATIONS • In accordance with the national strategies and action plans of the GOK, USAID should continue to work on strengthening judicial independence and accountability procedures and systems with the aim of increasing public trust in the judiciary. • As the AOJ is far from becoming a solid judicial education institution, USAID should conduct further analysis on the current needs of the AOJ and provide additional support to the AOJ taking into account the SC and AOJ plan for institutional strengthening (see EQ 3). 4.2 EQ2: WHAT ISSUES EXIST IN THE MONITORING SYSTEM ESTABLISHED BY ABA/ROLI WHICH MIGHT HAVE AFFECTED ITS EFFECTIVENESS IN REPORTING ON KJP PROGRESS? FINDINGS ABA/ROLI originally submitted a draft PMEP together with its Year 1 Annual Work Plan on May 31, 2012, which was approved after three revisions in September 2012.16 The PMEP developed definitions and measures to report progress towards its original goal of the project to increase independence and accountability of Kazakhstan’s judiciary. One of the important indicators, namely the number of laws, regulations, and procedures related to judicial independence supported with USG assistance, became irrelevant shortly after USAID’S approval of the PMEP due to resistance of the SC to engage with ABA/ROLI on the topics of judicial selection, appointment, and disciplinary measures. ABA/ROLI submitted a new Activity Monitoring and Evaluation Plan (AMEP) together with the work plan for Year 3, which became effective on November 1, 2013. Following the No-Cost Extension (NCE) for a three-month period, ABA/ROLI applied for an FCE in June 2016 which USAID approved in July 2016. Upon receiving a 38-month FCE, ABA/ROLI revised the project objectives in the technical proposal based on which USAID granted the extension and, upon receiving USAID approval, started reporting on four newly designed objectives. The Year 6 Work Plan was accompanied by a new Project Results Framework and PMEP reflecting the revised goal of supporting the GOK’s efforts in judicial modernization to respond to the international obligations of the state.17 The new PMEP combined some of the old progress indicators with new indicators measuring the new objectives aimed at improved capacity of the judiciary to adjudicate civil cases, criminal cases, and investment disputes. ABA/ROLI used the PMEP adopted in 2016 to report its progress until September 2019 when USAID requested it to return to the original program objectives (last approved by USAID on November 1, 2013). ABA/ROLI has now revised the PMEP for the 12-month 16 See Annual Work Plan Year One: April 16, 2012-September 30, 2012. 17 See the KJP’s Year Six Work Plan: October 1, 2016-September 30, 2017, resubmitted October 11, 2016, and the attached PMEP. 20 period accordingly. The ET identified the following strengths and weaknesses of the M&E system on the basis of the available documentary evidence and relevant KIIs. 4.2.1 Strengths ABA/ROLI has continuously sought to improve and refine its M&E system. A review of the project documents indicates that ABA/ROLI has made diligent efforts to improve its M&E system. Although, the ET did not have sufficient time to comprehensively examine the internal protocols and methodologies of data collection for each and every indicator and the means of objectively verifying progress, ABA/ROLI appears to collect, process, and store its M&E data comprehensively. Another important strength is that over the project lifespan ABA/ROLI sought to make their M&E system more elaborate, dedicating more time and human resources to better capture and report progress. Program indicators have largely prioritized the quantitative aspects of technical deliverables; however, since 2016, ABA/ROLI has tried to introduce qualitative elements to enable more responsive data collection that better capture the effects of its interventions aimed at improving the quality of written decisions by judges. At times, the ability of the project to establish baseline indicators and set targets towards which to measure the progress has also been adversely affected by the need to modify the project objectives. For example, the cancellation of the Judicial Independence Baseline Assessment in 2013 affected ABA/ROLI’s ability to generate baseline data for the progress indicators for other project components not strictly related to the judicial independence and accountability. This delayed the efforts to generate baseline data for a key indicator No. 1.2.4 “Increased knowledge among judges about judicial decision writing,” designed to measure the effectiveness of ABA/ROLI’s work on improving the quality of written decisions. Initial baseline analysis took place towards the end of FY 2016 which hampered the ability of the KJP to conduct follow-up research and analysis within the original planned end date of the KJP. The in-country and home office team have gradually dedicated more time and human resources to comprehensively measure and report progress. On the other hand, ABA/ROLI’s approach of using surveys before and after its training programs, seminars, or judicial exchange programs as well as conducting follow up evaluations of the usefulness of its activities demonstrates the willingness of the country-based project team to diligently collect project implementation data, and track and report progress to USAID. Survey questions solicited information on how judges are applying the lessons from the judicial writing manuals manual, and if and how the manual improved their decision writing skills. Although, as with other methods of data collection, self-reported levels of knowledge retention and application have their own limitations, the fact that ABA/ROLI systematically used participants’ surveys to improve subsequent editions based on the feedback collection and lessons learned over the years is very positive. In discussions with USAID and the ABA/ROLI team, it became clear that ABA/ROLI’s home office, based in Washington, DC, has provided important hands-on support to incrementally refine the conceptualization and definition of progress indicators and establishment of baselines and targets as well as to enhance data collection methods and their collection frequency. ABA/ROLI’s home office conducted internal assessments of the KJP’s M&E system during FY 2019. Although the KJP is a relatively small project, following an internal assessment by the ABA Evaluation Unit in DC (conducted in August 2019), KJP designated a M&E officer in October 2019. The M&E Manager will be leading the M&E efforts and conducting tasks which have previously been collectively discharged by the COP, staff attorneys, or program managers. The KJP has demonstrated flexibility and responsiveness to changing operational conditions of the program and reflected program changes into its M&E Plans. 21 4.2.2 Weaknesses Progress indicators largely measure output but are less useful to demonstrate KJP’s impact. Many of the progress indicators for FY 2013-2016 rely on the quantitative output of KJP’s work (number of courses, number of training participants, and number of DVDs produced and disseminated) to measure progress. For example, out of eight indicators used to measure progress under Objective 1 for FY 2013- 2016, only one can be considered more relevant and appropriate to measure how the KJP’s activities may lead to a change in the judicial capacity to write decisions.18 Similarly, out of six progress indicators for Objective 2, all are designed to measure project output (i.e., number of laws and regulations adopted with project, number of plans created) and are not adequate for measuring the impact of project interventions.19 These indicators are useful in that they show that activities are happening and generating output, but are less useful in demonstrating how the KJP’s outputs have led to increased judicial capacity or court efficiency. The new PMEP created to reflect the changes in the KJP’s objectives after the 38-month FCE shares some of the output-oriented features applied during FY 2013-2016. For example, Objective 1 for FY 2016-2019 called “Increasing judicial capacity to consider cases in civil law” tries to measure more or less the same things as the Objective 1 (outcome 2) for FY 2013-2016 through 11 indicators. Out of 11, eight indicators measure outputs and three try to capture change or increase of capacity brought about by the project. Similarly, Objective 2 for FY 2012-2019, which intends to measure the increase in capacity to adjudicate cases in criminal law with specific reference to CVE, has eight progress indicators, six of which are output￾oriented and two are change or impact-oriented. With regard to Objectives 3 and 4 for the FY 2016- 2019, two project indicators under Objective 3 and three indicators under Objective 4 are more output￾oriented measurements. Despite the more elaborate quantitative and qualitative approaches introduced by KJP to measure progress, a review of the table of indicators organized by program objective and corresponding outcomes shows that the vast majority of the indicators even for the FY 2016-2019 period are good to indicate output but are less helpful to show what changes were brought about by KJP’s activities. This does not contradict the earlier statement that ABA/ROLI showed increased commitment to improve the data collection methods and expected frequency of collecting such data after 2016. Instead, this reiterates that regardless of the positive changes introduced, the majority of indicators are still geared toward measurement of output. Several other factors have affected the effectiveness of progress reporting. First, in the case of the KJP, programmatic interventions after FY 2016 increasingly resembled on-demand TA activities agreed upon with the SC and approved by USAID. The purely technical approach prioritizes output and number of activities and is rather limiting for more relevant measurement allowing for “before and after” analysis of the effectiveness, either through demonstrable use of the introduced practices or discernible change in the capacity or behavior of beneficiaries. Second, activities aimed at higher-quality decisions, ADR, and court openness are part strategies and plans of the SC implemented through a multi-stakeholder and 18 Objective 1 has the following eight indicators: Outcome 1: Number of Laws, Regulations, and Procedures Related to Judicial Independence Supported with USG Assistance; Outcome 2: Number of Judges and Judicial Personnel Trained with USG Assistance, Number of courses developed to enhance judicial capacity, Number of TOT trainees trained to conduct independent trainings for judges, and Increased knowledge among judges about judicial decision writing; and Outcome 3: Number of Laws, Regulations, and Procedures related to court administration and management supported, Number of courts piloting improved court administration & management procedures, and Number of court administration & management manuals drafted and disseminated. 19 Objective 2 has the following six indicators: Outcome 1: Number of Laws, Regulations, and Procedures related to court administration and management supported, and Number of standardized procedures and updated management plans implemented by the SC; and Outcome 2: Number of Judges and Judicial Personnel Trained with USG Assistance, Number of courses developed to enhance judicial capacity, Number of DVDs produced and disseminated on judicial writing/court administration and management, and Number of TOT trainees trained to conduct independent trainings for judicial personnel. 22 multi-annual approach in which tracking and reliably reporting the contribution of individual organizations becomes difficult due to a certain degree of co-creation of various deliverables. Third, KJP has undergone various program modifications, obstacles, NCE, FCE, slow down orders, and priority shifts which have adversely affected the stability and predictability of the programmatic activities—both of which are essential elements of a solid M&E framework, particularly for conducting “before after comparisons” measuring the effects of sustained efforts over several years. Fourth, several Performance Indicators are less adequate for tracking the progress towards meeting targets because they do not have not established targets or the targets require only qualitative input (i.e., Indicator 4.1: Number of media reports after trainings in the regions where the training took place, Indicator 2.2: Progress based on the SC positive feedback). Lastly, many interventions of KJP (i.e., large conferences, roundtables, workshops) are useful to provide a platform of inter-institutional dialogue but are not necessarily expected to result in some tangible outcome even when they are contributing to project objectives. The ET found that the predominantly technical nature of ABA/ROLI’s interventions constrained programs ability to effectuate, track, and demonstrate the effectiveness beyond delivery of output required; and ABA/ROLI has largely delivered the technical output it was asked to deliver. Programmatic activities attempting to show increase of capacity (i.e., knowledge retention and utilization by program beneficiaries) usually face difficulties to reliably demonstrate change relative to other projects that can more easily measure the impact. For example, a business enabling environment program can show its effects by demonstrating that, due to project activities, the number of days to register a business has been reduced from 35 days into 12 days. Similarly, a program supporting farmers can show progress by demonstrating the increased export levels by the farmers benefiting from the program over a fiscal year. The above￾mentioned limitations made tracking progress towards achieving stated objectives hard to document even for the areas where the positive effect is discernible and confirmed by beneficiaries, such as investment disputes or improved transparency. Lack of baseline data limited the ability to measure effectiveness in a timely and meaningful way. In FY 2013-2016, according to its PMEP plan, KJP was expected to track the level of increased knowledge among judges about judicial decision writing. This was supposed to be done through three main instruments: pre and post self-evaluation forms completed by judges, interviews, and analysis conducted by the KJP staff after which baseline data could be generated. Baseline data generation was affected by the modification of the program description and until the third quarter of FY 2016 other data sources were used to measure the progress. While performance data under each objective showed that targets set by KJP were exceeded, these targets are not very useful as they do not measure how activities (i.e., production of manuals, training of judges through in-person and distance learning formats) have led to higher quality of written decisions by judges. As a result, baseline data for various indicators related to higher-quality written decisions (judicial acts and judgments) could not be obtained as planned. Baseline data did not exist to measure several indicators (i.e., Indicator 1.2, Indicator 2.1.3, and Indicator 4.1) even after the extension for FY 2016-2019. 20 The ET found that the lack of baseline data limited the ability of the project to set benchmarks or targets in order to capture and report the effects of project activities in a meaningful way. Even in the areas where KJP invested considerable resources and funding, i.e., the development of manual and training programs to increase the quality of decisions, it has been challenging for KJP to capture beneficiaries’ knowledge retention and application for several reasons. The self-evaluation forms show a very high level of acceptance of increased knowledge and its use (between 92-100 percent) but the reliability of these high levels can be questioned due the “halo effect” bias of participants. During the course of the field work, the ET observed a strong tendency among judges to under-report critical views toward foreign assistance projects. This limitation was well-articulated by a senior judge who stated: 20 See the new PMEP attached to the Work Plan covering the Year 2016-2017. 23 “I am not sure how useful my statements will be for your evaluation because it is against our moral codes and traditions to criticize individuals or organization who are trying to help us. Of course, it is up to us to make the best out of ABA/ROLI’s assistance but you should not be surprised if you only hear positive comments.” Statements like this may not be representative of the entire population of evaluation respondents as the tendency to report positively varies based on the informants’ background and links to the project. They merely alert the evaluator that some degree of positive reporting towards TA projects is present in a certain group and must be accounted for over the course of evaluation. This also because the main strength of self-reporting evaluation forms as a M&E instrument generally lies in informing the program about the need for course adjustments and change of approaches. In the case of KJP, the use of self-reporting necessitated additional sources of qualitative evidence to support data on the percentage of surveyed judges who reported that their skills in decision writing improved over the program (i.e., 98.7 percent for Years 2-5). Later in the project, KJP tried to combine the systematic use of the self-reported satisfaction, knowledge retention, and applicability with data generated by baseline studies. According to project documents and KIIs with KJP staff, the first baseline study included 102 civil law decisions made between 2015-2016 to assess the percentage of decisions that meet average/moderate clarity and consistency criteria based on ABA/ROLI’s methodology for judicial decision writing. The analysis found that 63 cases’ (61.7 percent) decisions followed the methodology. ABA/ROLI used the results of this analysis to set benchmarks for major PMP indicators to measure progress made over the course of the program. To measure progress, in 2019, ABA/ROLI conducted a follow-up analysis of 105 civil law decisions made between 2017-2019 to assess the percentage of decisions that met the standards set through the decision writing manuals and courses developed by ABA/ROLI. This analysis dated August 19, 2019, found that 74 (70.5 percent) of decisions followed the standards set through the decision writing manuals and courses showing a 10 percent increase in the number of decisions meeting the standards in comparison to the baseline study. KJP’s efforts to gather baseline data to more accurately measure changes resulting from program activities are positive and show its commitment to enriching its data collection approaches. In principle, external/independent assessments are much more reliable to demonstrate positive or negative change in the capacity of the beneficiaries. In such cases, the findings of external evaluators can refer to self-reported evaluation forms and user surveys as supplementary data to support the findings from in-depth analysis. However, the ET identified several limitations to the baseline and follow-up analysis conducted by ABA/ROLI. First, although the annual report indicates that ABA/ROLI coordinated the follow-up analysis by an independent evaluator, project documents also indicate that the same expert has been contracted by ABA/ROLI for various assignments related to the activity she was supposed to evaluate.21 The same expert has also been engaged by ABA/ROLI to develop five of the seven writing manuals and associated trainings (see Table 6). Second, the same expert has developed the ABA/ROLI methodology for judicial decision writing, thus also setting up the criteria and standards which are unique for the ABA/ROLI and are not externally known and publicized. ABA/ROLI’s new PMEP, starting in FY 2016, refers the evaluation criteria as “ABA/ROLI metrics” which presumably are developed by the same experts engaged by ABA/ROLI to develop resource materials and training on judicial decisions. Third, the same expert has collected and 21 For the baseline study, see the Quarterly Report covering the period October 1, 2016 through December 31, 2019, pages 3- 4 stating that the baseline analysis was conducted by retired Judge Kuanova. See also the Attachment B to the same Quarterly Report titled “Breakdown of Cases Analyzed for Baseline Survey of Judicial Decisions.” 24 analyzed the data for the follow-up analysis dated August 19, 2019 and confirmed the increase of percentage.22 While the ET understands the fact that KJP may have intentionally identified the same expert due to her accumulated knowledge of the quality of decisions and her ability to track change in the judicial quality of course-takers and manual-users, the fact that she has authored most of the manuals developed by the KJP over the years makes the qualification as “external or independent” expert as required by the KJP PMP and social science research protocols difficult to sustain. The ET observes that even if one leaves the question of contractual relations between the KJP and the expert aside, the validity and reliability of the results would have been strengthened if an advisory panel/committee consisting of a sitting or retired judge, law practitioner, and a law professor would have undertaken this work. Thus, while the approach of comparative analysis to show project effectiveness is highly recommended, engaging external experts would strengthen the reliability of the analysis. CONCLUSIONS • ABA/ROLI has diligently worked to improve its M&E system by dedicating more time, home office involvement, and in-country human resources to refine the progress indicators, diversify the data collection methods, and track progress towards meeting the program objectives. • Overall, qualitative aspects of progress monitoring remained weak as various factors constrained the ability of ABA/ROLI to effectuate, track, and demonstrate effectiveness beyond delivery of output required. • The introduction of qualitative elements allowing for “before and after” comparison based on the baseline data generated in 2016 is positive. • The validity and reliability of the findings for the first comparative study in July-August 2019 could have been strengthened by engaging outside experts who are more removed from the activity being assessed for impact. RECOMMENDATIONS • USAID should continue to encourage the integration of a qualitative element in ABA/ROLI’s M&E plans. For example, ABA/ROLI may need to collect thematic case studies to identify successful results in certain areas of assistance (CVE, ADR, investment justice) to better describe the contribution and added-value of the KJP’s interventions. This can be done through qualitative story collection and semi-structured interviews with beneficiaries and non-beneficiaries. • USAID should consider asking implementing partners (IPs) to utilize Outcome Harvest and Complexity Aware Monitoring Approaches to better document the impact of their interventions. For example, Outcome Harvesting—as an evaluation approach that is more amenable to tracing results in a situation when assistance is offered in a complex environment with changing contextual conditions and multiple stakeholders—could have addressed some of the limitations. 4.3 EQ3: GOING FORWARD, HOW, AND IN WHICH AREAS, SHOULD USAID WORK TO INCREASE PUBLIC TRUST IN THE JUDICIAL SYSTEM OF KAZAKHSTAN? FINDINGS 4.3.1 Political developments and their impact on judicial reforms 22 See the Annual Report covering the period October 1, 2018 through September 30, 2019 indicating in pages 22-23 that an analysis signed by retired Judge Kuanova concluded that there has been about 10 percent increase of the decisions deemed compliant with ABA/ROLI methodology on judicial decisions. Attachment B titled “Analysis of the Structure and Content of Court Decisions for Compliance with the Methodology for Drafting Court Decisions” was signed by Judge Kuanova on August 19, 2019. 25 The political will to implement large-scale reforms is questionable despite rising public pressure for deeper political changes. This evaluation took place at a time when Kazakhstan is nearing its third decade as an independent state and almost 10 years since USAID/Central Asia developed the KJP. KIIs and FGDs revealed diverse perspectives among respondents about the credibility of commitments to political reform and the breadth and pace of reforms. The vast majority of informants from the judicial system were hesitant to talk about the current political environment, but other legal professionals, representatives of CSOs, and international organizations were more forthcoming. Many respondents (18) commented on the political transition triggered by the resignation of the First President Nazarbayev in March 2019 and his succession by President Tokayev after the presidential elections of June 2019 and its consequences for political and rule of law reforms. Eight (8) program managers and senior officials (out of 11 interviewed) belonging to various international organizations (seven) mentioned that, while public pressure for political changes and rule of law reforms is rising, the political establishment is failing to grasp the extent of public dissatisfaction. Partial efforts to respond appear to be geared towards safeguarding the regime stability through system maintenance strategies combining limited repression and image-polishing initiatives without fundamentally changing the authoritarian nature of the regime. A long-serving senior official of an international organization asserted the following: “Most people do not believe anything that the ruling elite says about political liberalization and state reforms. There is a feeling that the political establishment wants to show that it is listening to the public, but it is not genuinely committed to deliver reforms.” International reports also claim that street protests since Nazarbayev’s resignation and Astana’s renaming into Nursultan have demonstrated a level of popular disaffection with the partial handover of power that is far higher than the authorities acknowledge.23 A senior program manager in one of international organizations which recently completed a fact-finding mission on the political and legal situation in Kazakhstan expressed her skepticism by stating: “We are very skeptical about the intentions, direction and the pace of the announced reforms. It is not clear if the political situation is becoming more favorable or the regime is merely changing strategies to continue its rule. Seven (7) informants (out of 18 who chose to comment) appeared to be more optimistic about the political direction of the country suggesting that the political establishment is trying to address the “public trust deficit” through cautious reforms. One of the respondents stated: “Our society and leadership are maturing and there is a positive momentum for deeper political changes. I cannot see how the regime would continue ‘business as usual’ and pretend nothing needs to change when protests keep spreading.” Although more information is required to fully understand Kazakhstan’s evolving political dynamics, based on the available evidence, the ET observed that most respondents still question the credibility of leadership’s commitment to reforms and the extent to which reforms will genuinely lead to the enhanced independence and accountability of the judicial system. The situation is fluid due to the rising popular dissatisfaction and growing public willingness to challenge the political establishment through street protests, but the political will for real and far-reaching reforms is questionable. This uncertainty over the direction and pace of transition warrants USAID to continuously monitor and reassess political 23 Chatham House Report (November 29, 2019), “Kazakhstan: Tested by Transition” available online at https://www.chathamhouse.org/publication/kazakhstan-tested-transition. 26 developments in the design and implementation stage of the new project in order to avoid a repetition of the KJP situation where the project could not realize its original objectives because of lack of political will. Judicial reform and increasing public trust in judiciary are high priority issues. Contrary to the earlier question, almost all respondents interviewed commented on the prevailing situation in the justice sector. The ET learned from the literature review, KIIs, and FGDs that several judicial sector strategies and action plans to modernize the court system have been implemented since 2012. The 2050 Strategy, Plan of the Nation-100 Concrete Steps, and “Rukhani Changyru” have focused on upgrading the legislation, improving judicial efficiency, digitalizing judicial services (e-Justice), and increasing public trust in the courts. Almost all courts now have video and audio recording equipment and well-resourced citizen reception offices (front offices) and are gradually becoming more user-friendly to the general public and media. Reforms of substantive and procedural codes, the introduction of adversarial elements in the criminal justice system, investigative judges, jury trials, expansion of ADR, the transfer of the AOJ, and consolidation of the High Judicial Council (HJC) were mentioned among the important developments of the justice sector since the KJP started in 2012. Despite these wide-ranging reforms, the judicial branch is not perceived to be independent, impartial, and professionally competent. It suffers from very low public trust and widespread perception of judicial corruption. Various international rankings show very little progress in areas related to rule of law, judicial independence, and civil liberties over the KJP’s project lifespan. Table 8: Kazakhstan in World Ratings Year Doing Business Report24 Freedom House Aggregate Freedom Scoring25 Judicial Independence26 (World Economic Forum) World Justice Project27 Bertelsmann Stiftung’s Transformation Rule of Law Sub￾Index28 Corruption Perception Index of Transparency International29 2012 28 23 94 n/a 3.3 133 2013 27 22 88 71 3.3 140 2014 30 24 86 65 3.0 126 2015 9 25 72 73 3.0 123 2016 9 23 68 64 2.8 131 2017 6 22 79 2.8 122 2018 4 22 73 65 3.0 124 Subordination to the executive power, poor judicial competence, and high level of corruption were identified as the most critical challenges to gain public trust. Judicial deference to the executive power is also confirmed by the SC which in 2018 identified the extremely high level of judicial deference to prosecutorial request for criminal convictions (99.8 percent) and arrests (93.4 percent) as well as a high number of judicial decisions in cases involving state bodies or state-controlled companies (67 percent of cases) among the reasons for low public trust. 24 Measures: Good practices; Maintaining a specialized commercial court, division, or judge; Introducing small claims courts or simplified procedures for small claims; Introducing or expanding case management system; Court automation; Computerization and court efficiency; and Using alternative means to resolve disputes. 25 Measures aggregate freedom scoring as a composite result of: 1) freedom, 2) political rights, and 3) civil liberties. “Least free” is zero (0) and “most free” is 100. Kazakhstan is rated as “not free” with average score 22 out of 100. 26 Measures judicial independence by the extent of undue influence of members of government, citizens, or firms on the judicial branch. Scores close to 1 mean “heavily influenced” and scores close to 7 mean “entirely independent.” 27 Measures countries’ rule of law performance across eight factors: constraints on government powers, absence of corruption, open government, fundamental rights, order and security, regulatory enforcement, civil justice, and criminal justice. 28 Measures Rule of Law Score by considering: 1) separation of powers, 2) independent judiciary, 3) prosecution of office abuse, and 4) civil rights. 29 A biannual study of Corruption Perception in Public Sector calculated using 13 different data sources from 12 different institutions that capture perceptions of corruption in 191 countries. Kazakhstan ranked 124/191 in 2018. 27 According to international indices and KII feedback, the low public trust is not unique for the court system but appears to be an endemic problem across state institutions and state-run companies in Kazakhstan. Among the important developments since the inauguration of the new president, KIIs (seven) mentioned the National Council on Public Trust, a 44-member forum of civil society representatives established by the new President in September 2019. The Public Trust Council is premised on the idea of a “state listening to the citizens” and aims to increase public trust in state institutions including the judicial system by promoting diversity of political opinions under the slogan “Different Opinions – One Nation.”30 A set of planned reforms were announced in the second meeting of this Council which took place during the evaluation in December 2019, including: strengthening Parliament’s role; creating conditions for political competition; removing restrictions on freedom of peaceful assembly; relaxing legal rules of libel and defamation; and making central and local institutions (Akimats) more inclusive and open.31 Due to the far-reaching powers the head of state has in relation to the judiciary, increasing public trust in the judiciary seems to be a high priority for the President. Since 2018, a presidential working group on judicial modernization consisting of senior judicial leaders and members of the presidential administration is working to improve the legal framework on selection, appointment, evaluation, and discipline of judges also based recommendations of the Venice Commission experts. Judicial reformers appear to be in a minority. The ET noticed that informants’ perspectives were divided on how to address the “public trust deficit.” Most judges (22 out of 24) recommended measures related to improving judicial capacity, modernization of courts, trainings, and improving judicial outreach efforts to media and the public. However, other informants who do not have positions in the judiciary thought that the GOK needs to tackle fundamental issues related to judicial independence and accountability to gain the public trust in the judicial system. Other respondents (7) also thought that changes are slow and incomplete but there has never been a better time to engage constructively to make judiciary more accountable and transparent. Another legal professional stated that: “There are many problems with the court system but we cannot deny that things are changing for the better. Many positive developments were only recently introduced whereas public trust will take time to increase” A very small minority of informants (3) asserted that the judicial system is so rotten that only radical changes would increase public trust in judiciary. A legal community representative stated: “As it is now, our judicial system is designed to produce ‘soldiers’ and not ‘independent judges.’ Over 90 percent of the judges come from the judicial personnel and law clerks without merit-based procedures but due to support from powerful people within judiciary. Most of the new judges are obedient to those who have helped them get into judicial office.” A large number of informants (40) observed that the SC has taken some positive steps to enhance court transparency and openness under the leadership of the new Chief Justice. In January 2018, the new Chief Justice announced the “Seven Pillars of Justice” package of reforms based on the analysis of more than 1,000 recommendations received by judges and member of the legal community. Several priority areas of reform have been launched to improve: quality of court services, quality of procedure, quality of information technology (IT) products in the judiciary, quality of judicial decisions, quality of judges, and quality of alternative dispute resolution. 30 Fredrick C. Starr (September 17, 2019) “First glimpses of Tokayev’s Kazakhstan: The listening state?” available at https://www.atlanticcouncil.org/commentary/long-take/first-glimpses-of-tokayevs-kazakhstan/. 31 Helsinki Times (December 31, 2019), “President of Kazakhstan Proposes New Political Reforms” available at https://www.helsinkitimes.fi/world-int/17153-president-of-kazakhstan-proposes-new-political-reforms.html. 28 The ET learned that many of reforms of the Seven Pillars of Justice are being implemented by a Project Management Unit with TA from international donors like the WB, UNDP, GIZ, USAID, and others. Twelve (12) non-judge informants mentioned that the SC is spearheading efforts to improve court relations with the media and the public and some of the initiatives have generated positive feedback from citizens. Many of these informants (8) also pointed out that the Chief Justice works well with international projects through the Project Management Unit. Six (6) informants were more reserved and pointed out that the Chief Justice has been successful in delivering on high-visibility, quick impact initiatives (better court and media relations) but, so far, has failed to undertake reforms on more relevant and sensitive issues in the judiciary. Even those who commended the Chief Justice highlighted that his ability to carry out reforms is constrained by other powerful people within and outside the judiciary who are resistant to fundamental changes in the system. Eighteen (18) informants who commented on the internal dynamics of the judicial system identified chairmen of regional courts as influential figures who, despite the erosion of powers through time-limits of their tenure effectuated by the amendments to Law on HJC in 2019, still exert significant power in the system. ET learned that the HCJ is increasingly becoming more important as the self-governing institution of judiciary. Initially created in the early 2000s as an advisory body to help President discharge his duties in relation to the judicial system, the HJC became an autonomous institution in 2016.32 Its legal mandate was broadened in 2017 and 2019 to better define its powers to administer a qualifying judicial examination and select, appoint, promote, tenure, and remove judges from office.33 During 2019, the HJC has tried to develop contacts and share best practices with international counterparts like the High Council of the Judiciary of the Italian Republic, Council of the Judiciary of the French Republic, and the International Association for Court Administration. Although results have yet to be seen, six (6) informants told the ET that the current leadership of the HJC appears committed to intensifying judicial reform and has also received technical support from the UNDP under the auspices of the WB justice strengthening project. 34 The ET also learned of divergent views within the judicial leadership on some important issues regarding judicial independence. For example, evaluation of judges is still under the SC but the ET learned about outstanding differences between the HJC and SC as to where the responsibility to evaluate judges should rest. While currently a weak and marginal institution in the judicial system, the UOJ is likely to be more vocal in advocating for a clearer separation of powers between the three branches of government and its leadership is largely critical of the subordinate role of judiciary. These dynamics may turn out to be important as USAID launches its future justice project. Issues like public trust in judicial examination, appointment, education, discipline, dismissal, and evaluation will continue to be topical in the years to come; cooperating with judicial leaders and institutions that can carry out relevant reforms will be essential for USAID and other international donors. 4.3.2 Working methods Very few informants provided ideas about most appropriate methods of work to implement justice sector assistance. The following findings primarily drew from desk research, KIIs with USAID and KJP staff (7) and representatives (11) of international organizations working on various aspects of the justice system. On-demand TA is a “safer instrument” but may prevent USAID from addressing more fundamental issues of the judiciary. 32 High Judicial Council was created by Law No. 436 “On High Judicial Council of the Republic of Kazakhstan» adopted on December 4, 2015 and effective as of January 1, 2016. 33 Amendments to HJC powers were recently made with Law No. 227-VI of the Republic of Kazakhstan dated February 21, 2019. 34 The World Bank has asked for 18-month extensions of this project which is financed through an investment loan managed by the Ministry of Justice of Kazakhstan. Program description is available in English at http://documents.worldbank.org/curated/en/861551506335166453/Kazakhstan-Justice-sector-strengthening-project. 29 Nine (9) informants noted that the GOK is considered to be the most resourceful, stable, and sophisticated government in the Eurasian region. With a highly educated population and stable state institutions, the GOK is very capable of articulating its goals and adopting strategies and action plans to realize them. International observers also recognize that Kazakhstan is gradually improving its international standing and has ambitions to further expand its global footprint.35 While it benefits from donors’ assistance and investment loans, the GOK also provides important development assistance in the Eurasian region and is consolidating its own official development assistance mechanism called KazAid.36 USAID and other donors observed that the GOK is sufficiently resourceful to be able to undertake large investment. For example, in the last four or five years it has funded ambitious e-Government and e-Justice projects. Astana International Financial Center (AIFC) and its arbitration tribunal is also financed by the GOK to offer large investors a more effective mechanism for resolving commercial disputes between them, modeled around British common law. Three program managers in international organizations noted the AIFC shows that funding for large investment is available when a project aligns with the GOK’s interests, although investment justice for the rest of the citizens remains inefficient. The vast majority of respondents told the ET that USAID’s reputation is very good and that USAID’s further assistance to the judiciary is welcome by justice institutions—not because of funding shortages but because USG support is important to the GOK. Another important lesson learned from the KJP is that providing on-demand TA can have positive results but prevents USAID from tackling more important issues of the judiciary. Through document review and discussions with USAID and KJP staff, the ET learned that ABA/ROLI’s implementation proceeded more successfully when it revised its objectives to align with the SC’s priorities. On the other hand, 15 informants suggested USAID should continue to support further modernization of court processes/administration in accordance with national strategies and action plans while trying to design a more ambitious project that strengthens the public confidence in the judicial system. A senior judge articulated the need to go beyond on-demand TA in the following way: “Everything we do with or without USAID, from the first instance courts to the Supreme Court, must benefit the society of Kazakhstan. Otherwise, trainings and study tours will simply benefit a small number of people without improving public trust in the judicial system.” CSO engagement may help keep the momentum for judicial reforms. Given that the ET did not have information on the USAID plans for next generation of judicial sector programming, the team tried to collect information on the existing civil society capacity to implement rule of law projects irrespective of the design and implementing mechanisms determined by USAID. All informants (12) who commented on this matter noted that the conditions for CSO inclusion in programming appear more favorable than five years ago as the GOK has declared its intention to engage CSOs in its efforts to respond to public dissatisfaction. However, the GOK appears resolved to control the agenda of the National Council of Public Trust, format of CSO inclusion in the planned reforms, and timing and sequencing of their implementation. USAID staff stated that although the number of direct grants to local organizations is growing in the recent years, the capacities to implement projects similar to or more ambitious than the KJP may still be lacking. Four (4) informants working in the rule of law sector confirmed that the limited capacities of the local CSOs to become main IPs of a relatively large judicial program are inadequate. Furthermore, three (3) senior representatives of CSOs working in the rule of law sector noted that very few CSOs have sufficient expertise in matters related to judicial self-governance, court monitoring, conducting evidence-driven research and implementing advocacy campaigns by constructively leveraging public engagement to support 35 Kazakhstan: Tested by Transition. 36 World Bank Report (April 2018), “Kazakhstan Systematic Country Diagnostic: A New Growth Model for Building a Secure Middle Class.” 30 judicial reforms. Five (5) representatives of international organizations converged on the idea that the local NGO capacities to conduct research on topics related to judicial independence, judicial competence, impartiality, and integrity could be strengthened, but warned that given the sensitive nature and broader consequences of the rule of law reforms, “too much” activity from CSOs may be seen as a political threat to the ruling elites and trigger a hostile response from the GOK. To address the problem of low capacity and maintain the support of host government, a future project could be predicated on the development hypothesis that the justice system Kazakhstan can be strengthened through: 1) improving the legal framework; 2) developing the capacities of the self-governing institutions to strengthen judicial independence and accountability system; 3) increasing the capacities of judicial education institutions to better train judges; and 4) encouraging CSO and public engagement to monitor, report, and advocate for a fairer, more transparent, and accountable judiciary. A possible Theory of Change for the next program that combines the Supply and Demand under one umbrella would hold: “If the positive momentum for further modernization and reforms of the court system is supported by capable CSOs which can engage in evidence-based advocacy efforts to strengthen judicial independence and judicial accountability, public interest in judicial reforms and public trust will also increase.” 4.3.3 Priority areas of justice sector TA Almost all informants expressed their opinions on the areas in which USAID should focus its work to increase public trust in judiciary. Those who work in private practice or the CSO sector mainly converged about how hard and difficult it is to transform the judiciary. Accordingly, they proposed initiatives that bypass or minimize citizen interaction with the courts, such as expanding mediation and arbitration, broadening the scope jury trials, or investing in the Bar Association or law schools. Judges proposed more training programs, exchange visits, and educational resources. Work with the HJC and SC on judicial independence and modernization. As mentioned in the preceding sections, increasing public trust will require USAID to provide assistance to the HCJ and SC in upgrading and improving the judicial independence and accountability systems. The fact that the HJC is becoming increasingly more important in self-governance of the judicial system was also confirmed by 11 informants from various backgrounds. Although the UNDP intends to continue supporting the HJC if the project is extended, by the time USAID awards the next judicial program, there may be an opening for TA. USAID should consult with the HJC on the potential assistance to improve policies and procedures on examination, selection, promotion, discipline, and dismissal of judges. The current leadership of the SC appears committed to proceed with full implementation of the “Seven Pillars of Justice” initiatives aimed at strengthening the rule of law and enhancing public confidence in the judiciary. These initiatives enter their second phase of implementation in 2020 and will target long-term systemic changes in the areas of court administration, judicial training, procedural legislation reforms, etc. The SC’s representatives underscored the importance of sustainability and strategic input by international programs like USAID to complement their ongoing reform activities. On the other hand, USAID can work with the SC to inject new ideas on improving the judicial system and offer tailored technical expertise that can lead to systemic improvement. Therefore, cooperation of USAID and the SC in developing a well￾informed policy proposal will be mutually beneficial. The ET also learned that while the judiciary does not formally have the right to initiate legislation, the SC is particularly influential in policy developments, often acting as a principal drafter of laws related to the judiciary. The following areas should be considered for USAID’s future programmatic assistance to the SC. 31 Enhance the SC’s capacity for policymaking to support further modernization of the civil and criminal justice system. In FGDs with all three panels of the SC, respondents mentioned their interest in learning more about the European and U.S. experience in administrative courts, pre-trial judges, jury trials, and other areas of law in order to assess the practical implications of some recent justice reforms and how to further align them with international best practices. USAID should consult further with the SC; however, the ET believes that there is room for an ambitious framework of cooperation in: 1) offering expertise to assess the Civil Procedure Code and identify areas where amendments are needed; 2) supporting the SC to conduct empirical research on the practical implications of procedural norms in terms of length, quality, and efficiency of adjudication in civil matters and their impact on citizens and companies; and 3) providing hands-on expert assistance in developing a comprehensive concept document to modernize the Civil Procedure Code. Similarly, the SC is initiating the next stage of criminal justice reforms which will revise the role of the courts and prosecutors with the aim of enhancing fairness and due process standards in criminal cases. The SC requires the assistance of legal scholars to fine-tune the new criminal procedure provisions norms to this effect. The types of activities include: legal and empirical research on the real situation with fair trial standards in criminal cases; and expert discussions to inform policymakers on the existing gaps and potential impact of new procedural norms. Modernize court processes and services. During the FGDs and through document review, the ET learned of the SC’s intentions to apply new public management principles to court administration. The Project Management Office is working on improving the efficiency, quality, and effectiveness of judicial services by using corporate methods of management and innovation. The Director of Department of Administration informed the ET that following are the directions which the SC plans to pursue in 2020: 1) reforming the organizational structure and functions of the Department of Court Administration; 2) process improvement and re-engineering; 37 and 3) developing the judicial system’s human capital. 38 Develop/strengthen the AOJ. With regard to the judicial education, nine (9) informants suggested further assistance to raise the capacity of the AOJ to provide higher-quality judicial education for aspiring and sitting judges. The new leadership of the AOJ also expressed its interest in working with various international actors and establishing long￾term partnership agreements to pursue modernization of judicial training. Specifically, the AOJ is seeking cooperation in the following areas: • Organizational and Management Change: This includes changes in the organizational structure and composition of the teaching staff, as well as in the approaches to designing training programs. The AOJ is interested in developing institutional cooperation and exchange experiences with judicial training schools in other countries. This includes building long-term partnerships with leading 37 According to the Department of Court Administration of the SC, based on the guide produced by the Joint Technology Committee (JTC) of the Conference of State Court Administrators (COSCA) and the National Association for Court Management (NACM), the process improvements start with developing an appropriate governance and management structure and aligning business objectives with the court’s strategic mission and goals. It follows with technical activities: analyzing existing processes “AS IS;” conducting gap analysis; developing alternative process solutions to existing court business process “TO BE;” and analyzing their cost/benefits. This is followed by implementing and testing process improvement models and measuring, revising, and standardizing the new processes. 38 For example, the Department of Court Administration would like to initiate a summit of regional court administrators to provide them with opportunities for learning and collaboration. The idea is to engage them as change managers to effectively promote current reform initiatives in their courts. This will also enable the SC to create a network of court human resources professionals. The idea is to conduct such fora annually to achieve sustainability of their results. Additionally, the SC would like to offer training to court managers and, applying innovative court administrative procedures, develop effective budget plans, supervise staff with confidence, manage relationships with judicial staff and conflicts between team members, etc. The long-term objective is to introduce permanent management courses for court administrators and judges-chairmen of courts at the AOJ. 32 judicial training schools and participating in important international events on the development of judicial training (i.e., participation at the conferences of the International Organization for Judicial Training, which facilitates sharing of the world’s best practices and advanced methods of judicial training). • Faculty Development: The AOJ is also keen to learn the managerial aspects of organizing judicial training services, such as: efficiency and quality management, selection and professional development of teaching staff to apply interactive teaching methods and soft skill building, training evaluation methods, integrating advanced IT into new programs, etc. • Curriculum Development: The new strategy of the AOJ entails revising their curriculum for acting judges and judicial candidates. It also has plans to revise curricula for some of the traditional subjects, such as civil and criminal law, and to introduce new subjects: administrative law (i.e., adjudication of public disputes with administrative agencies of the government), and cyber law (disputes related to the IT sphere, such as: e-commerce and virtual contracts, digital privacy, copyright law, etc.). The courses on building soft skills of judges—i.e., judicial leadership, communication, stress-management, and personal development—are also planned under the new approaches to curriculum development. Apart from the SC justices, 13 informants stated that USAID should continue judicial exchange programs and exposure to international best practices for judges and judicial leaders on topical areas of law, including: international human rights law, cybercrimes, intellectual property rights, patent law, CVE, investment disputes, and consumer protection. Eight (8) informants also suggested that USAID and other donors should provide more training in international human rights law and the case law/jurisprudence of international human rights protection bodies (EU, Inter-American, and African courts on human rights; UN committees). Provide further support to ADR expansion. Five (5) KIIs suggested that USAID should support ADR expansion in and outside courts through public awareness campaigns and legal changes to make ADR the default mechanism for a larger category of cases. As mentioned in earlier sections, in 2018-2019, the SC initiated several pilots on in-court mediation schemes, introduced the role of judge-mediators, and began working with Akimats to incentivize mediation. In 2020, the SC is planning to invest their efforts into institutionalizing the results of these pilot projects. The SC informed the ET that it would like to cooperate more with USAID or other international projects in the following aspects of the ADR expansion: 1) evaluating the effectiveness of current ADR pilot projects and their potential legislative and practical implications; 2) evaluating court-users’ demand for mediation in civil disputes and assessing the impact of current legislative efforts to introduce judge mediators; 3) conducting needs assessment for advanced courses on mediation and ADR for judges; and 4) organizing professional/expert consultation in the form of roundtables, workshops, and conferences enhancing the use of ADR. Nine (9) judges and six (6) non-judge informants mentioned the need to improve legal education in public law schools (i.e., human rights courses, legal writing and reasoning, interactive teaching techniques). Others suggested that USAID should focus on developing the capacity of the Bar Association and practicing lawyers’ skills; however, five (5) informants—in addition to KJP staff—thought that this would entail a very large project. Two (2) informants suggested USAID should do more to support the UOJ’s efforts to raise awareness about judicial independence, impartiality, judicial ethics, and integrity (i.e., constructive advocacy efforts, training for judicial ethics, strengthening committees to conduct comparative studies of judicial systems in Organization of Economic Cooperation and Development [OECD] countries, etc.). 4.3.4 CSO engagement in judicial programs The ET tried to collect information on the type of assistance and areas of work outside the justice institutions. However, fewer informants made suggestions on USAID’s potential assistance to CSOs and 33 the general public. Twelve (12) KIIs who chose to comment on the role that CSOs can play stated that CSOs need to develop their own capacities to better monitor and report on the justice system. Eleven (11) informants confirmed the inadequate capacity of NGOs to conduct evidence-based research and advocacy campaigns on judicial affairs (i.e., professional competence, integrity, independence, financial asset disclosure, and avoidance of conflict of interest). Six (6) informants suggested that USAID should continue to support demand for a better justice system through citizen information campaigns targeting youth. Three (3) informants emphasized that information campaign should be increasingly targeting Kazakh language audiences both in off-line and online/new media. RECOMMENDATIONS Methods of Work: • Due to questionable political will for broad rule of reforms, USAID should continuously monitor and reassess political developments in the design and implementation stage of the next judicial sector program. • TA in the areas agreed upon with justice institutions appears to be more of a “safe instrument,” but the future judicial program should attempt to be more ambitious in working with the HJC, SC, AOJ, and UOJ to go beyond a purely technical approach. • USAID should consider combining two streams of work in the next program: 1) work with the HCJ and the SC on strengthening judicial independence and accountability; and 2) support CSO research and advocacy support for a more transparent and accountable justice system in order to increase public engagement and trust in judicial system. • Notwithstanding the decision regarding the Supply/Demand Design, USAID should support CSOs in the future program—either through a separate component on CSO engagement or through a grant scheme to support CSO research and advocacy efforts for a more transparent and accountable judicial system. Areas of work with justice system institutions: • Support the HJC and SC to upgrade and streamline laws and regulations to strengthen the independence and accountability of judges (i.e., selection, appointment, promotion, evaluation, and dismissal of judges). • Continue the support for the SC multi-annual plans to reform the civil and criminal procedure legislation, court modernization initiative, and plans for expansion of ADR use both in and outside courts through public awareness campaigns and introduction of legal changes with smart incentives/penalties aimed at making the ADR the default mechanism for a larger category of cases. • Provide further assistance to the AOJ on improving formative and continuous judicial education programs. This component may focus on strengthening institutions and developing curricula in topical areas of law (i.e., human rights protection, cybercrime, CVE, investment disputes, consumer protection). In particular, USAID should consider providing more training in international human rights law and the case law/jurisprudence of international human rights protection bodies (EU, Inter-American, and African courts on human rights; UN committees). Working with CSOs: • USAID should help CSOs increase their capacities to conduct research on topics related to judicial independence, judicial competence, impartiality, and integrity. Evidence-driven research and advocacy efforts should be designed to raise public awareness on the functioning of the judicial system and educate public demand on judicial independence and accountability. This may require USAID to identify a group of CSOs in Almaty and Nursultan to act as a primary local sub-grantees and develop their capacities for both grant management and evidence-driven research on the justice system. 34 • USAID should try to support projects that do not simply criticize the courts but also highlight judicial achievements that have been made over the years, such as: o Conducting public information campaigns to develop a deeper understanding of the workings of a court, the role of judges in upholding citizen rights, and the daily demands upon judges. o Supporting NGOs to conduct analysis of the laws, rules, and regulations needing revision to comply with international standards that the GOK aspires to meet. o Engaging the public in discussions without evoking hostility and resistance to reforms. o Supporting CSO research on issues related to court efficiency, judicial competence, and judicial corruption and creating opportunities for periodic meetings between the CSO coalition, presidential administration, representatives of the parliament, HJC, and SC to address the findings of CSO research before publicizing them. 35 ANNEXES 36 ANNEX 1: KJP EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK COMPONENT I – KJP EVALUATION C.1 TITLE Kazakhstan Judicial Program Evaluation. C.2 PURPOSE The purpose of this final evaluation is to determine the extent to which the Kazakhstan Judicial Program (KJP) achieved the objectives set forth in Cooperative Agreement. The evaluation will provide pertinent information and analysis that will assist USAID in assessing the success of the project. In addition, the evaluation will provide recommendations on interventions which USAID may wish to undertake in the future to address the failure to achieve any intended results. Specifically, the evaluation will help USAID to understand better the results from its judicial sector investments in Kazakhstan. C.3 BACKGROUND Kazakhstan is positioning itself to further align national practices with international norms and standards and rank higher in political indices. As discussed in the GOK’s “Strategy 2050,” not only are efforts focused on the immediate outcome of modernizing the judiciary to be synchronized with the dynamically developing international legal system, but initiatives also recognize the need for improved accountability to increase public trust in the court system. Specifically, reforms to four foundational legal codes, the Criminal Code, the Code of Criminal Procedure, the Penal Code, and the Administrative Code to humanize the legislation and make them better equipped to address modern legal challenges and preserve citizen’s rights. In 2011, Kazakhstan announced its desire to join the Organization of Economic Cooperation and Development (OECD). To meet OECD requirements, the government amended its investment legislation. In May 2015, President Nazarbayev identified “100 Concrete Steps” towards the implementation of the five institutional reforms and created an Investment Collegium under the Supreme Court. The President assigned 35 judges to the Supreme Court and over a 100 judges to lower-level courts. The 100 Steps strategy was developed as a roadmap for achieving goals set out in the “Strategy 2050.” At the end of December 2017, Mr. Zhakup Assanov was appointed as Chairman of the Supreme Court, and by January 2018, the new Chairman of the Supreme Court presented a package of reforms aimed at achieving the goals set forth in the “100 Concrete Steps,” the Kazakhstan 2050 Strategy, and the “Rukhani Zhangyru” Program for Modernization of Public Consciousness. These initiatives are a clear sign of the government’s interest in judicial reform. However, there is much work to be done, particularly with respect to cultivating public trust in the legal system—as mentioned by Kazakhstani President Nazarbayev in October 2018 during his annual address. The Supreme Court seeks assistance in the development of legislation and regulations to promote economic and investment courts, streamline decision-writing, and promoting the use of mediation/arbitration in business disputes. In particular, KJP also focuses on the priorities set out in cooperation between the United States and Kazakhstan in the field of two strategic aspects—countering violent extremism and improving the investment climate for international investors. These two aspects were identified as among the main directions of an enhanced strategic partnership between the U.S. and Kazakhstan when President Nazarbayev visited the White House in January 2018. C.4 PROJECT INTENT The KJP program’s period of implementation is from April 1, 2012-September 30, 2019 with a total estimated cost of $4,050,000. The initial period of performance was from April 2012 to July 2016, and the award was subsequently extended for 38-months through September 2019. The goal of the program is to 37 support the modernization of and increase the public trust in the judicial system of Kazakhstan, as well as capitalize on the achieved results. KJP cooperates closely with such institutions as the Supreme Court, the Union of Judges, and the Academy of Justice. The program implements multiple activities, which have influenced positive changes within the judiciary in Kazakhstan. KJP works to familiarize judges with international best practices in judicial decision writing techniques and substantive law, support the implementation of new legislation, and develop alternative dispute resolution methods. KJP contributes to the implementation of the “100 Concrete Steps” and the Kazakhstan 2050 Strategy. KJP provides technical assistance to the judicial system by developing resources and conducting training which enables judges to consistently issue well-reasoned decisions on cases related to investment law and violent extremism. KJP also developed and promotes alternative dispute resolution resources to increase the accessibility of judicial decision making and lessen the caseload of judges. Complementing these efforts, KJP works with the courts and press offices to conduct public outreach related to recent changes to the legal system, which in turn seeks to foster greater public trust in the judiciary. Partnering with key organizations in the judiciary and the public, KJP aims to support the modernization of and increase the public trust in the judicial system of Kazakhstan by pursuing two objectives: 1. Objective 1: Improved capacity of judges. 2. Objective 2: Enhanced efficiency and accessibility of the judicial system. C.5 EVALUATION QUESTIONS The evaluation will address the following key questions: 1. To what degree has the American Bar Association Rule of Law Initiative (ABA/ROLI) met the two objectives as stated in the Cooperative Agreement and what gaps in addressing the program objectives still exist? 2. What issues exist in the monitoring system established by ABA/ROLI which might have affected its effectiveness in reporting on KJP progress? 3. Going forward, how, and in which areas, should USAID work to increase public trust in the judicial system of Kazakhstan? C.6 DATA ANALYSIS METHODS This evaluation will be qualitative in nature with some secondary, quantitative data analysis. Prior to the start of data collection, as part of the evaluation work plan, the evaluation team must develop and present, for USAID review and approval, a Data Analysis Plan detailing: • What procedures will be used to analyze qualitative data obtained from key informants and other sources; • How the evaluation will weigh and integrate qualitative data from various sources with quantitative data from project performance monitoring records; and • How data will be used to develop conclusions about the effectiveness and efficiency of the Kazakhstan Judicial Program. The Contractor will have flexibility in proposing the most appropriate methodology to accomplish the tasks outlined in the most efficient way. However, the Contractor must address the following: Task 1: Desk Review The Contractor must conduct a desk review to gain a general overview of the judicial sector in Kazakhstan and identify areas (as related to the KJP objectives and evaluation questions) that merit closer attention once the team begins its field work. The Contractor must also review a broad range of other relevant 38 non-project related documents such as peer-reviewed articles and other publications and incorporate the review findings into the desk study and the data analysis plan. USAID will provide the evaluation team with a package of background materials, including: • USAID Central Asia Regional Development Cooperation Strategy (2015-2019); • KJP Cooperative Agreement (2012); • KJP Work Plans, Years 1-8; • KJP Annual Reports, Years 1-7; • KJP Quarterly Reports, 2012-2019; • KJP Initial Performance Monitoring and Evaluation Plan, 2012; • KJP Annual PMEP Report, Years 1-7; • Additional resources may be made available to the Project Team and USAID upon award and request. Task 2: Work Plan to include draft description of overall methodology In conjunction with USAID, the Contractor will develop a plan for conducting field work using information from the Desk Review and this SOW. The Work Plan must clearly outline the methodological approach the evaluation team will use. The Work Plan must be submitted for review to the COR for approval prior to arrival in country. The Work Plan must include a field work schedule, including a preliminary list of all key organizations, the evaluation team plans to meet. Task 3: Conduct field data collection The in-country data collection includes, but is not limited to, facilitated discussions through interviews with the following counterparts: ABA/ROLI; Supreme Court, Academy of Justice; Union of Judges: Government of Kazakhstan (Ministry of Justice); Donor counterparts (EU, UNDP); Judges, judiciary specialists (beneficiaries). The evaluation team will spend at least ten days in Kazakhstan with the evaluation team visiting Astana and Almaty. Travel to other cities of Kazakhstan is not anticipated but can be considered if recommended by the evaluation team. The identification of informants/respondents and scheduling data collection must be arranged as far in advance as possible to maximize efficient use of time while in the field. Task 4: Final Evaluation Report The final evaluation report must triangulate information obtained from the pre-field work desk review and from the field to articulate KJP contributions to the objectives as well as any opportunities to strengthen achievements during the remaining time frame of the program and beyond. 39 ANNEX 2: MATRIX OF FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Main Findings Conclusions Recommendations EQ 1: To what degree has the American Bar Association Rule of Law Initiative (ABA/ROLI) met the two objectives as stated in the Cooperative Agreement and what gaps in addressing the program objectives still exist? Findings on Extent of Accomplishments • Results vary across objectives but the Kazakhstan Judicial Program (KJP) has generally delivered the technical outputs it was asked to deliver. • In particular, ABA/ROLI has a strong reputation for delivering timely and high-quality assistance to the local beneficiaries based on agreed needs. • The KJP has met the Objective 1 technical assistance (TA) contributed to higher-quality written decisions and improved application of new legislation by judges. • The KJP has largely been successful in achieving Objective 2. Despite weak public trust in the judicial system, there is evidence to support the KJP’s positive contribution to the gradual improvement in efficiency and accessibility of courts. • Targets under indicators 1.1.1, 1.2.1, and 1.3.1 for FY 2016-2019 on the number of alternative dispute resolution (ADR) courses developed for judges and Master’s-level students and the Academy of Justice (AOJ) and on the number of ADR resources developed and distributed to judges have been met; but the AOJ’s utilization of those resources is not clear. Conclusions on Accomplishments • The KJP has been largely successful in achieving Objective 1 on higher quality of judicial decisions and improved judicial capacity to apply new laws. • The KJP’s assistance to improve the capacity of the AOJ to provide high-quality judicial education has been less successful due to weak capacity of the beneficiary. • The KJP’s contribution to expansions of the ADR mechanisms (mediation and arbitration) is showing positive results. The KJP’s role as a synergy provider and enabler for dialogue between the courts and media has contributed to enhancing court openness and transparency under Objective 2. • Under both objectives, due to the predominantly on-demand nature of the TA and the high number of output-oriented indicators it is difficult to reliably determine the impact of the KJP, but partners confirm the positive impact of the KJP’s deliverables. -- 40 Main Findings Conclusions Recommendations Findings on Gaps • Despite the extent of the KJP’s accomplishments, significant gaps to address the initial objective of strengthening judicial independence and accountability exist due to lack of willingness of the Supreme Court (SC) to work with the KJP in those areas at the start of the KJP. • The KJP has largely provided the TA to the AOJ according to the approved work plans but due to institutional weaknesses, which plagued the AOJ during 2012-2018, such as lack of strong leadership, strategic vision and effective management practices, its impact and sustainability have been limited. • Most activities at the AOJ were perceived as one-off initiatives by the beneficiary. For instance, courses supported by ABA/ROLI have yet to be formally integrated in the curriculum of training for sitting judges or Master’s students and evidence is weak to suggest that TOT programs improved teaching practices at courses offered to judges by the AOJ. Conclusions on Gaps • Remaining gaps in addressing the original program objectives (i.e., improving judicial selection, judicial independence, impartiality, evaluation of judges, etc.) exist but they did not depend on the KJP’s technical capacity. • The KJP’s ability to address key challenges of the judicial system, i.e., independence and accountability was limited by the lack of willingness on the part of the SC at the onset of the project. • The KJP’s assistance to improve the capacity of the AOJ to provide higher-quality judicial education has been less successful despite the delivery of technical outputs by the KJP. The impact and sustainability of KJP’s interventions is limited due to weak capacity of the beneficiary and frequent leadership changes at the AOJ. Recommendations on Gaps • USAID should continue to work on strengthening judicial independence and accountability procedures and systems in accordance with national strategies and action plans of the Government of Kazakhstan (GOK) with the aim of increasing public trust in the judiciary. • USAID should conduct further analysis on the current needs of the AOJ and provide further support to the AOJ in taking into account the ongoing UNDP interventions and future plans. EQ 2: What issues exist in the monitoring system established by ABA/ROLI which might have affected its effectiveness in reporting on KJP progress? Findings on the Strengths of the Monitoring System • ABA/ROLI’s monitoring and evaluation (M&E) system has evolved over the years due to both the KJP’s team and home office efforts. • ABA/ROLI diligently collects, processes, reports, and stores M&E data. • ABA/ROLI’s home office has provided important support to incrementally strengthen the qualitative aspects of monitoring for results. • Following internal assessments by the ABA Evaluation Unit in Washington, DC, the KJP has a designated M&E officer to cover tasks previously shared between the Chief of Party (COP), staff attorneys, and program managers. Conclusions on EQ 2 • ABA/ROLI has diligently worked to improve its M&E system by dedicating more time, home office involvement, and in-country human resources to refine the progress indicators, diversify the data collection methods and track progress towards meeting the program objectives. • Overall, qualitative aspects of progress monitoring remained weak until 2016 as various factors constrained the ability of ABA/ROLI to effectuate, track and verify impact beyond delivery of output required. • The introduction of qualitative elements allowing for “before and after” comparison based on the baseline data generated in 2016 is positive. Recommendations • USAID should continue to encourage the integration of qualitative elements in ABA/ROLI’s M&E plans. For example, ABA/ROLI may have to collect thematic case studies to identify successful results in certain area of assistance (countering violent extremism [CVE], ADR, investment justice) to better describe the contribution and added-value of the KJP’s interventions. This can be done through qualitative story collection and semi￾structured interviews with beneficiaries and non￾beneficiaries. 41 Main Findings Conclusions Recommendations Findings on Weaknesses of Monitoring System • Almost all indicators under Objective 1 are output￾oriented focused on reporting the activities and the delivery of assistance in the form of number of decision-writing manuals developed, number of seminars conducted, and number of resource material provided which are not designed to capture change in the capacity of judges. • The only indicator intended to measure the increase in the quality of decisions (Indicator 1.2.6) is conceptualized as a form of “before and after” comparative analysis to measure change but its implementation raises questions of validity of findings. • Objective 2 Indicators are good to report output but less useful to demonstrate how the KJP’s activities have led to change of capacities either through demonstrable use of the introduced practices or discernible change in the capacity or behavior of beneficiaries. • The validity and reliability of the findings for the first comparative study in July-August 2019 could have been strengthened by engaging outside experts who are more removed from the research activity. • USAID should consider asking implementing partners (IPs) to utilize Outcome Harvest and Complexity Aware Monitoring Approaches to better document the impact of their interventions. For example, Outcome Harvesting—as an evaluation approach more amenable to tracing impact in situation when assistance is offered in a complex environment with changing contextual conditions or when certain intervention needs the contribution of multiple stakeholders to succeed—could have addressed some of the limitations. EQ 3: Going forward, how, and in which areas, should USAID work to increase public trust in the judicial system of Kazakhstan? Findings on working methods • There is growing understanding in the GOK on the need to increase public trust in state institutions and the judicial system but most respondents questioned the credibility of political leadership’s commitment to reforms and the extent to which reforms will genuinely lead to enhanced independence and accountability of judicial system. • Although judicial independence and accountability systems remain incomplete, there are positive trends which may facilitate further programmatic work to increase public trust. • Expectations of court openness and transparency among civil society and the general public are increasing and can be leveraged constructively to push for more reforms and increase public trust in the judicial system. -- Recommendations on working methods • Due to questionable political will for broad rule of reforms, USAID should continuously monitor and reassess political developments periodically in the design and implementation stage of the next judicial sector program. • TA in the areas agreed with justice institutions appears to be more of a “safe instrument” but the future judicial program should attempt to be more ambitious in working the High Judicial Council (HJC), SC, AOJ, and UOJ and go beyond a purely technical approach. • USAID should support CSOs in the future program either through a separate component on CSO engagement or through a grant scheme to support CSO research and advocacy efforts for a more transparent and accountable judicial system. 42 Main Findings Conclusions Recommendations • Currently the conditions for civil society organization (CSO) inclusion in programming appear more favorable than five years ago as the GOK has declared its intention to engage civil society in its efforts to respond to public dissatisfaction through the Council of Public Trust but the GOK maintains control of the agenda, process, timing, and sequencing of planned reforms with CSO engagement. • Reformers within the justice system are few and not univocal and resistance to comprehensive reforms is very high among influential actors within and outside judicial system. • USAID’s reputation and the KJP’s diligent performance constitute a valuable asset for future programming in the judiciary. • On -demand TA in the areas where the United States (U.S.) and GOK priorities converge facilitates the results of the implementer but may constrain USAID’s ability for more impactful programming to increase public trust. • On the other hand, many observers warn that donors often unintentionally become part of regime’s “face -lifting” or system maintenance efforts and are not able to work on relevant matters of the judicial system. Findings on Areas of Work • USAID’s support for further modernization of court processes/administration in accordance with national strategies and action plans is welcome by justice system stakeholders. • In accordance with the GOK’s multi -annual strategies and action plans of the justice sector, more work is needed to upgrade and improve the quality of procedural/substantive law in both criminal and civil justice. -- • USAID should consider combining two streams of work in the next program: 1) work with the HCJ and SC on strengthening judicial independence and accountability; and 2) supporting CSO research and advocacy support for a more transparent and accountable justice system in order to increase public engagement and trust in the judicial system. Recommendations on Areas of Work • Support the HJC and SC to upgrade and streamline laws and regulations to strengthen the independence and accountability of judges (i.e., selection, appointment, promotion, evaluation, and dismissal of judges). • Continue the support to the SC multi -annual plans to reform the civil and criminal procedure legislation and court modernization initiatives. • Offer further assistance to the AOJ on improving formative and continuous judicial education programs. This component may focus on institutional strengthening and curricula development in topical areas of law (i.e., human rights protection, cybercrime, CVE, investment disputes, consumer protection). In particular, USAID should consider providing more training in international human rights law and the case law/jurisprudence of international human rights protection bodies (European Union [EU ], Inter - American, and African courts on human rights; United Nations [UN] committees). • Continue the support to the SC plans for expansion of ADR use both in and outside courts through public awareness campaigns and introduction of legal changes with smart incentives/penalties aimed at making the ADR the default mechanism for a larger category of cases. 43 Main Findings Conclusions Recommendations • Despite the KJP’s work, more targeted interventions to strengthen the judicial education and professional competence of judges and the AOJ’s capacity are needed. • Judges need more training on topical areas of law (i.e., international human rights law and their protection, cybercrimes, intellectual property rights, advanced programs on ADR, CVE, intellectual property, consumer protection law). • Exposure to international best practices, peer-to￾peer professional exchange and supporting reform is seen as beneficial by informants within and outside the justice sector. • Capacities of the Union of Judges (UOJ) to demand more independence and accountability for judges (i.e., constructive advocacy efforts, training for judicial ethics, strengthening committees to conduct comparative studies) are currently lacking and would benefit from USAID assistance. • More work is needed in strengthening CSO court monitoring and reporting capacities and using evidence-driven research to educate public demand for more open and independent courts. This would require a significant upgrade in CSO capacities to plan, execute, and disseminate evidence-based research and implement public advocacy campaigns by CSOs on judicial competence, integrity, independence, financial asset disclosure, and avoidance of conflict of interest by judges. -- • USAID should consider CSO engagement in the future judicial program and work to enhance local non-governmental organization (NGO) capacities to conduct research on topics related to judicial independence, judicial competence, impartiality, and integrity. Evidence-driven research and advocacy efforts should be designed to raise public awareness on the functioning of the judicial system and educate public demand on judicial independence and accountability. • Support citizen information campaigns on the court system and judicial reforms—particularly targeting youth—in Kazakh language. 44 ANNEX 3: EVALUATION INSTRUMENTS/ILLUSTRATIVE QUESTIONS FOR KIIS AND FGDS KEY INFORMANT INTERVIEWS (KIIS) – USAID AND ABA/ROLI EQ 1: Extent of Accomplishment/Gaps 1. What are some of the successes of the KJP in working with the Supreme Court, Union of Judges, and Academy of Justice? 2. What type of activities have been more successful (i.e., providing support and advice, capacity building with training and study tours? 3. To what extent was the KJP successful in improving the judges’ capacity? Can you point to examples where the KJP’s interventions resulted in increasing judicial capacity to deliver justice in a fair and timely way? 4. To what extent did the KJP’s interventions enhance the judiciary’s efficiency and public accessibility? Do you have any example that shows the KJP’s accomplishments in making courts more user-friendly, transparent, accessible, and efficient in delivering justice to citizens? 5. To what extent did the interventions improve the investment dispute resolution and judicial response to the CVE-related cases? 6. What interventions did not show sufficient progress and why is this case? To what extent are political developments and justice sector reforms enabled or impeded by the KJP’s work? 7. What were some of the barriers to achieve results? In retrospect, what are the project objectives outlined in the Cooperative Agreement where gaps are still evident? EQ 2: Internal Monitoring Systems of KJP 1. Can you describe the way you measured progress towards stated objectives? 2. Is there any area where results collected were not relevant or did not align well with the stated objectives? 3. How were the data on the USAD standard indicators and KJP program indicators collected, processed, and reported? 4. Did the KJP use any baseline against which to measure performance results? In what areas and what were the accomplishments? 5. Did the KJP have a system in place to ensure the accuracy of the data generated during its interventions? 6. Are there any areas where you feel the KJP’s monitoring systems did not accurately capture the progress? Why is that the case? 7. How was “Monitoring for Results” used to learn lessons and adjust project management and implementation? 8. What could have been done better to monitor and report the KJP’s progress towards accomplishing anticipated results? 45 EQ 3: Future Programmatic Interventions 1. Given the current state of Kazakhstan’s judiciary, where should USAID’s assistance focus in the future? 2. What areas of the judicial system still need to be strengthened? Judicial capacity, lawyers’ skills? court transparency, tackling judicial corruption? 3. What resources need to be allocated and what programmatic changes are necessary to maximize USAID’s future assistance in the justice sector? 4. What methods or work and approaches would work better to improve the justice sector? Should USAID focus more on justice institutions? Is it possible to work with CSOs to increase pressure for reform or should work only focus on judiciary? 5. Is it feasible to combine technical assistance to courts with CSOs’ interventions (i.e., judicial watchdog) under the same mechanism? Cross-Cutting: Donor Coordination Approach and Good Practices 1. How did the KJP work with other donors? Was donor coordination formalized? 2. Are there cases of MOUs or agreements to implement joint activities with other donors? 3. Are there examples of pooling funds or expertise with other donors? 4. What has worked better and why? What are the challenges faced in working with other donors? ILLUSTRATIVE KII AND FGD QUESTIONS JUSTICE INSTITUTIONS: 1. What assistance have you received from the KJP to help you achieve your objectives? How would you rate the KJP’s performance, quality, and timeliness of the assistance? 2. What laws and policies on the judicial system have been drafted and/or adopted with assistance from the KJP? 3. What are the results of the KJP’s work to increase judicial capacity? Can you point to examples in the Supreme Court, lower courts, or Academy of Justice where the KJP’s work improved the capacity of judges? 4. How would you assess the KJP’s work in improving the quality of written judicial decisions, providing resource manuals, and training to judges? 5. What laws and policies on improving access to justice and court efficiency were adopted with assistance from the KJP? 6. What assistance have you received to increase public trust and accessibility in the court system? 7. What are the results of the KJP in promoting ADR? 8. In your opinion, what did the KJP’s assistance to other justice institutions accomplish? 9. To what extent has judicial capacity been improved? What capacity can be almost exclusively attributed to the KJP’s assistance? What still needs to be done? 10. What is your opinion of the other interventions of the KJP in terms of their strengths and weaknesses and what is the overall opinion on the project performance? 11. In your opinion, what has the KJP’s assistance accomplishment in improving the capacity of judiciary to handle investment disputes? 12. In your opinion, what has the KJP’s assistance done to better deal with CVE cases achieved? 46 13. In your opinion, what are the KJP’s achievements to improve public trust in judiciary? 14. What is your opinion on the KJP’s cooperation with other donors? KIIS WITH OTHER DONORS/PARTNERS 1. What was your interaction with the KJP? 2. How would you rate their donor coordination approach? 3. What were the KJP’s results in improving judicial capacity in criminal cases? What about investment disputes, CVE-related cases? 4. In your opinion, what is the state of judiciary at the moment? What are some of the positive developments and challenges? 5. Based on your knowledge of the donor situation in the rule of law, in which areas of the justice sector and judiciary should assistance be focused? 6. Do you think USAID is well positioned to continue its work to improve the judicial system? 7. What type of interventions have better prospects at being successful? Improving the legal framework, training judges, court efficiency interventions, tackling judicial corruption, NGO and citizen engagement in monitoring the judiciary? Focus Group Discussions with Press Secretaries in Courts General Questions 1. What is your role in your court? 2. How does public information work in your institution? 3. Are the court users satisfied with the service they receive? KJP Activities 4. Did you participate in the KJP’s activities designed to improve public outreach and trust in the judiciary? Please describe? 5. How did you benefit from the KJP’s activities? Have your skills been increased and how? 6. As a result of the KJP’s assistance, has your work been positively impacted? If so, how; and if not, why? 7. What can be done to improve the public trust and access to courts? 8. What type of assistance do you need to further enhance the public information and outreach? 9. Would USAID’s assistance make any positive contribution in this regard? 47 ANNEX 4: DATA SOURCES No. Date Respondent(s) Organization Position 1 Dec. 3, 2019 Aigul Berdygulova USAID Central Asia Program Specialist, SPO 2 Dec. 3, 2019 Marina Kozhevnikova USAID Central Asia M&E Specialist, SPO 3 Dec. 3, 2019 Irina Mitrofanova (follow￾up call on Dec. 10, 2019) USAID Central Asia Project Management Specialist, DGO 4 Dec. 3, 2019 Olga Didenko InterNews Media Law Expert 5 Dec. 3, 2019 Marat Bigaliyev InterNews Program Director 6 Dec. 3, 2019 Suleimenov Maidan Arbitration Chamber Chairman 7 Dec,.3, 2019 Assel Duisenova Arbitration Chamber Executive Director 8 Dec. 4, 2019 Bakhitzhan Karamanov Almaty City Court Chair of Criminal Panel 9 Dec. 4, 2019 Kanat Uteubayev Almaty City Court Judge 10 Dec. 4, 2019 Mariyam Karabayeva Almaty City Court Judge 11 Dec. 4, 2019 Irina Fedotova Almaty City Court Judge Coordinator 12 Dec. 4, 2019 Batagoz Dauletova Medeu District Court Judge 13 Dec. 4, 2019 Zhansaya Namazbaeva Turksib District Court Judge 14 Dec. 4, 2019 Gulnara Baigazina Almaty City Bar Lawyer/Mediator 15 Dec. 4, 2019 Valentina Stepanova Center for Mediation & Peacekeeping Mediator 16 Dec,.5, 2019 Tatiana Zinovich Legal Policy Research Center Acting Director 17 Dec. 5, 2019 Zhumageldy Yelubayev KazBar Chair 18 Dec. 5, 2019 Aigerim Koishibayeva KazBar Deputy Chair 19 Dec. 5, 2019 Daniyar Kanafin Almaty City Bar Lawyers 20 Dec. 6, 2019 Andrey Yemelin USAID Nursultan Office Project Management Specialist 21 Dec. 6, 2019 Sholpan Tashmukhambetova (follow up on Dec. 11, 2019) ABA/ROLI Country Director 22 Dec. 6, 2019 Adilet Mukushev OSCE Legal Officer 23 Dec. 6, 2019 Zauresh Shutova Friedrich Erbert Stiftung Head of Office 24 Dec. 6, 2019 Tatyana Kovalyova Court Reporters Guild Chairwoman 25 Dec. 8, 2019 Regina Larioshina H-R Practice Director 26 Dec. 9, 2019 Zauresh Baimoldina AOJ Rector 27 Dec. 9, 2019 Sadykova Kuralai AOJ Head of Civil Law 28 Dec. 9, 2019 Aikumbayev Talgat AOJ Head of Criminal Law 29 Dec. 9, 2019 Djusupov Murat AOJ Director of Retraining and Advanced Training 30 Dec. 9, 2019 Kanat Musin AOJ Post-graduate education 31 Dec. 9, 2019 Aigul Saparova SC Chair of Civil Panel 32 Dec. 9, 2019 Nurlan Sultanov SC Judge of Civil Panel 33 Dec. 9, 2019 Aisulu Slambekova SC Judge of Civil Panel 34 Dec. 9, 2019 Yerbol Umraliyev SC Head of the Apparatus 36 Dec. 9, 2019 Gulshara Baibossynova SC Criminal Cases Secretary 37 Dec. 9, 2019 Gulnara T.Khuanova SC International Department 38 Dec. 9, 2019 Aidos Saduakassov SC Press Secretary 39 Dec. 9, 2019 Jörg Pudelka (Skype Call) GIZ Country Director 40 Dec. 10, 2019 Aigul Kydyrbayeva SC Chair of Investment Panel 41 Dec. 10, 2019 Gulzhan Almagambetova SC Judge of Investment Panel 42 Dec. 10, 2019 Mr Madiyar Balken (follow up on Dec. 11, 2019 SC Judge of Investment Panel 43 Dec. 10, 2019 Denis Shipp SC Judge of Investment Panel 48 No. Date Respondent(s) Organization Position 44 Dec. 10, 2019 Abay Rakhmetulin SC Judge of Investment Panel 45 Dec. 10, 2019 Alma Essymova Nursultan City Court Chair of Criminal Panel 46 Dec. 10, 2019 Asem Ramazanova Nursultan City Court Judge 47 Dec. 10, 2019 Eleonora Ungarsynova Nursultan City Court Judge 48 Dec. 10, 2019 Sairan Alimbayeva Nursultan City Court Judge 49 Dec. 10, 2019 Aiman Maikibayeva Nursultan City Court Judge of Investment Panel 50 Dec. 10, 2019 Botagoz Bayanova Nursultan City Court Press Secretary 51 Dec. 10, 2019 Mussabek T. Alimbekov UOJ Chairman 52 Dec. 11, 2019 Zhukenov Abdrashid SC Chair of Criminal Panel 53 Dec. 11, 2019 Ms Laura A. Agybayeva SC Judge of Criminal Panel 54 Dec. 11, 2019 Yerden Aripov SC Judge of Criminal Panel 55 Dec. 11, 2019 Roza Zhakudina SC Judge of Criminal Panel 56 Dec. 11, 2019 Yelena Maksyuta SC Judge Coordinator 57 Dec. 11, 2019 Liana Kravets ABA/ROLI Staff Attorney 58 Dec. 11, 2019 Aizhan Almussina ABA/ROLI M&E Manager 59 Dec. 11, 2019 Gulmira Bizhanova Legal Media Center Lawyer 60 Dec. 11, 2019 Diana Okremova Legal Media Center Director 61 Dec. 11, 2019 Christopher Campbell Holt Arbitration Court Head Registrar 62 Dec. 12, 2019 Colin McCullough OSCE Head of Political Office 63 Dec. 12, 2019 Gulnara Dussupova EU Program Officer 64 Dec. 12, 2019 Silvia Pietropaolo EU Program Manager 65 Dec,.12, 2019 Konstantin Sokulskiy UNDP Program Analyst 66 Dec. 12, 2019 Eldar Idiyatov UNDP Project Manager 67 Dec. 12, 2019 Gulmira Tulesbayeva UNDP Project Manager 68 Dec. 12, 2019 (Written KII) Elizabeth S. Stong U.S. Bankruptcy Court Eastern District of New York Judge and ADR Consultant to ABA/ROLI 69 Dec. 5, 2019 (Written KII) Aleš Zalar European Center for Dispute Resolution￾ECDR in Slovenia President and ADR Consultant to ABA/ROLI 70 Dec. 17, 2019 (Written KII) Dr. Azar Aliyev University of Halle￾Wittenberg Law Professor and ADR Consultant to ABA/ROLI 71 Dec. 26, 2019 Skype Call Alma Nurshaikova World Bank Rule of Law Project Manager 49 ANNEX 5: LIST OF DOCUMENTS REVIEWED # Title of the Document Date Author 1 Cooperative Agreement AID-176-A-12-00011 (as amended) 16-04-12 USAID 2 Regional Development Cooperation Strategy FY15-FY19 29-05-14 USAID 3 Implementation Plan Year 1 Apr 16, 2012 –Sep 30, 2012 31-05-12 ABA/ROLI 4 Implementation Plan Year 2 Oct 1, 2012 –Sept 30, 2013 09-11-12 ABA/ROLI 5 Implementation Plan Year 4 Nov 1, 2013 – Sep 30, 2014 29-08-13 ABA/ROLI 6 Implementation Plan Year 4Oct 1, 2014 – Sep 30, 2015 29-08-12 ABA/ROLI 7 Implementation Plan Year 5 Oct 1, 2015 – Apr 15, 2016 27-08-15 ABA/ROLI 8 Implementation Plan Year 6 Oct 1, 2016 – Sep 30, 2017 01-09-12 ABA/ROLI 9 Implementation Plan Year 7 Oct 1, 2017 – Mar 31, 2018 30-08-17 ABA/ROLI 10 Implementation Plan Year 8 Oct 1, 2018 – Sep 30, 2019 31-08-18 ABA/ROLI 11 Annual Report: Apr 16, 2012 – Sep 30, 2012 31-10-12 ABA/ROLI 12 Annual Report Oct 1, 2012 – Sep 30, 2013 31-10-13 ABA/ROLI 13 Annual Report Oct 1, 2013- Sep 30, 2014 31-10-14 ABA/ROLI 14 Annual Report Oct 1, 2014 – Sep 30, 2015 31-10-15 ABA/ROLI 15 Annual Report: Oct 01, 2015 - Sep 30, 2016 31-10-16 ABA/ROLI 16 Annual Report: Oct. 1, 2016—Sep 30, 2017 31-10-17 ABA/ROLI 17 Annual Report October 1, 2017—September 30, 2018 31-10-18 ABA/ROLI 18 Draft Performance Monitoring Plan (as amended) N/A USAID 19 Quarterly Report: Year One. Apr 16, 2012 – Jun 30, 2012 31-07-12 ABA/ROLI 20 Quarterly Report: Year One. Oct 1, 2012 – Dec 31, 2012 31-01-13 ABA/ROLI 21 Quarterly Report: Year One. Apr 1, 2013 – Jun 30, 2013 31-07-13 ABA/ROLI 22 Quarterly Report: Year One. Jan 1, 2013 – Mar 31, 2013 30-04-14 ABA/ROLI 23 Quarterly Report Year Two: Oct 1, 2013 – Dec 31 2013 31-01-14 ABA/ROLI 24 Quarterly Report: Year Two. Jan 1, 2014 – Mar 31, 2014 30-04-14 ABA/ROLI 25 Quarterly Report: Year Two. Apr 1, 2014 – Jun 30, 2014 30-07-14 ABA/ROLI 26 Quarterly Report: Year Three. Jan 1, 2015 – Mar 31, 2015 30-04-15 ABA/ROLI 27 Quarterly Report: Year Three. Apr 1, 2015 – Jun 30, 2015 30-07-15 ABA/ROLI 28 Quarterly Report: Year Four. Jan 1, 2016 – Mar 31, 2016 29-04-16 ABA/ROLI 29 Quarterly Report: Year Four. Apr 1, 2016 – Jun 30, 2016 30-07-16 ABA/ROLI 30 Quarterly Report: Year Five. Oct 1, 2017 – Dec 31, 2017 30-01-18 ABA/ROLI 31 Quarterly Report: Year Six. Jan 1, 2018 – Mar 31, 2018 30-04-18 ABA/ROLI 32 Quarterly Report: Year Seven. Apr 1, 2018 – Jun 30, 2018 30-07-18 ABA/ROLI 33 Quarterly Report: Year Seven. Oct 1, 2018 – Dec 31, 2018 30-01-19 ABA/ROLI 34 Quarterly Report Year Seven April 1, 2019 – June 30, 2019 30-07-19 ABA/ROLI 35 Implementation Plan Year 9 Oct, 1, 2019-Sep, 2020 30-08-19 ABA/ROLI 36 Annual Report Oct, 1, 2018-Sept, 30, 2019 30-08-2019 ABA/ROLI 37 Annual PMP Report, October 1, 2015 – September 30, 2016 N/A ABA/ROLI 38 Annual PMP Report, October 1, 2016 – September 30, 2017 N/A ABA/ROLI 39 Annual PMP Report, October 1, 2017 – September 30, 2018 N/A ABA/ROLI 40 Annual PMP Report, October 1, 2018 – September 30, 2019 N/A ABA/ROLI 41 Kazakhstan Systematic Country Diagnostic: A new growth model for building a secure middle class 13-04-2018 World Bank Group 42 Kazakhstan: Tested by Transition 28-11-2019 Chatham House 43 Relations of Council of Europe with Kazakhstan 12-10-2017 Council of Europe 44 Strategy 2050 15-12-12 GoK 45 100 Concrete Steps 15-12-15 GoK 46 Seven Pillars of Justice 28-01-18 Supreme Court 50 # Title of the Document Date Author 47 Normative Regulation of SC No. 3 on introducing amendments and additions to the normative regulation of the Supreme Court of the Republic of Kazakhstan as of July 11, 2003 No. 5 “On the Court Decision” 16-03-18 Supreme Court 48 Normative Regulation of SC No. 11 On Some Issues of Judicial Practice on Administration of Legislation on Terrorist and Extremist Crimes 17-12-17 Supreme Court 49 Annual Training Plan of the Academy of Justice N/A Academy of Justice 50 Doing Business Report for Kazakhstan 2012-2018 World Bank Group 51 Freedom House Country Reports for Kazakhstan 2012-2018 Freedom House 52 World Economic Forum Judicial Independence Reports for Kazakhstan 2012-2018 World Economic Forum 53 World Justice Project Reports for Kazakhstan 2012-2018 World Justice Project 54 Bertelsmann Stiftung’s Transformation for Kazakhstan 2012-2018 Bertelsmann Foundation 55 Transparency International Corruption Perception Reports for Kazakhstan 2012-218 Transparency International 56 President of Kazakhstan Proposes New Political Reforms 31-12-19 Helsinki Times (31 December 2019), 57 First glimpses of Tokayev’s Kazakhstan: The listening state? 17-09-19 Atlantic Council 58 U.S. To Publish New Central Asia Strategy Amid Russian, Chinese Competition, Afghan Threat 14-12-19 Radio Free Europe/Radio Liberty 51 ANNEX 6: FIELD WORK SCHEDULE EVALUATION OF USAID KAZAKHSTAN JUDICIAL PROGRAM FIELD WORK SCHEDULE December 1-14, 2019, Almaty and Nursultan Date Time Organization Name, Position Venue 1.12.19 evening -- Roland Gjoni’s arrival Renion Park Hotel, 66, Kunayev St. 3.12.19 11:00 USAID BRIEF IN Aigul Berdygulova, Program Specialist, Strategy and Program Office Irina Mitrofanova, Project Management Specialist, Office of Democracy and Governance Marina Kozhevnikova, Monitoring and Evaluation Specialist, Strategy and Program Office 41, Kazybek bi, Almaty 3.12.19 12:45 InterNews Kazakhstan (Almaty) Olga Didenko, Lawyer Marat Bigaliyev, Chief of Party in CA Алматы, A15G7M6, ул. 280 Baizakova St., Almaty Towers, South, Floor 5 3.12.19 14:00-14:15 Kazakhstan International Arbitration (Almaty) Ms. Assel Erikovna Duisenova, Executive Director: info@palata.org info@arbitrage.kz; ceo@arbitrage.kz Mr. Suleimenov Maidan Kuntuarovich, Chairman of the Board Timiryazeva 15 B, office 3, Floor 2, ул. Tel. +7 (727) 328-05-42, 313 11 61 Mob.+7 777 2611056 4.12.19 9:00 – 11:00 Judges of the City Court District Courts of Almaty city Mr. Karamanov Bakhitzhan – Chair of Criminal Panel Ms. Karabayeva Mariyam – Almaty City Court Ms. Fedotova Irina – Almaty City Court Ms. Dauletova Botagoz – Medeu District Court Ms. Namazbaeva Zhansaya – Turksib District Court Mr. Kanat Uteubayev – District 2 Almaty City Court, 66, Kazybek bi 4.12.19 14:00 Almaty City Bar Association Gulnara Baigazina – Lawyer (ukadvokat@mail.ru) + 7 777 231 50 30 49/61, Nauyryzbai Batyr, Almaty 8(727)248 20 25 United Center +7 701 566 22 72 for Mediation and Peacekeeping Stepanova Valentina Alexeyevna, Mediator 5.12.19 10:00 Legal Policy Research Center (Almaty) Tatiana Zinovich – Acting Director (tatyana.zinovich@gmail.com) Shevchenko str. 90, 101 office, BC "Karatal" + 7 727 356 03 86 + 7 778 007 17 18 +7 708 807 17 18 52 Date Time Organization Name, Position Venue 5.12.19 12:00 Kazakhstan Bar Association Yelubayev Zhumageldy Sakenovich – Chairman of KazBar (former SC Judge), Member of the International Council under the SC Aigerim Koishibayeva – Deputy Chair of KazBar 160, Dostyk av., 050051, Almaty, BC «D-160», Floor 2, office 207 + 7 702 777 06 38 a.koishibayeva@kazbar.org.kz 5.12.19 14:30 Almaty City Bar Association Daniyar Kanafin – lawyer, (daniyar_kanafin@mail.ru) Cafe +7 707 223 49 94; 5.12.19 16:00 -- Departure to Astana – Nazgul -- 5.12.19 19:50 -- Departure to Astana – Gjoni R. Senkina A. Rixos President Hotel ASTANA 6.12.19 Fri. 9:00 USAID Andrey Yemelin, Project Mangement Specialist, Governanc Hotel, +7 771 790 2098 6.12.19 10:00 ABA/ROLI Representative office in Kazakhstan Sholpan Tashmukhambetova Country Director, sholpan.tashmukhambetova@abaroli.org +7 7172 272318; 18 Turan Ave., Office A209 |Nur-Sultan +7 701 743 06 80 (mob) 6.12.19 12:00 OSCE Adilet Mukushev 10, Beibitshilik, Nursultan city 6.12.19 15:00 Friedrich Ebert Foundation Kazakhstan Shutova Zauresh Yesdauletovna, Head of Office 8 (7272) 91 41 08 +7-701-543-09-39 mob +7-7172-50 46 13 fes.almaty@fes.kz Nur-sultan, Dostyk 1, App.63, not far from show-room KÖRME 6.12.19 16:30-17:00 Kazakhstan League of Court Reporters PF Tatyana Kovalyova - Chairperson of the Board +7 775 100 43 93 Tanyakovaleva333@gmail.com Place tbd 8.12.19 Sun. 20:00 Skype call HR-Practice Kazakhstan Larioshina Regina Alekseyevna, Director 8 (727) 354 3799 157, Abai st., офис 32 office Almaty +7 777 300 68 86 +7 701 711 0752 – whatsapp, emailed 9.12.19 Mon. 10:00-12:00 Academy of Justice M.s Zauresh Baimoldina – Rector of the Academy Ms. Sadykova Kuralai – Head of “Civil Law Disciplines” Mr. Aikumbayev Talgat – Head of “Criminal Law Disciplines” Mr. Djusupov Murat – Director of Retraining and Advanced Training of Judges and Employees of the Judicial System Mr. Kanat Musin – post-graduate education 46, Beibitshilik, Astana + 7 778 788 86 38 – Talgat Aikimbayev 53 Date Time Organization Name, Position Venue 9.12.19 15:00-16:00 Civil Collegium of Supreme Court Ms. Aigul Saparova – Judge of the Civil Panel Mr. Sultanov Nurlan – Judge of the Civil Panel Ms. Slambekova Aisulu – acting Head of the Secretariat 39, Kunayev St. 9.12.19 16:15 Department of Court Administration Mr. Umraliyev Yerbol – Head of the Apparatus (executive office) Ms. Gulshara Baibossynova – Head of the Secretariat on Criminal Cases Ms. Gulnara T.Khuanova – Head of the department of International Relations Mr. Saduakassov Aidos – Supreme Court Press Secretary 39, Kunayev St. 9.12.19 22:00 Skype call GIZ Office Kazakhstan Country Director Jörg Pudelka at giz￾kasachstan@giz.de Most of the time is in Tashkent GIZ Office Astana T +7 7172 58 09 19 F +7 7172 58 03 10 GIZ Office Almaty T +7 727 2 777 008 F +7 727 2 777 007 E joerg.pudelka@giz.de I www.giz.de 10.12.19 Tues. 10:00-11:00 Investment Collegium of SC Ms. Aigul Kydyrbayeva – Chair of the Specialised Investment Panel Ms. Almagambetova Gulzhan – Judge of the Specialised Judicial Board Mr. Madiyar Balken – Judge of the Specialised Investment Panel Mr. Denis Shipp – Judge of the Specialised Investment Panel Mr. Abay Rakhmetulin – Judge of the Panel on Criminal Cases 39, Kunayev St. 10.12.19 14:00 - 14:30 USAID Irina Mitrofanova, Project Management Specialist, Office of Democracy and Governance Tel. call 10.12.19 15:00-16:30 City Court Judges Mr. Barpibayev Tlektes – Chairman of Astana City Court (no show) Ms. Abdrakhmanova Gulnara – Judge (no show) Ms. Essymova Alma – Chairman of the Panel on Criminal Cases Ms. Ramazanova Asem – Judge Ms. Ungarsynova Eleonora – Judge Ms. Alimbayeva Sairan Mamiyevna – Judge Ms. Maikibayeva Aiman – Judge of the Specialised Investment Panel 32, Sauran St. -- -- Other Courts Mr. Kazhymukan Mekemtas – Judge of the Specialised Interdistrict Court on Criminal Cases (no show) Ms. Sharapatova Guldana – Judge of Sary￾Arka District Court (no show) -- 10.12.19 16:30-17:00 City Court Ms. Bayanova Botagoz – Press Secretary 32, Sauran St. 54 Date Time Organization Name, Position Venue 10.12.19 17:15-18:30 Union of Judges Mr. T. Alimbekov – Chairperson of the Union Of Judges 32, Sauran St. . 11.12.19 Wed. 10:00-11:00 Criminal Collegium of SC Mr. Zhukenov Abdrashid – Chair of the Panel on Criminal Cases Ms. Laura A. Agybayeva – Judge of the Panel on Criminal Cases Mr. Yerden Aripov – Judge of the Panel on Criminal Cases Ms. Roza Zhakudina – Judge of the Panel on Criminal Cases 39, Kunayev St. 11.12.19 11:00 – 11:45 Civil Panel of the SC Ms. Maksyuta Yelena – Judge Coordinator 39, Kunayev St. 11.12.19 12:00 - 13:00 confirmed ABA/ROLI Representative office in Kazakhstan Sholpan Tashmukhambetova Country Director, sholpan.tashmukhambetova@abaroli.org +7 7172 272318; 18 Turan Ave., Office A209 |Nur-Sultan +7 701 743 06 80 (mob) 11.12.19 13:00 - 13:30 ABA/ROLI Liana Kravets, Staff Attorney 7 7172 272 318, liana.kravets@abaroli.org 11.12.19 13:30 – 14:00 ABA/ROLI Aizhan Almussina, Monitoring and Evaluation Manager + 7 701 555 62 82, aizhan.almussina@abaroli.org 11.12.19 15:00 – 16:00 Legal Media Center (NS) Gulmira Bizhanova, Lawyer Diana Okremova- Director 15, Otyrar str. Office 504 , right bank, TC Zhanur 11.12.19 16:30 Confirmed International Arbitration Center AIFC Court & International Arbitration Center Almat Igenbaev, AIFC and IAC (NS) (no show) Mr. Christopher Campbell-Holt Registrar & Chief Executive Bibinur, Assistant E: info@aifc-iac.kz Mangilik El Avenue, EXPO, 55 Mangilik El Avenue, Building 15, Level 1, Astana Block C3.1 Level 1 11.12.19 21:30 Supreme Court Mr. Madiyar Balken – Judge of the Specialised Investment Panel, Union of Judges Steering Committee member Hotel . 12.12.19 Thurs. 9:30 – 10:30 OSCE Program Office in Nursultan Colin McCullough, Political Officer 10, Beibitshilik, Nursultan city; + 7 7172 58 00 70 office, + 7 701 727 97 66 mobile 12.12.19 11:00-11:30 Delegation of the European Union to the Republic of Kazakhstan Gulnara Dussupova, Program Officer EU Silvia Pietropaolo, Program Manager 62, Kosmonavtov Street, Z05E9E1, Nur-Sultan, Kazakhstan, Phone: +7 717 297 45 40 (reception); +7 7172 97 45 95 (direct) Mob: +7 777 022 85 85 gulnara.dussupova@eeas.europa .eu http://eeas.europa.eu/delegations /kazakhstan silvia.pietropaolo@eeas.europa. eu 55 Date Time Organization Name, Position Venue 12.12.19 17:00 confirmed UNDP Konstantin Sokulskiy, Program Analyst, Governance Eldar Idiyatov, Project Manager Gulmira Tulesbayeva, Project Manager (konstantin.sokulskiy@und p.org) 14, Mambetov, Nursultan/Astana +7 701 913 50 70 gulmira.tulesbayeva@undp.org +7 701 715 51 14 . 13.12.19 Fri. 11:00 Out Brief USAID USAID Nur-Sultan Office Ryan Weddle – Office Director Andrey Yemelin – Program Management Specialist (Governance) USAID/Central Asia (Almaty) Julie Southfield – Deputy Mission Director Karolyn Kuo – Deputy Director of Democracy and Governance Office Aigul Berdygulova – Program Development Specialist Irina Mitrofanova – Program Management Specialist Marina Kozhevnikova – M&E Specialist 3, Koshkarbayeva, U.S. Embassy, Nursultan city