Submitted to USAID on March 29, 2012 This publication was produced for review by the United States Agency for International Development. EVALUATION Midterm Evaluation of the Armenia Rule of Law Program ARMENIA AED ABA ROLI EVALUATION REPORT Submitted to USAID on March 29, 2012 This publication was produced for review by the United States Agency for International Development. ABA ROLI EVALUATION REPORT MARCH 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Task Order Number AID-111-TO-11-00002 Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Authors: Janet Katz and Ester Hakobyan ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Acknowledgements The evaluation team would like to thank USAID/Armenia management for their support including Bella Markarian and Mariam Gevorgyan, and also thank Steve Kessler of the Department of Justice for his assistance. The team is grateful to the staff of the American Bar Association Rule of Law Initiative (ABA ROLI) Armenia Rule of Law Program for their time and invaluable knowledge, including Ronald Mangum, Gayane Mkrtchyan, Liane Haroyan, Narine Gasparyan and Elizabeth Givens. Marina Sargsyan provided logistical assistance and translations for which we are grateful. Our special thanks go to Kris Merschrod, Susan Kupperstein, Bryan Shipp and Rania Qura at IBTCI for their support in the field and their careful preparation of this final report. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. TABLE OF CONTENTS Executive Summary ......................................................................................................................... i I. Introduction ......................................................................................................................... 1 II. Scope and Methodology ..................................................................................................... 3 III. Findings............................................................................................................................... 6 A. Judicial Reform ...................................................................................................................... 6 B. Legal Profession Reform ...................................................................................................... 11 C. Legal Education Reform ....................................................................................................... 17 D. Human Rights ....................................................................................................................... 21 E. Donor Coordination .............................................................................................................. 23 F. Program Management and Staffing ...................................................................................... 24 IV. Responses to the Evaluation Questions Presented in the Scope of Work......................... 24 V. Conclusions ....................................................................................................................... 32 VI. Recommendations ............................................................................................................. 35 Annex I: Evaluation Scope of Work ............................................................................................. 39 Annex II: ABA ROLI Work Plan ................................................................................................. 47 Annex III: List of Interviews and Contacts................................................................................... 51 Annex IV: Questionnaires and Interview Guides ......................................................................... 53 Annex V: Bibliography ................................................................................................................. 64 Annex VI: Professional Survey .................................................................................................... 65 Annex VII: Conflict of Interest Statements .................................................................................. 69 Annex VIII: Statement of Difference……………………………………………………………71 ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. LIST OF ACRONYMS ABA ROLI - American Bar Association’s Rule of Law Initiative AJRA - Association of Judges of the Republic of Armenia CLE - Continuing Legal Education ECHR - The European Convention on Human Rights ECtHR - The European Court of Human Rights EU – European Union GIZ - Deutsche Gesellschaft für Internationale Zusammenarbeit LOA - Law on Advocacy INL - International Narcotics and Law Enforcement MOJ - The Ministry of Justice OSCE – Organization for Security and Co-operation in Europe OSF - Open Society Foundation PSA – Public Service Announcement SLF - Strategic Litigation Fund SOW - Statement of Work USAID - United States Agency for International Development YSU - Yerevan State University ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page i EXECUTIVE SUMMARY Introduction Armenia has made positive changes and enacted many new laws affecting rule of law since its independence from the Soviet Union in 1991. Armenia has become a signatory to the European Convention of Human Rights and other international treaties. Despite these new laws, however, corruption remains a significant impediment to legal reform. Economic conditions, though improving, still have an effect on Armenia, whose government and justice system are dependent on donors for implementing their programs and carrying out their duties. This is a midterm evaluation of the Armenia Rule of Law Program currently implemented by the American Bar Association Rule of Law Initiative (ABA ROLI). The purpose of the evaluation is to:  Assess the results and outcomes of ABA ROLI and whether they are achieving USAID’s programmatic goals; and  Inform future USAID/Armenia decision-making with regard to similar undertakings.1 The evaluation was based on 12 days of field work from December 6 to December 17, 2011 that included interviews with 35 key informants, two focus group discussions, of which one was in Tsakghadzor, and surveys of legal professionals. The evaluation team also visited legal clinics and law faculties at the Progress University of Gyumri and Gavar State University both located outside of Yerevan. ABA ROLI’s programming is implemented in four main activity areas: judicial reform, legal profession reform, legal education reform and human rights. In general, ABA ROLI has proven to be a success in certain critical areas of legal reform that are untouched and unmatched by other donors and stakeholders. ABA ROLI has become the authority on assisting with the Judicial Code of Conduct, the ethics code for advocates as well as providing technical assistance for legal education and legal clinics. ABA ROLI has been particularly successful providing assistance to continuing legal education (CLE) and other trainings especially on the European Court of Human Rights and trial advocacy. However, donors such as the European Union that are dedicating larger amounts of money toward the judges and advocates may have greater impact on legal reform if ABA ROLI does not coordinate closely and participate in the new Justice Action Plan of the Armenia Government. Key Findings  The Judicial Code of Conduct and the associated Commentaries are among the greatest achievements of ABA ROLI.  The Judicial Department has been resistant to ABA ROLI assistance and suggestions related to the process and outcomes of court monitoring. ABA ROLI has not been as persistent as it should have been with the Judicial Department to obtain results on the court monitoring reports.  ABA ROLI assists the judiciary by publishing essential criminal cases that establish precedents. The judges in the lower courts seem either unable or unwilling to use the 1 Statement of Work, Evaluation of USAID/Armenia Rule of Law Initiative (ABA ROLI) at 1. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page ii precedent system since they either do not know how or are wary due to the possibility of discipline for misapplication of law.  ABA ROLI from its experience knows what kind of training judges respond to and is well positioned to design trainings that the judges will accept.  The amended Law on Advocacy enacted with ABA ROLI assistance has important elements that will affect rule of law reform including: the new School of Advocacy, required CLE, increased legal aid by the Public Defenders; and changes in the Code of Ethics for the advocates. This provides ABA ROLI ample opportunity to provide continuing assistance in these areas.  The institutional development that was a necessary precursor to the Chamber of Advocate’s future has mostly been addressed through ABA ROLI’s persistence.  The Council of Europe is taking the lead on developing the curriculum for the new School of Advocates and without apparent linkage to the scope of the bar examination, which it claims is the responsibility of the Chamber of Advocates and ABA ROLI.  ABA ROLI’s CLE training has had a positive impact on the Chamber of Advocates and ABA ROLI has been instrumental in creating courses for trial advocacy skills, European Court of Human Rights (ECtHR), and other specialized courses. ABA ROLI will have to assert itself into the development of the curriculum by other donors for new advocates and for CLE of the new School of Advocacy to maintain influence and provide its unique expertise.  The Chamber of Advocates has been slow in addressing the issue of the lack of modern skills of advocates in the regions.  Practical skills have been a noted deficiency being taught to law students, and ABA ROLI is the only program addressing this. Although ABA ROLI has developed an extremely useful curriculum for legal writing for several law faculties, some of them are implementing it very slowly.  Legal ethics, considered essential to any law faculty curriculum, and especially in Armenia where corruption is a problem, are still not taught in the law faculties as of the beginning of 2012.  With the training provided by ABA ROLI, the legal clinics are providing excellent and necessary practical experience for law students, but they are reaching only a small number of students.  None of the law faculties visited by the evaluation team are offering practical skills training to the vast majority of their students.  Although the clinics in Gyumri and Gavar have improved their sustainability through additional support from their universities, they are operating precariously even with support from ABA ROLI.  ABA ROLI has provided important support to human rights in Armenia, especially through the program to educate advocates and judges on the ECtHR.  ABA ROLI maintains very good relations with other donors, but the European Union is increasingly taking the lead in legal reform due to its larger budget.  The Justice Action Plan for 2012 to 2016 to be implemented by the Ministry of Justice and a Justice Coordinating Council will offer numerous opportunities for assistance for ABA ROLI or any USAID future rule of law project.  ABA ROLI has been accomplishing a wide range of legal reform projects mostly with Armenian staff and with relatively limited resources. Partners and stakeholders ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page iii unequivocally had positive opinions of the program and are requesting continuing assistance. Key Recommendations  Training should be provided on the Code of Judicial Conduct using the Commentaries.  ABA ROLI is well placed to assist with the amendments of the Judicial Code contemplated in the Justice Action Plan to prevent the practice of informal consultations with higher courts and the issue of discipline for court decisions as “grave” violations of material or procedural law.  ABA ROLI should persist in following up on the results of the court monitoring reports with the Judicial Department to achieve the goals of this program.  To improve relations with the Judicial Department and to exemplify the value ABA ROLI adds to the Department, ABA ROLI should consider a study tour using USAID best practices for leadership of the Department to places where they will be exposed to international standards ABA ROLI should implement the Justice Action Plan’s activities on the precedent system including amendments to the Judicial Code and curriculum design at the proposed justice school.  ABA ROLI should continue to support CLE for advocates and persist with the larger donors to assure that it has a share in participating in curriculum design for both new advocates and CLE for the School of Advocates.  ABA ROLI should emphasize that the Chamber of Advocates provide more services and programs to advocates in the regions.  ABA ROLI should assure that its legal writing course be taught at the undergraduate level to as many students as possible.  A curriculum for an undergraduate course in legal ethics should be developed and supported as a required course in all law faculties with which ABA ROLI partners.  The legal clinics at Gyumri and Gavar should continue to receive support if the universities maintain the corresponding funding.  To avoid overlap or conflict with the programs of donors such as the European Union and ensure formal cooperation, ABA ROLI should consider a Memorandum of Understanding with the other donors to specify mutual tasks and areas of collaboration.  ABA ROLI should analyze the Justice Action Plan and decide where it can intervene and assist.  ABA ROLI should consider retaining a long-term consultant for the judicial reform activity, especially a consultant who is or has been a respected judge to offer greater access to judges and provide valuable expert assistance to ABA ROLI.  Because the future of rule of law programming seems inextricably bound to the Justice Action Plan for 2012 to 2016, ABA ROLI and USAID should use this plan as guidance for future programming. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 1 I. INTRODUCTION Armenia declared independence from the Soviet Union in 1991. The establishment of the justice sector since then has been challenging. Although many new laws affecting the structure of the legal system have been adopted, e.g., two versions of the Constitution, the ratification of the European Convention of Human Rights, and joining the Council of Europe, serious challenges remain. Some areas of the justice sector are showing definite improvement such as the legal profession1 and legal education, for example, which are slowly heading in the right direction.2 The judiciary is better prepared and qualified than in previous years,3 but it appears to be making slow gains in reform as seen in the evidence of influence on decisions and corrupt practices that undermine its independence and the public’s confidence.4 A Freedom House report on Nations in Transit showed that the ratings of democratic progress for Armenia’s judicial framework and independence were unchanged for 2011 with a score of 5.50 out 7, with 1 being the highest. Scores had actually increased (showing less progress) since 2002.5 The December 2011 Transparency International Corruption Perception Index, which ranks countries according to their perceived levels of public-sector corruption, ranked Armenia at 129 out of 182 countries.6 Lack of economic growth affects social conditions and can also contribute to corruption. After economic growth in the years 2002 to 2008, Armenia was hit by a recession in 2009. The borders with Turkey and Azerbaijan are still closed and Armenia is dependent on trade with Russia, whose economic downturn affected Armenia. In 2010, the economy began to recover, however, with a growth rate of nearly five percent.7 USAID Rule of Law Support in Armenia In support of rule of law in Armenia, USAID has initiated and supported programs since 1996. Currently, USAID is sponsoring the Armenia Rule of Law Program implemented by the American Bar Association’s Rule of Law Initiative (ABA ROLI). The program is currently in its third and last year of implementation and will be extended until FY 2013 to include the monitoring of the Parliamentary and Presidential elections. The initiative has programs in four general areas: 1) judicial reform, 2) legal profession reform, 3) legal education reform, and 4) human rights. The program’s objectives are to support judicial independence through assistance with drafting the Code of Judicial Conduct, monitoring of court cases, publishing the criminal cases of the Court of Cassation, and training on the Code of Conduct.8 Other activities of ABA ROLI include assisting the Chamber of Advocates with institutional development and developing the Continuing Legal Education (CLE) program of the Chamber. 1 2008 Legal Profession Reform Index for Armenia at p. 1-2. 2 2007 Legal Education Reform Index for Armenia at p. 1. 3 2008 Judicial Reform Index for Armenia at p. 1. 4 Freedom House Report on Nations in Transit 2011, p. 79. 5 Id. 6 Transparency International Corruption Perceptions Index 2011, Dec. 1, 2011. 7 CIA – The World Factbook, Armenia, https://www.cia.gov/library/publications/the-world-factbook/geos/am.html 8 See ABA ROLI Armenia Rule of Law Program Work Plan for June 1, 2011 – May 31, 2012, p. 4. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 2 ABA ROLI also provided assistance with revisions to the Law on Advocacy,9 has established legal clinics with universities intended to provide practical legal education for students, and is working with the law faculties on curriculum reform.10 Human rights are promoted through training on the European Court of Human Rights for both judges and advocates and support for strategic litigation on significant human rights cases. Structure of the Judiciary Based on the changes in the Judicial Code in 2007, the judiciary in Armenia consists of courts of general jurisdiction (first instance courts) that hear all criminal and civil cases except for administrative cases reserved for the Administrative Court. The first instance courts are based in the marzes (regions) of the country and in Yerevan. There are three types of appeals courts: the Civil Appeals Courts, Criminal Appeals Courts and an Administrative Appeals Court. The Court of Cassation is the highest court and reviews the cases from the Courts of Appeals upon its discretion. In addition, the Constitutional Court reviews the constitutionality of laws in cases brought either by state bodies or individuals. In 2005, the Constitution of Armenia was amended to allow Armenian citizens the right to challenge the constitutionality of legislation directly at the Constitutional Court. The 2005 amendments to the Constitution also established the Council of Justice, whose primary function is to nominate judicial candidates for Presidential appointment and to discipline judges. Nine of the Council of Justice members are elected judges and four Council members are legal academics appointed by the President and the National Assembly. The Chairman of the Council of Justice is the Chairman of the Court of Cassation. Chamber of Advocates The Chamber of Advocates was created from the merger of two bar associations in 2005 based on the authority of the 2004 Law of Advocacy. This Chamber regulates the profession of lawyers, administers the bar examination, and provides CLE for advocates and lawyers. The bar examination has been continuously conducted since 2006. Public Defenders The 2004 Law of Advocacy established the office of the Public Defender that operates as a subunit of the Chamber of Advocates and provides legal aid in criminal cases and some civil cases. 9 Id. at p.12. 10 Id. at p. 21. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 3 II. SCOPE AND METHODOLOGY Evaluation Purpose This is a midterm evaluation of the Armenia Rule of Law Program currently implemented by ABA ROLI. The purpose of the evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the program in order to inform future USAID/Armenia decision-making with regard to similar undertakings.11 For midterm evaluations such as this, as stated in the evaluation scope of work (See Annex A), “the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives.”12 The methodology for the midterm evaluation of ABA ROLI was formulated in response to the key questions in USAID’s Statement of Work. Evaluation questions sought to assess the initiative and its progress at two levels: 1) broader strategic level with a focus on the relevance of the development theory, strategies used and attribution of changes to the USAID intervention, and 2) on a more specific project level with questions about project accomplishments, its contribution to the improved rule of law situation in Armenia, and its support to improved judicial reform, legal profession and education reform. Overall, USAID is interested in understanding rule of law developments in Armenia and outlining some future programmatic directions. As part of the evaluation, a literature review identified the issues that were investigated through a set of structured interview questions applied to key informants from the strategic partners and individuals who could provide evidence. Methodology The methodology utilized by the evaluation team in response to these key points of interest had four major elements: 1. Review of relevant documents: A number of reports, studies, laws, concept papers and other documents were reviewed and analyzed both in preparation for the field work and in the report-writing phase (see Annex E). 2. Key informant interviews and focus groups: These two methods were used as the major sources of information for the evaluation and were carried out during the 12 days of field work from December 6 to December 17. The selection of key informants was based on several factors including document analysis, USAID and ABA ROLI recommendations, and referral by selected interviewees. Thirty-five key informants were interviewed by the team. Major interviewee groups included the following (See Annex C for the complete list of interviewees):  USAID management and ABA ROLI staff; 11 Statement of Work, Evaluation of USAID/Armenia Rule of Law Initiative (ABA ROLI) at 1. 12 Id. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 4  Major project stakeholders in judicial reform, legal profession reform, legal education reform and human rights;  Other donors involved in the sector;  Policy makers. Interview questions were developed by the evaluation team in preparation for the field work and were included in the work plan submitted to USAID for approval. Throughout the field work, adjustments were made in each specific set of questions based upon relevant information as it became available to the team (See Annex D). Additionally, two focus groups were organized during the field work: 1) law students in Gavar Legal Clinic supported by ABA ROLI and 2) experienced advocates involved in the work of the Chamber of Advocate’s Scientific Analytical Center, which is also assisted by the program. 3. Legal professional surveys: Surveys were administered to two groups of advocates￾experienced and young advocates. The survey had four major sections: 1) involvement in ABA ROLI projects and assessment thereof; 2) institutional/legal challenges to the rule of law in Armenia; 3) political challenges to the rule of law in Armenia, and 4) changes required to improve the rule of law in Armenia. Eighteen advocates (12 young and 6 experienced) provided responses to the Professional Survey (See Annex F). 4. Analysis and triangulation of data: Data received through the document review, interviews, focus groups and surveys has been analyzed and cross-compared to provide a sound foundation for the evaluation findings, conclusions and recommendations presented in this report. Limitations of Evaluation Methodology There were certain data limitations of which to take note, including:  The evaluation team had a limited number of interviews with judges or representative of the judicial system. The team managed to conduct three interviews with representatives of judiciary: the Chairman of the Judges Association, the Deputy Director of the Judicial Department and one Administrative Judge. Despite continuous efforts to reach out to more judges both through the Judges Association, ABA ROLI and the U.S. Embassy’s Department of Justice Resident Legal Advisor, this proved to be a challenging task. To overcome this limitation, the team asked judiciary-related questions to other informant groups. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 5  There are two major limitations with the legal professional surveys. The interviewees were not necessarily a comprehensive or selected by random sample because contact was made based on availability and reference. Additionally, since the surveys were self￾administered, some of the responses were either missing or incorrectly marked. However, given that the primary purpose of these surveys was confirmation of information received through other sources, they served this objective and thus references are made to its results in some parts of the evaluation report. Not all of the activities of ABA ROLI have been evaluated, but only those that most directly relate to the questions in the statement of work due to the limited time for the field work and the large number of small activities by ABA ROLI. The evaluation includes field work that took place between December 5 and December 19, 2011 and was conducted by Janet Katz, Legal Consultant and Ester Hakobyan, Monitoring and Evaluation Expert. Gender Considerations One of the key factors examined under the Judicial Reform Index for Armenia, conducted by ABA-ROLI in 2008, is Minority and Gender Representation. Summarizing the situation with relation to this factor, the reports states that “ethnic and religious minorities, as well as both genders, are represented amongst the pool of nominees and in the judiciary generally” (emphasis added).13 In 2008, female judges comprised “less than a quarter of the judiciary” and the situation had not changed significantly by late 2011. Out of 132 judges of the first instant courts, 21 to 28 per cent are women. They constitute 28 and 17 percent of the Appeals and Cassation courts respectively. 14 Women are relatively underrepresented at the Administrative Court where only one of the 16 judges is female. The situation is considerably better within the legal profession where women comprise over 39 per cent (440 out of 1129 advocates registered with the Chamber of Advocates15) of the practicing cadre of advocates. However, it should be noted that ABA ROLI does not track its relevant indicators (e.g., number of judges trained, number of advocates trained, percentage increase of Young Advocate Committee members and others) based on their gender. The only data for which we were able to find gender disaggregation was provided as part of the reporting on clients served by Gavar and Gyumri legal clinics and hotlines. According to the information provided in ABA ROLI’s April￾June 2011 Quarterly Report, 95 female and 88 male beneficiaries were served by the clinic, while hotline calls were answered for 68 female and 59 male beneficiaries. These numbers further support the hypothesis that program benefits are equally distributed between men and women. \ To further promote the involvement of women in this field, however, it is recommended that some kind of system is introduced into the PMP that would ensure both availability of gender 13 ABA ROLI Judicial Reform Index for Armenia January 2008, p. 18. 14 www.courts.am 15 Administrative data provided by the Chief of staff of the Chamber of Advocates. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 6 disaggregated data for all relevant indicators and responsive action in the event the data reveals any adverse changes. III. FINDINGS The findings are organized based on the structure of the ABA ROLI Year Three Work Plan, which focuses on the following areas:  Judicial reform;  Legal profession reform;  Legal education reform; and  Human rights. In addition to the findings section, specific answers to the overall USAID Evaluation Questions (both general and specific to ABA ROLI) can be found in the subsequent section titled, “Responses to the Evaluation Questions presented in the Scope of Work.” A. Judicial Reform Overview of ABA ROLI’s Activities in Judicial Reform ABA ROLI has been working with the judiciary either through the Association of Judges of the Republic of Armenia (AJRA or Judges Association) or through the Judicial Department, an administrative arm of the courts. The goal of ABA ROLI’s initiatives in this field is to increase the independence, transparency and professionalism of the judges, while increasing the judiciary’s knowledge of European Court of Human Rights (ECtHR) decisions and the provisions of the European Convention on Human Rights (ECHR). The activities to accomplish these goals include: providing suggested changes to the Judicial Code of Conduct and support with expert advice on the commentaries, monitoring court cases for adherence to international standards, and publishing of the criminal cases of the Court of Cassation. A1: Increased Judicial Accountability and Transparency Judicial Code of Conduct During the first two years of the program, ABA ROLI submitted suggested amendments to the Judicial Department for the Code of Judicial Conduct so that the Code would comply with international standards. This review was based on an assessment of the Code16 conducted by ABA ROLI and these recommendations were incorporated into the Code in 2010.17 With the AJRA, ABA ROLI continued its assistance on the Commentaries that provide guidance on the 16 ABA ROLI, Assessment of the Code of Judicial Conduct of the Republic of Armenia and the Current Mechanisms for its Enforcement, January 2010. 17 ABA ROLI Armenia Rule of Law Program, Annual Report, 2009-2010, p.4. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 7 application of the Code of Conduct with expert reviews.18 The AJRA stated that the assistance on the commentaries by ABA ROLI was invaluable and wanted more expert help. The work on the Code of Conduct Commentaries was so important that the association members interviewed stated, “the Commentaries became an example for other countries.” During the interview with the AJRA, the respondents stated that it has a good relationship with ABA ROLI even though the AJRA admitted that activities were decreasing due to a perceived lack of financial resources available from ABA ROLI. ABA ROLI is planning to print and distribute 500 copies of the Judicial Code of Conduct to judges, advocates, prosecutors and other interested parties.19 ABA ROLI recommended continued training on the Code of Conduct in 2011 to follow-up on the training programs of the first two years of the program. However, the recommendation was not accepted by the Judicial Department, which carried out ethics training on its own.20 The Judicial Department also did not respond to ABA ROLI’s offer of assistance on the ethics code’s enforcement mechanisms, except to state that legislative initiatives were underway.21 Published decisions of the Ethics Commission of the Council of Justice indicate that since the enactment of the Code of Conduct, with ABA ROLI’s support, appropriate application of the Code had increased 25 percent.22 23 Stakeholders outside of the judiciary told the evaluation team their opinion that, despite the resistance of the Judicial Department, if ABA ROLI had not provided any assistance, none of this progress would have been made or a Code of Conduct might not have been passed at all. Stakeholders interviewed during the evaluation did not expect to see immediate compliance with the Code and that the Code is only useful for those who intend to comply. In summary, expectations of application on the part of the profession are low. Court Monitoring ABA ROLI selected nine cases to monitor during the first two years of program implementation, including high profile criminal, civil and administrative cases. The evaluation team compared the court monitoring of the other groups with which ABA ROLI was coordinating and found that it was generally satisfactory. The Organization for Security and Co-operation in Europe (OSCE) and the Open Society Foundation (OSF) also monitored court cases between 2008 and 2010 in close coordination with ABA ROLI to avoid duplication. The methodology of ABA ROLI differed from OSCE and OSF. ABA ROLI monitored cases in their entirety; OSCE and OSF monitored a number of cases but of a short duration. In discussions with the evaluation team, OSCE and OSF considered the combination of the two methodologies complementary and a way 18 ABA ROLI, Assessment of the Revised Commentaries to the Rules of Judicial Conduct of the Republic of Armenia, March 2011. 19 ABA ROLI Armenia Rule of Law Program, Work Plan for June 1, 2011 – May 31, 2012. 20 ABA ROLI Armenia Rule of Law Program, April – June 2011 Quarterly Report, p. 18. 21 Id. 22 Id. at 18-19. 23 The role of the Ethics Commission, the disciplinary body of the Council of Justice, is not very clear, however. Stakeholders also felt that advocates, who would be most likely to witness violations, would be wary of bringing complaints for fear of antagonizing the judges in their cases. Note that the statistics on improved judicial ethics indicators were based on self-assessments and as such are subjective. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 8 to obtain the most data. OSF would have preferred to do monitoring the way ABA ROLI did but did not have the resources to follow cases in that fashion. ABA ROLI's monitoring assessed whether judges were complying with the Code of Conduct, abiding by ECtHR decisions and generally providing due process rights and conforming to international fair trial standards. OSCE and OSF stated, however, that the presence of monitors by all of the monitoring programs, including ABA ROLI’s, especially international monitors, resulted in some immediate changes in the behavior of the judges and that monitoring with generally accepted methodology was the only way to obtain useful data on the issues of human rights and fair trials. An additional issue, related to judicial transparency and independence that was raised consistently by stakeholders and partners of ABA ROLI was the issue of internal influence on the judges by the Court of Cassation during or after court proceedings. Also, judges were being disciplined for a “bad decision” as opposed to misconduct. The Judicial Code allows for the discipline of a judge by the Court of Cassation for an obvious and grave violation of procedural or substantive laws. Although appeal is the usual remedy for what may be a mistake by the judge, this part of the Judicial Code, which has been enforced in the past, has a chilling effect on judges’ independence when making decisions. To avoid disciplinary action, lower court judges have contacted the Court of Cassation or other higher level judges before making a decision. There was an example mentioned during one of the interviews, where a judge of the first instance court announced to the parties that he had called the Court of Cassation and that the Court of Cassation had advised the judge that one of parties would lose. The AJRA commented to the evaluation team that such grounds for discipline interfered with the judges’ independence, and, accordingly, the AJRA asked for assistance from the donor community with changing this practice. Use of New Precedent System A unique system of precedent of cases was introduced in 2006 in the Judicial Code.24 The reasoning of the Court of Cassation and the ECtHR, including the construal of the law, of cases with identical/similar circumstances are binding and parties have the right to invoke such cases as legal argument. Besides assuring some kind of consistency for parties in the courts, the Court of Cassation also would have fewer cases to review, but neither of these circumstances has occurred in the courts to date as will be explained below. Although the opinions differed in intensity, the general view of stakeholders interviewed by the evaluation team was that the new precedent system, which was introduced without a methodical scheme, did not work. One donor said that it appeared that the Chief Justice of the Court of Cassation still wanted many cases to go to the Court of Cassation and that this reduced the 24 ‘[Everyone has the right to invoke…the reasoning of a final judicial act (including construal of the law) of a Republic of Armenia court in another case with identical/similar factual circumstances. Judicial Code of the Republic of Armenia, Art.15 (3). In addition, “[t]he reasoning of a judicial act of the Cassation Court or the European Court of Human Rights in a case with certain factual circumstances (including the construal of the law) is binding on a court in the examination of a case with identical/similar factual circumstances, unless the latter court, by indicating solid arguments, justifies that such reasoning is not applicable to the factual circumstances at hand. Id. at Art.15 (4). ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 9 quality of the cases decided. The Chamber of Advocates’ Strategic Litigation Fund stated that there were no consistent grounds why the Court of Cassation would take one case over another on appeal. There might be a rejection of the case according to the representative of the Strategic Litigation Fund, but that “the decision was not grounded,” and “the Court was not distinguishing cases and its analysis was weak.” Another donor said that the lower courts did not know how to use the system and that there is “no jurisprudence.” One stakeholder said many first instance cases would be a list of citations and a decision at the end without any reasoning, ignoring the cases cited by the advocates. The most extreme opinion among stakeholders interviewed was that of the Deputy Minister of Justice who stated that the precedent system was "terrible" because "the judges would never understand it". Publishing of the Criminal Decisions of the Court of Cassation ABA ROLI prepared and printed 1,500 copies of the Second Volume of the Criminal Cases of the Court of Cassation.25 Because of the relatively new system of precedent adopted in the Judicial Code, this has become extremely important. Publishing cases is also an essential element of transparency, which “has been highlighted as a key ingredient in reducing improper influences and fostering independence.”26 Having previously provided financial support for this project, ABA ROLI wanted to make the information available by having the volumes online and on CDs for those without internet connection. The Judicial Department stated, however, that the judges would not use CDs and would prefer books. A2. Development of Recommendations and Amendments to Strengthen Human Rights and other Relevant Legislation ABA ROLI, in coordination with its INL-funded program (Department of State Bureau for International Narcotics and Law Enforcement), provided expertise on amendments to the Criminal Procedure Code and the Law on Advocacy (LOA) related to speedy trials and the right to free legal aid. The LOA, recently put into effect, includes the additional right to free legal aid by the Public Defender. The Criminal Procedure Code is still being drafted, and the Resident Legal Advisor of the US Department of Justice stated that it is proceeding well. ABA ROLI planned to use the gaps observed during court monitoring as a basis for discussion with the Judicial Department. This was supposed to result in areas for improvement including legislative amendments or proposed changes to judicial practice. During 2011, monitoring reports were prepared by ABA ROLI on three cases for the use of the Judicial Department but received no action on these reports. The Judicial Department, when questioned during the evaluation about the monitoring by ABA ROLI, indicated that it was helpful but complained that it was only done for high profile cases. A3. Increased Knowledge of ECHR Provisions and ECtHR Decisions, Human Rights, Ethics, and New Legal Developments 25 ABA ROLI Armenia Rule of Law Program, Work Plan for June 1, 2011 – May 31, 2012, p. 6. 26 USAID Office of Democracy and Governance, Guidance for Promoting Judicial Independence and Impartiality, Revised Edition, January 2002, p. 33. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 10 ECHR Provisions and ECtHR Decisions Armenia ratified the European Convention on Human Rights in 2001, which now has the same force of law as decisions of the Court of Cassation. ABA ROLI surveyed the incidence of citation of ECtHR cases in judicial decisions between January 2009 and April 2011 in the Court of Cassation and first instance courts. The results showed that the Court of Cassation has “considerably improved the level of the application of the ECHR provisions and the judgments of the European Court in their judicial acts” while less than two percent of citations from six selected first instance courts were to the ECtHR.27 The survey states that it did not address whether the citations were relevant or appropriate, which would have added more value.28 As a result of its survey, ABA ROLI decided to shift resources from training on ethics to training more judicial servants on the ECtHR. A report29 released jointly by Protection of Rights without Borders and Counterpart in November 2011 appears to be duplicative of the ABA ROLI survey in that it also examines the citation of ECtHR cases in Armenian courts. The report, however, did address the legal reasoning behind the citations of the ECtHR, something the ABA ROLI report does not do but should have done to get the best data on the judges’ knowledge in this area. However, both reports note awareness of the ECtHR is generally on the rise for the highest court in Armenia, which sets the precedents for the lower courts. Training on Ethics In the first two years of the project, ABA ROLI trained 60 judges and 80 judicial servants on judicial ethics.30 The goal was to develop the capacity of the Judicial School, but the school “proved a challenging partner.”31 When ABA ROLI asked to assist in this training, the Judicial Department stated it was doing its own training. Currently, preparatory training for new judges and CLE is conducted by the Judicial School under the Judicial Department. Although just established in 2008, stakeholders interviewed did not deem the school a success, based on the selection process of judicial candidates and the poor training by the judges themselves for CLE. An administrative court judge noted that there was no special training for new judges of his court at the Judicial School. Interviews with stakeholders also indicated that the courses taught by higher court judges only used their own cases as examples, which had no relevance to first instance judges. The lectures were described to be very formal. It was also noted that judges refused to do any kind of post-training evaluations. Funding for a new Judicial Academy is going to be provided by the European Union through a program of the Council of Europe along with its other project overseeing the School of Advocates. The new school would have new leadership, though the Law on the Judicial 27 ABA Rule of Law Initiative, Final Report on the Results of the Survey Assessing the Extent to Which the European Convention on Human Rights and the Judgments of the European Court of Human Rights are Referenced in the Judicial Acts of the First Instance Courts of the Republic of Armenia, July 2011, p. 13. 28 Id. 29 Implementation of Precedents of European Court of Human Rights by the Courts of RA, November 2011. 30 ABA ROLI Armenia Rule of Law Program, Work Plan for June 1, 2011 – May 31, 2012, P. 11 31 Id. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 11 Academy is still a draft just arriving at the National Assembly and organization of the school is not established yet. Stakeholders expressed two opinions as to how it should be organized: 1) The Judges Association thought the new law was a way for the Minister of Justice to gain more influence over the school; and 2) The Deputy Minister said that the government and parliament had political responsibility for the judiciary and the school so they should decide how it should be organized. The Judges Association was also adamant that the prosecutors not be included in the school since this would be a conflict. The Deputy Minister of Justice and the Council of Europe claimed that the prosecutors would physically be using the same facility with the Council of Europe stating that the curriculum would be different and separate. The World Bank also agreed that putting the two groups together would be a cost effective approach. The Council of Europe had not adopted a specific core curriculum yet, preferring to wait until the law was adopted, and the World Bank agreed saying that without a law, there would be “no ownership” of the curriculum. The Judges Association also did not want the academy to teach substantive subjects for new judges such as was taught in the law faculty, but only courses on “how to be a judge.” Whether there would be training for CLE outside of the proposed academy was not clear, but there was an assumption at least by the Judges Association that the specialized courses by donors would continue. Another issue was the location and quality of the training of judges in the marzes. Most training is centered in Yerevan, which is a far distance to travel for judges mainly in the south of Armenia for one day of training. Advocates participating in a focus group commented that judges in the regions are poorly prepared and qualified and are in need of CLE. The advocates, as discussed below, also agreed that: 1) advocates in the regions were in need of CLE and 2) the Chamber of Advocates had not provided sufficient programming outside of Yerevan. Donors and stakeholders who had conducted trainings indicated that judges also seemed resistant to training, especially if it does not come from a judge. They would leave trainings before they were finished and go back to work. The Deputy Minister of Justice suggested fingerprinting judges as evidence they have attended trainings. B. Legal Profession Reform Overview of ABA ROLI Activities in Legal Profession Reform ABA ROLI has been working with the Chamber of Advocates since it was created in 2005. The program has a variety of activities with the Chamber including the bar examination, institutional development and CLE, which encompasses training and providing legal resources. The Chamber viewed the most important contributions by ABA ROLI as 1) establishment of the bar examination, 2) assisting with drafting of the Law on Advocacy, and 3) the formation of the Scientific and Analytic Center. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 12 72% 28% 44% 17% 28% 17% Trainings Scientific Analytical Center Chamber Club discussions supported by ABA Development of materials (electronic, textbooks and others) Strategic meetings on institutional structure Other Graph 1: Respondent Involvement in the ABA ROLI Initiatives Graph 2: Respondent Perceptions on the Key Systemic Factors Limiting RoL in Armenia Surveys of Advocates The advocates provided a valuable sample of opinions on issues relating to rule of law and ABA ROLI initiatives. As part of the field work, a survey was administered to 18 advocates of whom six were more experienced members and 12 who had recently been admitted to the bar. ABA ROLI Training Seventy-two percent of the advocates had received some sort of training by ABA ROLI. Among other popular activities supported by ABA ROLI, 44 percent noted involvement in the discussions/round tables initiated by the Chamber Club and 28 percent participation in the Scientific Analytic Group and Strategic Litigation Fund’s meetings. On a scale of 1-5 (5 being the highest), around 56 percent of the advocates surveyed considered trainings provided by ABA ROLI excellent, with the rest scoring the trainings as a 4. Over 75 percent of the respondents considered that they have improved their legislative analysis skills because of ABA ROLI, while 78 percent also improved their practical skills and 67 percent became more engaged in Chamber operations. The most frequently mentioned topics for future CLE included training on ECtHR (50 percent) and training on Advocacy Skills (44 percent), followed by Administrative Procedure (39 percent). The most frequently mentioned topics for future CLE included training on ECtHR and Administrative Procedure (55 percent both), followed by training on Advocacy Skills (44 percent). Corruption, Enforcement and Institutional Weaknesses Issues relating to corruption were at the forefront of the opinions of the advocates, although institutional weaknesses such as poor enforcement of the laws and lengthy delays in the court 61% 17% 39% 17% 17% 0% 20% 40% 60% 80% Poor enforcement by the judiciary Lengthy delays Lack of transparency within the justice system Public apathy ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 13 39% 22% 22% 39% 33% 0% 20% 40% 60% 80% 1 2 3 Political interference from gov decision makers Corruption among gov decision makers Corruption among the judiciary Corruption among prosecutors Graph 3: Respondent Perceptions on the Key Political Factors Limiting RoL in Armenia Graph 4: Respondent Perceptions on the Changes Needed to Improve RoL in Armenia system also rated highly. The last issue, delays, was never raised in individual interviews or focus groups, perhaps because corruption related issues aroused more discussion. Sixty-one percent of all respondents ranked poor enforcement of the laws by the judiciary as either the primary or next to primary key systemic limitation to rule of law in Armenia (and 17 percent). This was followed by concerns over the lengthy delays with the judicial system (over 39 percent). The third problem mentioned by 17 percent of the respondents was citizens’ public awareness of their legal rights. Among the three most important political factors limiting rule of law in Armenia, “Corruption in the Judicial System” and “Political Interference of the Governmental Decision Makers,” were ranked as the two most important influences on the system (39 percent each) followed by “Corruption among Governmental Decision Makers” (22 percent). Of the respondents, 56 percent considered that reducing corruption in the justice system is the most important change in support of improved rule of law in Armenia. More training for judges (28 percent) and more training for judges on ethics (22 percent) were ranked as second important. B1. Helping the Chamber of Advocates Implement its Institutional Development Strategy A recent ABA ROLI study commissioned by USAID32 showed institutional weaknesses of the Chamber of Advocates. These include the lack of involvement of many of its members, human resources limitations at the headquarters of the chamber and communication, monitoring and reporting weaknesses. During interviews, ABA ROLI and the Chamber management and advocates thought most of the issues raised in this study had been addressed. The Chamber has made changes in its organization adding two deputies and two advisors for the President and increasing the staff size from six to 19. The new Law on Advocacy provides that the general meeting of the Chamber 32 ABA ROLI 2011 Review of Chamber of Advocates of Armenia, May 2011. 56% 28% 22% 17% 17% 0% 20% 40% 60% 1 2 3 Reduce corruption within the justice system Increase the training of the judges Increase training of judges on ethics Increase training of advocates Raise public awareness of peoples’ rights ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 14 elects the Chairman, which is a change from the old system when the board selected the Chairman, and now increases the involvement by advocates.33 Advocates commented that some of the lack of involvement could be explained by other factors including busy schedules or apathy of the advocates. ABA ROLI has been working with the Chamber to change the organization to include an executive board that would be more flexible in making decisions but the Chamber has not done this yet. B2. Strengthened Role of Advocates within the Legal System The advocates in interviews and during the focus group expressed the belief that their role within the legal system is strengthened through the ABA ROLI-assisted Scientific Analytic Center, which was started as a Legislative Working Group and expanded to included four subcommittees.34 In interviews, the management of the Chamber of Advocates and the Scientific Analytic Center were of the opinion that the Chamber exerts the most influence through its Constitutional Justice Committee as a source of expertise to the Constitutional Court. In 2011, the committee provided 30 expert opinions to the Court and 90 percent of the expert opinions in the Chamber management’s view were very close to the final opinion of the Court. The focus group of advocates agreed with this view. Law on Advocacy Though delayed, ABA ROLI was able to see a new, amended Law on Advocacy in December 2011 that provides for increased legal aid by the Public Defender and the new School of Advocacy, which should provide better qualified advocates. The amended Law on Advocacy increases the responsibility of the Public Defenders from defending criminal cases to providing aid in all civil cases based on the financial need of the applicant. According to the Public Defender, the public was unaware of this new change and a public campaign was needed to inform them. The Public Defender was unclear if his office would be allowed to hire more defenders for these new duties. Even with the possibility of new defenders, the salaries especially for Yerevan are not high enough to attract quality candidates or to provide incentives for existing defenders. These issues were confirmed in interviews with the donors and stakeholders. The office of the Public Defender also has to look outside the government to sources such as ABA ROLI to make up for gaps in its budget for equipment and materials. B3. Improved Disciplinary Measures and Enforcement Through various forms of technical assistance in the first two years of the project, ABA ROLI was able to increase the appropriate application of the Code of Ethics of the Chamber of 33 Law on Advocacy, Art. 9. 34 ABA ROLI Armenia Rule of Law Year Three Work Plan 2011 – 2012, at p. 13. The Normative Legal Acts Committee provides expert opinions on such acts to the National Assembly, while the European Court of Human Rights Committee takes positions on cases in Armenia involving that. There is also an Internal Chamber Act Committee to help improve internal Chamber regulatory documents such as the charter, bylaws and regulations. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 15 Advocates by 25 percent.35 Interviews and focus group discussions with the advocates did not raise any issues with the Code of Ethics. The new amended Law on Advocacy will have some new measures that will require changes in the Code of Ethics, however, and ABA ROLI will provide technical assistance under its work plan to the Chamber’s committee that will be responsible for these revisions. The Deputy Minister of Justice, a former advocate, thought that the biggest problem with the advocates was related to bribes and ethics. Other stakeholders were more concerned with judicial ethics, although bribes to judges would likely be passed on by advocates. B4. Strengthened Capacity to Administer Effective Continuing Legal Education (CLE) Programs School of Advocacy According to its 2009 to 2012 Work Plan, ABA ROLI’s expected results included the establishment of the School of Advocacy. This includes an initial training for new advocates and CLE of experienced advocates. The new Law on Advocacy would change the current system where law graduates must work for two years before they qualify to take the bar examination. The law does not specify a particular curriculum for trainees but only that it be both theoretical and practical and of at least six months in duration.36 ABA ROLI expects to provide training materials, curriculum and trainer information for the new school once established. Through the Chamber of Advocates, ABA ROLI has already provided more than 20 courses of CLE for more than 300 advocates in the first two years of its work plan.37 Courses include trial advocacy for young advocates, and courses on subjects such as the ECtHR and administrative law. To assess the effectiveness of its trainings, ABA ROLI conducted pre- and post- tests to determine trainees’ increase in knowledge, finding that advocates demonstrated a 52 percent increase in knowledge on substantive questions.38 Complementing the training needs assessment conducted after the second year, these inputs are the usual prerequisites for the effectiveness of training and determining future training. The courses have been conducted by advocates, however, as a precursor to the Chamber taking over its own CLE training since the new Law on Advocacy requires annual CLE training for advocates. The European Union through the Council of Europe is funding the new School of Advocates including the possible refurbishing of the building provided by the Government of Armenia. In an interview with the evaluation team, the Council doubts that such a school will be sustainable with only admission fees. In addition, the Council, will assist with the drafting of the rules of admission, discipline for the school, and rules for CLE. Because the new Law of Advocacy will 35 Id. at p. 18. 36 Law on Advocacy, Art. 45.8. 37 ABA ROLI Armenia Rule of Law Program, Work Plan for June 1, 2011 – May 31, 2012 at p. 20. 38 Id. Award ceremony at trial advocacy course for young advocates at the Chamber of Advocates, Yerevan, December 17, 2011. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 16 go into effect before a building is ready, the Chamber of Advocates will have to rent space to receive the first applicants in spring 2012, according to the Council. Stakeholders raised concerns during interviews that the School would only have a very limited number of spaces for new advocates and when questioned, the Vice Chairman of the Chamber of Advocates would not specify how many places were available. Currently 300 lawyers take the bar examination. The Deputy Minister of Justice claimed that all those who wanted to become advocates (and met the minimal educational qualifications as before) would be admitted to the school. There were varying ideas about what the curriculum would consist of. The Chamber of Advocates anticipated generally that there would be two stages: a section that would be teaching “how to be an advocate” through “drafting documents and making speeches,” and training on the Code of Ethics of the Chamber. The second stage would be more practical by placing the trainees in law firms for “some months” as interns. After completing this training, the trainee would be allowed to take the bar examination. The new Law on Advocacy contemplates a “simplified procedure” for those who have practiced for five years and want to take the bar examination, but, in the beginning of 2012 , the Chamber of Advocates will decide on what that procedure will be. During an interview, the Council of Europe indicated it had already developed a core curriculum (not shared with the evaluation team), and that they had already started establishing a pool of trainers for the school. Through pilot “train-the-trainers” exercises with international experts, the Council of Europe stated that it had picked “the brightest” local lawyers with both theoretical and practical experience to teach at the school. When asked how the curriculum would be coordinated with the bar examination, the Council of Europe said that the examination “was the responsibility of the Chamber of Advocates and ABA ROLI.” The new Law on Advocacy requires 48 hours of training courses for advocates every two years at the School of Advocates or institutions accredited by the Board of the Chamber of Advocates.39 The Chamber was unsure how much the funding would be for the CLE requirement of the School. The Chamber stated that it anticipates that training programs may be organized outside of the School depending on interests and availability. During the focus group with the experienced advocates, they confirmed that there was a problem with the lack of capacity of advocates, especially the elderly ones, in the regions farther from Yerevan, especially in the south. One advocate stated that there are 116 advocates over 70 years old who were still practicing, most of whom do not use a computer. According to the focus group of advocates, CLE was not going to be sufficient to bring the skills or knowledge of these elderly advocates up to date or the skills of other advocates in these regions who had fallen behind. The number of advocates was not concentrated in these areas far from Yerevan so the Chamber of Advocates had not found it efficient to provide much in the way of training or events to the advocates there. The focus group discussed having young advocates move down to some areas that are actually underserved by advocates. They dismissed it as unrealistic since even if there was no competition, the market was too small to provide enough income and the living conditions were not comparable to Yerevan. The Chamber’s “Friday Club”, a networking and 39 Law on Advocacy, Art. 45.12. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 17 discussion group was preparing to expand its meetings to Vanadzor and to the south, with ABA ROLI’s assistance. Bar Examination The Chamber of Advocates in an interview stated that one of the most successful projects supported by ABA ROLI was the bar examination. Started in 2006, the two hour 100 question multiple-choice examination with rotating questions is “completely transparent” without room for favoritism. The new School of Advocacy in the Chamber’s view, however, would require a more thorough exam, including a written section to test analytical skills. According to the Chamber, this will require resources to develop, which it claims not to have. C. Legal Education Reform Overview of ABA ROLI Activities in Legal Education Reform The 2007 Legal Education Reform Index for Armenia states that, “quality legal education is an essential element in producing legal professionals who can competently represent clients and contribute to the establishment of the rule of law.”40 ABA ROLI has supported legal education by the development of new courses at the law faculties, and by assisting legal clinics currently at three universities in Armenia. It has started a legal clinic association to promote the sustainability of the clinics. Armenia is also a signatory to the Bologna Process since 2005, by which academic degrees are standardized through a credit system and a quality assurance system.41 C1. Sustainable Practical Skills Programs at Law Schools (Legal Clinics) According to the 2007 Legal Education Reform Index for Armenia, professional skills instruction was deficient at the law faculties.42 As part of its efforts to strengthen the quality of legal education, ABA ROLI uses training at legal clinics to develop sustainable practical skills at the law schools. USAID is now the only major donor through ABA ROLI involved in legal clinics, of which there are three: Yerevan State University (YSU), Gavar University and Progress University of Gyumri; all were visited by the evaluation team. The latter two receive about 20 percent of their funding from ABA ROLI while the YSU clinic is totally funded by the university after initial funding in past years by OSF. ABA ROLI has provided the training for the professors and the supervisory attorneys of the clinics at the Gyumri and Gavar clinics including a study tour and a summer school for the clinic students. All printed materials are developed and provided by ABA ROLI. ABA ROLI also funds the mobile clinics that Gavar and Gyumri operate and the salary of the part time supervisory attorney of the mobile clinic. Participation in Legal Clinics The Gyumri legal clinic has only eight to 12 students participating in the clinic at one time because there is only one supervisory attorney available. They come from third and fourth years 40 Legal Education Reform Index for Armenia 2007, ABA ROLI, at i. 41 The Bologna Process is a European plan that provides for comparable degrees to facilitate mobility of students and prepare them for careers. To achieve this, it has an “overarching framework” of qualifications consisting of three cycles, a bachelors, masters and doctorate degree. It also includes Standard Guidelines for Quality Assurance in the higher education area. See http://www.ond.vlaanderen.be/hogeronderwijs/bologna/about/index.htm. 42 Id. at p. 29. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 18 of the law faculty or even the LL.M program and have to submit an essay, take a test, and participate in an interview in order to be admitted. The program is very popular and many students apply. The Legal Clinic at Gavar enrolls 15 students out of 60 third year students. Admission is based on grades, a letter of interest, and an electronic test. The director of the Gavar clinic said in an interview that he would extend the clinic to all students if he had more resources. The legal clinic at YSU was originally funded by OSF starting in 1999. By the 2005-2006 school year, they enrolled 21 students, which the director said was an ideal number. In 2007, the clinic was made mandatory with two sections of 55 students, one for civil law and one for criminal law, but this size was unmanageable. After three years, the clinic was made into a graduate program with a smaller enrollment so it could go back to the quality of faculty and students it had previously. Now the clinic will start over again with a select group of undergraduate students. Nevertheless, it will still have the issue of paying staff. ABA ROLI invites the clinic director and students to trainings, but provides no other resources to the YSU clinic. All three clinics provide a semester of substantive training before the students can meet with clients. The students receive training in case analysis and legal writing at Gavar University, though the director did not feel comfortable teaching legal writing and had to consult the authors of the textbook at YSU. At Gyumri, the students were learning how to analyze cases and other practical skills to handle the cases but substantive legal ethics was taught as a graduate course. Moot court was only offered to clinical students. The professor in charge of the clinic stated that the clinic provided essential practical skills that were never available under the Soviet system. She noted the various and exceptional jobs that the graduates of the program have around Armenia and even abroad. She did not have ideas for how the other students that did not participate in the clinic could get such practical training. The Gavar clinic students stated that they shared ideas with the other students who were not in the clinic. The director of the YSU clinic noted that under the CLE system, a student could graduate without knowing how to write a legal memorandum but the clinic helped shape the personality as a lawyer by exposing them to legal ethics and legal writing. He stated that for the clinic system as a training source, “there is no going back.” The Dean of the YSU Law School had considered other means of including more students including specialized clinics but funding was always an issue. Most Gavar students ended up in the judicial system as assistants as compared to the YSU clinic, where most students became advocates handling civil cases, much as they had done in the clinic. The Gavar students decided not to be advocates, as indicated during the focus group, after seeing Focus group with students of the Gavar State University legal clinic, December 10, 2011. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 19 how hard they worked and noting the increasing number of cases the advocates were taking to court. They were not particularly worried about their careers as the clinic students seemed to get jobs. Sustainability of Legal Clinics The legal clinics have reached ABA ROLI’s goal of receiving 80 percent of their funding from their universities.43 The portion of funding that ABA ROLI provides for the Gavar and Gyumri clinics covers the salaries of the supervisory attorney, the mobile clinics and training materials. The funding of the clinics by the universities is nevertheless precarious and the 80 percent will depend on future budgets. The funding received from ABA ROLI is considered essential, along with the technical support. All personnel associated with the Gyumri and Gavar clinics, including the students, were highly appreciative of ABA ROLI and particularly noted the Summer School provided by ABA ROLI as essential training. Support to the Legal Clinical Association ABA ROLI established the Legal Clinical Association to support sustainability and share technical expertise and experience among its members. The association was fairly new to the directors of the clinics and although they had a positive view of it, they did not have much experience with it. The director the clinic at YSU was very positive about it because it had given him a chance to interact with clinics in Russia and Poland. He thought a key function of the association should be to maintain minimum standards for the clinics. The director had been on a 2009 trip to the United States where he had four months training at University of California at Berkeley. He said this allowed him a unique view of how clinics should be run because before he had only observed Russian clinics. Street Law Program ABA ROLI developed a Street Law program where the clinical students taught 2,000 children about human rights.44 All of the clinics directors noted that they now participate in Street Law training at schools around Armenia and consider it a success. ABA ROLI provided the “train the trainers” sessions to the clinic students and materials to begin the process of transferring this program over to the Legal Clinical Association. C2. Law Schools Develop and Adopt New Curriculum ABA ROLI has developed three new courses, legal writing, intellectual property and human rights. In conjunction with YSU professors and an American law professor, ABA ROLI developed a legal writing curriculum and published the third edition of the legal writing textbook. The ABA ROLI Year-Three work plan suggests that that four law faculties now have legal writing as a requirement and 170 students have taken the course.45 43 This 80 percent figure from the ABA ROLI work plan was confirmed by the directors of the clinics and the president of Gavar University. See also ABA ROLI Armenia Year Three Work Plan, p. 23, n. 23. 44 Id. 45 Id. at 25. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 20 For the intellectual property course, ABA ROLI supported the development of the course, which is an elective course in the LLM program at YSU and is incorporated into the civil law curriculum at the Gavar and Gyumri law faculties.46 ABA ROLI intends to publish and distribute the supplement to the textbook published in 2009, but YSU stated that the book was not completed by the faculty at the time of the evaluation. ABA ROLI trained professors on a new human rights curriculum during the first two years of its current program. 150 students at YSU took the course using a textbook developed by YSU professors.47 Interviews did not raise any other issues with the intellectual property and human rights courses, which were developed in response to the changes in the law in Armenia such as the signing of international conventions and treaties. Legal Writing Legal writing is the basic course for students in drafting documents necessary in the practice of law and all who were questioned about it from academics to advocates agreed that a course like this should exist. The ABA ROLI work plan notes that they successfully lobbied four universities’ academic councils to make Legal Writing a mandatory for-credit course. 170 students have taken the course.48 Of the three major law faculties visited, only YSU has legal writing as a required undergraduate course, which is important for such a basic course. YSU also has legal writing as a graduate course. Gavar teaches legal writing in its clinic and at the graduate level, while Gyumri does not teach legal writing as an undergraduate course but does at the graduate level. ABA ROLI has also established a legal writing program at the LL.M. level at Open University. All of the LL.M. courses in legal writing are mandatory. General Observations about Legal Education In an interview, the Dean of YSU was appreciative of the contribution of USAID and ABA ROLI over the years, noting the development of textbooks and especially the retraining of the faculty in new curricula and new methods of teaching by ABA ROLI. The three main obstacles he still faced, however, in improving legal education were 1) not being able to choose his own students, 2) the environment including classroom conditions and 3) the remuneration of the faculty. The Dean complained that the low quality of the secondary education in Armenia contributed to his problems. Also, because the Ministry of Education selected the students, he ended up with a “grey average” of students. In her interview with the evaluation team, the president of Gavar University agreed with these problems. The Dean at YSU stated that only public demand would cause the Ministry of Education to change the system of selection as most parents and students just looked at YSU as a prestige institution without caring about the actual preparation of the students. He admitted to a problem with corruption at YSU and had attempted to control it by allowing the faculty to take outside jobs to “improve their social condition.” As stated by the Dean at YSU, although 70 percent of the students at YSU were employed in legal positions after graduating, fewer were choosing to go into the state system. The Dean stated that it was a reality that optimistic young lawyers could get co-opted by the system and that was 46 ABA ROLI Armenia Rule of Law Program Year Three Work Plan, 2011-2012, p. 24. 47 Id. 48 Id. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 21 the reason they were choosing other positions. The President at Gavar was well aware of the corruption in the judicial system awaiting the students, but she thought that values have to start somewhere and her school provided that in her view. All of the three law faculties visited stated that in response to the Bologna Process, they were retraining their professors to some extent in new methods of teaching as opposed to the old Soviet style of lecture and rote memorization. The new methods of teaching seemed to be targeted to the younger professors who were more amenable to change. The Gavar president also noted that there were no text books for the administrative law course that was in demand by the students. Legal Ethics The 2007 Legal Education Reform Index for Armenia found that because the state-required core curriculum did not require ethics instruction, it was not offered by any of the law faculties and this was noted as a deficiency.49 Although it is now taught to a few students in the legal clinics supported by ABA ROLI or as an LL.M course, none of the law faculties visited was teaching legal ethics generally to undergraduates. A senior professor at YSU and other stakeholders who were also law professors specifically spoke about the lack of values taught to the students as opposed to just knowledge since the students might not get another opportunity to learn them once out of school. The YSU professor thought the teaching of legal ethics would help mitigate the corruption at the school. D. Human Rights Overview of ABA ROLI Activities in Human Rights In addition to previously described human rights activities in this evaluation (sections A-C), ABA ROLI’s work plan specifically refers to the following activities: training of advocates and other lawyers on the ECtHR, sponsoring of the Strategic Litigation Fund (SLF) and supporting the mobile (traveling) clinics. There should be a relationship between justice sector reform and human rights protection that still improves project effectiveness and is consistent with existing resource constraints.50 Many of ABA ROLI’s activities appropriately consider human rights to some extent and its activities do promote it fairly effectively. Providing legal aid to the underserved through legal clinics and traveling clinics and funding training and litigation for cases that promote human rights laws add value to a rule of law project. Human rights are promoted with the dual goal of providing practical skills training for law students, a valuable result. D1. Expanded Application of Available Legal Mechanisms for the Protection of Human Rights Training on ECtHR for Advocates and other Lawyers ABA ROLI has supported training on the ECtHR for 300 advocates, public defenders and representatives of NGOs in the first two years of its work plan and is continuing this through the 49 Legal Education Reform Index for Armenia 2007, ABA ROLI, p. 29. 50 P. Alston; M. Robinson, The Challenges of Ensuring the Mutuality of Human Rights and Development Endeavors, in Human Rights and Development, P. Alston, M.Robinson, Eds; Oxford Univ. Press,. 2005; p. 12. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 22 third year.51 OSCE had also cooperated on some of this training and stated that it was very successful. Although ABA ROLI has taken the lead in this area, stakeholders other than OSCE did not refer to the program during interviews suggesting that more outreach on the program is warranted. Strategic Litigation Fund As stated by the director of the Strategic Litigation Fund (SLF), the fund with ABA ROLI coordination, selects 15 cases incorporating important human rights issues to appeal to the Constitutional Court and the ECtHR. The director of the fund in an interview said that the advocates handle the cases on a “semi-pro bono basis driven by public good.” The SLF provides training to the advocates handling the cases and support in research. The SLF has seen achievements on the cases it has selected. The advocates obtain decisions from the Constitutional Court when they have been unsuccessful previously at the Court of Cassation and saw tangible results because the Constitutional Court declared the law at issue in the cases void. Though the director of the SLF thought the ECtHR was important, he warned about “putting all the eggs in one basket,” now that there was only one ECtHR judge in Strasbourg hearing the cases coming from Armenia. He suggested using UN conventions whose procedures had a shorter timetable. D2. Access to Legal Services ABA ROLI’s expected results was 1,200 in person consultations during the 2009-2012 work plan and exceeded that with more than double that amount at 2,768 free consultations.52 According to interviews with the directors, the legal clinics are currently providing access to justice with the Gyumri clinic serving 90 clients a year and providing more consultations through the hot line. The mobile (traveling) clinic goes out four times a month up to 60 kilometers away. The Gavar legal clinic handled 300 clients a year with an additional 80 to 100 clients served by the mobile clinic. Though not directly supported by ABA ROLI, YSU’s clinic handled about 200 cases a year and the overwhelming majority of them are civil matters as the citizens have had no other place to go for assistance. D3. Increased Public Awareness about Human Rights ABA ROLI prepared and broadcast Public Service Announcements (PSA) about human rights relevant to the public including one on children’s rights and another on the right to certain criminal legal aid by the Public Defender’s office. A focus group conducted by ABA ROLI found that 90 percent of participants did not know that the Public Defenders’ Office provides free legal aid to arrestees until having watched ABA ROLI’s PSA.53 The Public Defender did not raise any issues with the PSA related to criminal legal aid that ABA ROLI supported. As told to 51 Id. at p. 27. 52 Id. at p. 23. Because this number could be based on consultations provided by legal clinics that no longer exist, the evaluation team could not independently verify the number of consultations provided by the legal clinics since 2009. Interviews with the directors indicated that legal clinics and mobile clinics were in demand and they would like to expand to provide even more services. 53 Id. at p. 31. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 23 the evaluation team, the Public Defender, though, did state a need for publicity of the aid available under the new amended Law on Advocacy. E. Donor Coordination ABA ROLI has cooperated with many donors and stakeholders, who all expressed positive opinions to the evaluation team. OSF and OSCE coordinated on court monitoring and OSCE also partnered on training with ABA ROLI. The World Bank, which is a large donor to the judicial system, relied heavily on the Judicial Reform Index of ABA ROLI, as did OSCE. ABA ROLI is being outspent by the European Union (EU) on similar activities and the EU does not appear to coordinate as closely as other donors. As an example, the Council of Europe said in an interview that it had over four million Euros for a project that focuses solely on the School of Advocacy and the planned judicial school.54 Strategic Action Program for Judicial Reforms The European Union is now a major participant in legal reform and through its European Neighborhood Policy has helped Armenia develop a Strategic Action Program for Judicial Reforms with an accompanying Justice Action Plan. Justice Action Plan Succeeding the Action Plan for 2009 to 2011, the new one is to be implemented from 2012 to 2016. Originally, the Ministry of Justice (MOJ) was the implementer, but several entities are not under the authority of the MOJ such as the judiciary and the Chamber of Advocates. To overcome this issue, it is anticipated that the new Action Plan will be approved by the President, and the Minister of Justice will coordinate a Justice Sector Coordinating Council that will, in turn, establish a donor coordinating body that “can be mobilized in an organized and consistent manner in support of the priorities identified by the Council.”55 The Justice Action Plan has numerous elements for implementation either initially through studies or by amending laws providing ample opportunity for ABA ROLI to participate. The Justice Action Plan, for example, has a component that will amend the Judicial Code to eliminate discipline for grave violations of laws and to prevent the informal consultations between judges. This will have considerable impact on judicial independence, according to the judges association. The Action plan also has activities to address the issue of non-advocate lawyers (lawyers who are graduates of the law faculty but who are not members of the Chamber of Advocates and may still practice civil law) who are not regulated. The Action Plan also notes that the Administrative Court has a backlog of cases, confirmed by the interview with the administrative judge, and intends to do a study to determine efficient ways to reduce this backlog. In an interview, ABA ROLI staff suggested a mediation/settlement project that ABA ROLI could initiate for the administrative court. 54 See http://www.coe.int/t/dghl/cooperation/capacitybuilding/projects/access_justice_armenia_en.asp for more details on this project. 55 Draft Strategic Action Program for Judicial Reforms, rev. 110701. One donor noted that if the judges believed the President, as opposed to the Minister of Justice, was making policies a priority, then they were more likely to concur in those policies. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 24 Funding for the Justice Action Plan The Deputy Minister of Justice stated that the first discussions for funding and support for the Justice Action Plan would be with the European Union and the World Bank. The second round of discussions would be with USAID and GIZ. F. Program Management and Staffing ABA ROLI has a hard working, knowledgeable and enthusiastic Armenian staff. The evaluation team did not observe any problems with management but in interviews, the staff requested a long term consultant for judicial programs, especially a judge, perhaps retired. According to the staff, this would give the project more influence with other judges and the Judicial Department. One issue is the fragmented nature of the program. Although the work plan is divided into four areas of assistance, some areas have diverse and disjointed programs. The assistance to the Chamber of Advocates encompasses five activity areas with varying levels of management complexity from successful support on the Law on Advocacy to the Friday Club that promotes networking. Human rights support is carried on in various activities without defining what particular human rights are important for a rule of law program. It is unclear if human rights are limited to those related to fair trials or any that may be the subject of a case in the courts or even further from general rule of law issues. Some funding of small activities appeared to be ad hoc to partners and when the partners were rejected from funding by ABA ROLI, the reason was not always clearly communicated. During an interview, the Judges Association stated that it wanted to know the focus of ABA ROLI so it could know what would be funded. IV. RESPONSES TO THE EVALUATION QUESTIONS PRESENTED IN THE SCOPE OF WORK A. General USAID Armenia Questions 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? ABA ROLI has been successful in achieving most of its expected results, demonstrated as follows:  The training on the ECtHR has raised awareness of the Court of Cassation, and for advocates, provided knowledge for enforcement of the ECHR through the Strategic Litigation Fund. For additional information see A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments; D1. Expanded Application of Available Legal Mechanisms for the Protection of Human Rights.  The Chamber of Advocates is becoming more sustainable, organized and participatory with the addition of new staff and rules on the election of the chairman. It also is a force for promoting legal reform through the Scientific and Analytic Committee and the Strategic ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 25 Litigation Fund. The Chamber is providing more training for its own members. For additional information see B1 Helping the Chamber of Advocates Implement Its Institutional Development Strategy; B4 Strengthened Capacity to Administer Effective Continuing Legal Education Programs.  The Law on Advocacy, with assistance from ABA ROLI was passed with requirements for Continuing Legal Education, a new School of Advocacy and an expanded role for the Public Defender. For additional information see B2 Strengthened Role of Advocates within the Legal System.  The legal clinics supported with funds and training by ABA ROLI are successful in providing practical skills to the law students who participate and also providing legal services to the underserved. The Traveling Clinics also were successful and provided services to a broad group with limited resources. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics); D2 Access to Legal Services.  Court monitoring has provided important data on cases in conjunction with other court monitoring projects based on complementary methodologies. For additional information see A1 Increased Judicial Accountability and Transparency. The area with the least success has been the judiciary but there have been results on the Code of Conduct and relations with the judges’ association. The Judicial Department has been less cooperative. The new system of precedent has not been introduced in a methodical manner by the Court of Cassation to allow judges to adapt to it. This has added complications to the training of judges and but also transparency based on a coherent set of case decisions. For additional information see A1 Increased Judicial Accountability and Transparency. 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? Some processes and partnerships have gained certain amounts of sustainability but other processes must be viewed in the longer term and need continued support.  ABA ROLI is trying to transfer publishing of the criminal decisions of the Court of Cassation to the Court since this something for which it should have responsibility after having instituted the precedent system. For additional information see A1 Increased Judicial Accountability and Transparency.  The Judicial Code of Conduct and the associated reforms need continual support to keep moving forward as it is not widely accepted by all elements of the judiciary. For additional information see A1 Increased Judicial Accountability and Transparency.  The Chamber of Advocates is slowly more sustainable with a new strategic plan, and its advocacy school under the Law on Advocacy is supposed to be sustainable with fees. It will still rely on donors for the actual building and for technical assistance for the curriculum. Other activities with the Chamber still rely on assistance such as the Scientific and Analytic Committee, the Strategic Litigation Fund and the Friday Club, all of which cannot be sustained financially. For additional information see B1 Helping the Chamber of Advocates ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 26 Implement Its Institutional Development Strategy; B4 Strengthened Capacity to Administer Effective Continuing Legal Education Programs.  The legal clinics are 80 percent sustainable as planned but will always need some form of outside funding since the budgets of the universities will not be able to provide more assistance. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics).  Most of the courses introduced at the law faculties are still running but may need some intermediate assistance to keep from lagging in terms of training faculty (legal writing) and updating teaching materials (intellectual property). For additional information see C2 Law Schools Develop and Adopt New Curriculum, see Recommendation under Legal Education. 3. What were the main achievements of the program?  The Judicial Code of Conduct and the Commentaries to the Code. ABA ROLI suggested amendments to the Code that were enacted and the Commentaries were praised as extremely useful by the Judges Association. Other stakeholders stated that the Code might never have been enacted without ABA ROLI assistance. For additional information see A1 Increased Judicial Accountability and Transparency.  Awareness and knowledge of European Convention of Human Rights for advocates and judges. The Court of Cassation has increased its awareness and citation of the ECHR after trainings by ABA ROLI. The Chamber of Advocates has used its training by ABA ROLI on the ECHR for strategic case litigation. For additional information see A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments; D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights.  Passage of the Law on Advocacy with requirements of continuing legal education, public defender for civil cases and new school for advocates. For additional information see B2 Strengthened Role of Advocates within the Legal System.  Development of the Chamber of Advocates with better organization, the Scientific and Analytic Committee and Strategic Litigation Fund, and more responsibility for its own CLE. For additional information see B1 Helping the Chamber of Advocates Implement Its Institutional Development Strategy; B4 Strengthened Capacity to Administer Effective Continuing Legal Education Programs.  Two new legal clinics including traveling clinics and development of new curriculum for law faculties. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics). 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? Most interventions were relevant and necessary and were generally requested by the stakeholder or partner. ABA ROLI has been in Armenia since 1996 and is part of organization experienced in rule of law projects in many countries. The development hypothesis seemed tailored to the ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 27 local and institutional problems and political structures. There were problems with the judiciary but other donors experienced this as well. ABA ROLI worked around some of the problems with the Judicial Department by working with the Judges Association but will have to confront these issues with the Judicial Department. See Recommendation under Judicial Reform. 5. How did the implementer perform in terms of project management and how effective was the project leadership? Project management was effective generally except for a suggested change in the area of small activities. The staff was competent, enthusiastic and knowledgeable. 6. What “lessons learned‟ can be provided for future USAID programming in this area?  Human rights should be considered as part of the rule of law activities, not as a separate activity. See Recommendation under Human Rights.  Judicial reform should be accomplished as much as possible as “peer to peer” with judges both for the consultant and for general training. See Recommendation under Judicial Reform.  Small fixed obligation grants should be issued instead of ad hoc funding for small activities. See Recommendation under Management and Staffing.  Following up court monitoring results with the target group are necessary in order to achieve goals of highlighting weaknesses and compelling changes. See Recommendation under Judicial Reform. 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? Smaller activities should be managed under small fixed obligation grants, which in turn will promote sustainability with partners who will have to manage more of their own activities. This will also increase more demand driven activities since the partners will have to request the grants and meet the obligations in order to receive money. See Recommendation under Management and Staffing. 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? The agreement was not provided. Although the impact of rule of law is difficult to assess in the short term, the activities in the program could be measured generally through the methods used by ABA ROLI. The survey of the ECtHR citations by the courts could have gone further to assess actual application. 9. Analyze attribution of project successes to USAID involvement.  The Judicial Code of Conduct and commentaries would not have been done without USAID involvement. For additional information see A1 Increased Judicial Accountability and Transparency. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 28  The Chamber of Advocates as a sustainable organization and the Law on Advocacy would not have happened without help with strategic planning and assistance from ABA ROLI. For additional information see B1 Helping the Chamber of Advocates Implement Its Institutional Development Strategy; B2 Strengthened Role of Advocates within the Legal System.  Legal clinics and curriculum reform are critical areas in which ABA ROLI provided assistance. Practical skills training was a specific deficiency in legal training that the clinics are addressing with the added benefit of providing legal aid to the underserved. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics).  Legal writing, intellectual property and human rights courses were also not offered at law faculties until ABA ROLI developed them as requested by the universities.. For additional information see C2 Law Schools Develop and Adopt New Curriculum. 10. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Any future rule of law program will have to coordinate with the Ministry of Justice and the Justice Action Plan that will dominate rule of law reform for the next few years. The larger donors such as the European Union will also take the lead in areas that ABA ROLI had been working such as the judicial training and training of advocates. Any future implementer will have to be particularly assertive to maintain influence with the partner with which it has had such long relationships. For additional information see D Donor Coordination. B. ABA-ROLI specific Evaluation Questions 1. What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009?  The Judicial Code of Conduct and the Commentaries to the Code. ABA ROLI suggested amendments that were enacted in the Code. The Commentaries, provided with assistance of ABA ROLI that helped interpret the Code, were praised by the Judges Association as extremely useful. For additional information see A1 Increased Judicial Accountability and Transparency.  Awareness and knowledge of European Convention of Human Rights for advocates and judges. The Court of Cassation has increased its awareness and citation of the ECHR after trainings by ABA ROLI. The Chamber of Advocates has used its training by ABA ROLI on the ECHR for strategic case litigation. For additional information see A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments; D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights.  Passage of the Law on Advocacy with requirements of continuing legal education, public defender for civil cases and new school for advocates. For additional information see B2 Strengthened Role of Advocates within the Legal System. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 29  Development of the Chamber of Advocates with better organization, Scientific and Analytic Committee and Strategic Litigation Fund and more responsibility for own CLE. For additional information see B1 Helping the Chamber of Advocates Implement Its Institutional Development Strategy; B4 Strengthened Capacity to Administer Effective Continuing Legal Education Programs.  Two new legal clinics including traveling clinics and development of new curriculum for law faculties. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics) 2. What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary? The judiciary is subject to both external and internal pressure, i.e., from the executive and from the Court of Cassation and the Council of Justice to the lower courts. Currently, the internal pressure of the Court of Cassation and Council of Justice is the most prominent. Their ability to initiate disciplinary actions against judges for “grave violations of material or procedural law” and the lack of transparency of the disciplinary system has a chilling effect on independence. By introducing the Code of Conduct and its commentaries, ABA ROLI has pushed the judiciary to take a significant step toward transparency and independence. For additional information see A1: Increased Judicial Accountability and Transparency; D Donor Coordination. Also, training generally contributes to independence as judges who are not well versed in the law are particularly vulnerable to outside pressure and more likely to accept direction from other judicial officials.56 ABA ROLI has been active in continuing legal education for judges on various topics. For additional information see A1: Increased Judicial Accountability and Transparency; A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments. 3. Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable? Increasing awareness of the ECtHR through training will help the judiciary play a role in making the government more accountable. Also, as noted, continuing legal education is essential for increasing the capacity of the judiciary. ABA ROLI became more proficient at providing targeted training to the judiciary. For additional information, see A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments. 4. What did the project do to ensure proper implementation and enforcement of new or existing laws? As in most countries, the assistance is training for familiarization and help with commentaries if necessary. Further along, there can be court monitoring and a survey of cases to see how the laws are being enforced in practice. 56 USAID, Office of Democracy and Governance, Guidance for Promoting Judicial Independence and Impartiality, Rev. Ed., January 2002, p. 27. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 30 ABA ROLI has been training on the ECHR, of which Armenia is a fairly recent signatory. Also, they have been doing training on the fairly new administrative court for the advocates. To encourage compliance with the new Judicial Code of Conduct, they assisted with commentaries but have been requested to support more. ABA ROLI has a court monitoring project and did a survey of the number of citations of the ECtHR after training. For additional information see A1: Increased Judicial Accountability and Transparency; A3 Increase Knowledge of ECHR Provision and ECtHR Decisions, Human Rights, Ethics and New Legal Developments; D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights. 5. What are the activities aimed to establish mechanisms for oversight of court proceedings? The publishing of criminal decisions and establishment of the Judicial Code of Conduct are intended to make court proceedings more transparent. For additional information see A1: Increased Judicial Accountability and Transparency. 6. How do these mechanisms affect judicial performance? Publishing of judicial decisions deters rulings not based on the law and facts. Compliance with the code of conduct would, in theory, provide impartial hearings without ex parte discussions and more serious forms of corruption. 7. Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector? If yes, please name, if not please mention gaps. Although ABA ROLI has supported the publishing of criminal decisions by the Court of Cassation, this should at some point be transferred over to the Court. The Code of Conduct is only one part of the disciplinary system for the judiciary. The Ethics Committee of the Council of Justice that decides on the cases must have clearer rules and ABA ROLI has been working with them. Although ABA ROLI reported a 25 per cent increase in proper application of the Code of Conduct by the Ethics Commission, there are gaps. Stakeholders believed general compliance with the Code at this time is low but the Judges Association thought the commentaries to the Code were helpful and asked for additional ones. For additional information see A1: Increased Judicial Accountability and Transparency. 8. Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest? Using training in ECtHR provided by ABA ROLI, members of the Chamber of Advocates use this to select important cases to bring to the Constitutional Court or the ECtHR through the Strategic Litigation Fund, also supported by ROLI. This has been a successful endeavor. For additional information see D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 31 9. What has the project accomplished in regard to defending human rights through legal protection? The Strategic Litigation Fund supported by ABA ROLI has been successful in cases brought to the Constitutional Court or to the ECtHR. For additional information see D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights. Also legal clinics and traveling clinics supported by ABA ROLI bring legal aid to the underserved. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics); D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights. 10. How did the project affect legal education in law schools? ABA ROLI is the only program working consistently with the law faculties. The legal clinics it helped established are providing practical skills for law students. The courses in legal writing ABA ROLI developed also increase the skills of law students who still have mostly theoretical training. The courses in human rights and intellectual property ABA ROLI were developed in response to changes in the law in Armenia, such as the signing of international conventions and treaties. For additional information see C1 Sustainable Practical Skills Programs at Law Schools (Legal Clinics); C2 Law Schools Develop and Adopt New Curriculum. 11. Where is the most viable stakeholder support for rule of law reform likely to be found? The Chamber of Advocates is the most active group supporting rule of law reform through its Scientific and Analytic Center and Strategic Litigation Fund, both of which it believes are capable of causing changes in case law. For additional information see B1 Helping the Chamber of Advocates Implement Its Institutional Development Strategy; D1 Expanded Application of Available Legal Mechanisms for the Protection of Human Rights. To a lesser extent, the Judges Association is a more active partner for reform than the Judicial Department. The association is willing to work with ABA ROLI and seeks additional guidance on the Code of Conduct and the Judicial Code. For additional information see A1: Increased Judicial Accountability and Transparency. 12. What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector?  Changes in the judicial code to prevent disciplinary action based on “grave violation of material or procedural law.” See Recommendation under Judicial Reform.  Changes in rules on amendments of decisions and guidelines on writing cases for judges. See Recommendation under Judicial Reform.  More specific commentaries on the Judicial Code of Conduct. See Recommendation under Judicial Reform.  More activities for advocates outside of Yerevan. See Recommendation under Legal Profession ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 32  More practical skills training for law students. See Recommendation under Legal Education. 13. How successfully does the project coordinate with the international community to promote human rights? ABA ROLI has coordinated successfully with OSCE on court monitoring and training on the ECtHR and with OSF on court monitoring. For additional information see A1: Increased Judicial Accountability and Transparency. 14. What recommendations can be made for a more effective, integrated project design?  Human rights does not have to be a separate activity but can be integrated into the other activities. See Recommendation under Human Rights.  Small fixed obligation grants should be issued instead of ad hoc funding for small activities. See Recommendation under Management and Staffing. V. CONCLUSIONS Judicial Reform The Judicial Code of Conduct and the associated Commentaries are some of the greatest achievements of ABA ROLI. The Judges Association was extremely pleased with the assistance and all stakeholders agreed that without ABA ROLI taking this role, the Code would not be as useful or perhaps even have existed. The results of this work will not be evident for years, however, and the stakeholders are of the opinion that the Code of Conduct will be followed initially by a select group of judges who value judicial reform. The Judicial Department has been the most resistant to ABA ROLI assistance, but this could change with the new judicial academy and under the new Justice Action Plan. The European Union will be a dominant player, however. ABA ROLI has not been as persistent as it should have been with the Judicial Department by not following up on the results of the court monitoring report. The court monitoring program, which was successful in highlighting weaknesses for improvement in the fair trial system, did not completely achieve that goal. The Judicial Department after it received the monitoring report has been slow to react and this is frustrating the purpose of the monitoring. The OSCE, in contrast, was able to obtain from the Judicial Department a timetable for compliance with the recommendations in its reports and also had a program for publicizing and training on the issues raised in the monitoring. The system of precedent has not been successful because the Court of Cassation did not prepare the judiciary for this change and has not set any rules for what cases will be accepted for appeal and how the Court’s cases are to be applied by the lower courts. The judges in the lower courts seem either unable or unwilling to use the system since they either do not know how or are wary due to the possibility of discipline for misapplication of the law. This system of analysis and application of cases cannot be learned in a two day CLE course even for experienced judges. Many judges are resistant and that makes the task harder. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 33 The number of judges trained is not necessarily indicative of judges using and applying the learning. ABA ROLI and others are learning from experience what kind of training judges respond to and this is more important that the quantity of trainings the judges receive. Judges only respond to training either by other judges, i.e., “peer to peer training” or extremely qualified experts with relevant training materials. Short intervals of training that do not interfere with work also seem more popular though for those traveling from further away, this can be inefficient. Legal Profession Reform The amended Law on Advocacy is now enacted with ABA ROLI assistance and has important elements including the advocacy school, required CLE, increased legal aid by the Public Defenders and changes in the Code of Ethics for the advocates. These changes will provide continued activity for ABA ROLI for the remainder of the program. The institutional development that was a necessary precursor to the Chamber of Advocate’s future has mostly been addressed through ABA ROLI’s persistence. The new Law on Advocacy also allows for the election of the Chairman and the Board, and this will increase participation by advocates in the organization. The ABA ROLI suggested Executive Committee that would ease decision making has not been created yet. The bar examination that was considered a sustainable activity by ABA ROLI is an issue again under the new School of Advocacy. The curriculum for new advocates is now dominated by the Council of Europe without apparent linkage to the scope of the bar examination. The Chamber of Advocates realizes that the bar examination will have to change but needs more resources for this. ABA ROLI’s CLE training has had a positive impact on the Chamber of Advocates as indicated in the tests and interviews, focus group discussions and surveys done by the evaluation team. CLE for attorneys tends to be demand driven as they look for courses that enhance their ability to service their clients. ABA ROLI has been instrumental in creating courses for trial advocacy skills for advocates, both generally and specifically for the ECtHR. ABA ROLI also provides such support for specialized and newly relevant continuing legal education courses for the advocates that the Chamber of Advocates cannot afford to provide. The Chamber of Advocates assumes that this will continue since advocates will be able to take their required CLE courses either in the School of Advocacy or with other institutions accredited by the Chamber of Advocates. The regions lack advocates with modern skills, especially in the south, and the Chamber of Advocates is slow in addressing this. The number of advocates in the regions is limited and geographically dispersed making it difficult to provide efficient training in the current manner. Legal Education Reform Legal education is an essential element of the future of the legal profession and ABA ROLI is filling a gap that few other projects and donors are addressing. ABA ROLI has shown that it is very effective in developing curricula and providing training to professors. Practical skills were a noted deficiency in the 2007 Legal Education Reform Index and ABA ROLI is the only project addressing this. Although ABA ROLI has developed an extremely useful curriculum for legal writing, the law faculties are implementing it very slowly. YSU now has it as an undergraduate requirement, but the other schools, Gyumri and Gavar, plan to teach it at the graduate level or in the clinic. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 34 The Bologna Process has had an effect on curriculum reform and the increased training of professors in more modern methods of teaching. The administration of the law faculties would prefer to have a more competitive selection process so they can enroll more qualified students, otherwise these educational reforms will be less effective. As noted as a deficiency in the 2007 Legal Education Reform Index, legal ethics is still not taught in the law faculties as of the beginning of 2012. With the training provided by ABA ROLI, the legal clinics are very proficient in their teaching methods and provide excellent and necessary practical experience for law students, but they are reaching only a small number of students. The additional opportunities for improving practical skills such as legal writing and moot court are only offered to law clinic students who are viewed as the best and most deserving students. None of the law faculties visited by the evaluation team are offering practical skills training to the vast majority of their students. The law clinics along with the mobile clinics that ABA ROLI supports are also filling a social need of free legal aid that the Public Defender’s office may not be able to fill given its budget constraints. Although the clinics in Gyumri and Gavar have improved their sustainability through additional support from their universities, they are operating precariously even with the support from ABA ROLI. Human Rights ABA ROLI considers human rights as a separate activity though many of its activities affect human rights as development programs in justice reform should. ABA ROLI has provided important support to human rights in Armenia, especially through the program to educate advocates and judges on the ECtHR. The Strategic Litigation Fund with the Chamber of Advocates is achieving success with cases at the Constitutional Court. The head of the Strategic Litigation Fund pointed out that the ECtHR has its limits and that other remedies should be considered. The term “human rights” is used very loosely in some parts of ABA ROLI’s work plan, however, and it is unclear whether the term relates to human rights beyond those for fair trials or exactly what human rights ABA ROLI would address in its program through legislation or other means. Human rights does not have to be a separate activity as it contributes to the loss of focus. Donor Coordination ABA ROLI maintains very good relations with other donors. The European Union, however, is increasingly taking the lead in legal reform due to its larger budget. Many of the Council of Europe’s projects affect ABA ROLI’s partners and projects, such as the Chamber of Advocates, the School of Advocacy and the bar examination, and future training of the judges. The Justice Action Plan, however, will offer opportunities for assistance for ABA ROLI or any USAID future rule of law project, under a coordinating body led by the Ministry of Justice. Management and Staffing ABA ROLI has been accomplishing a wide range of legal reform projects mostly with Armenian staff and with relatively limited resources compared to the Council of Europe. The program fills important gaps that are not addressed by other donors. Partners and stakeholders unequivocally had positive opinions of the program and wanted more projects and continuing assistance. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 35 Due to its longstanding relationships with partners and its extended presence in Armenia, ABA ROLI has developed many activities, some of lengthy duration. The Chamber of Advocates, the Judges Association and other partners, however, have become dependent on ABA ROLI for funding of many small activities such as publishing and meetings for which they now have the capacity. VI. RECOMMENDATIONS Judicial Reform Continued Training on the Code of Judicial Conduct: Training should be provided on the Code of Judicial Conduct using the Commentaries, which are the specific examples for compliance with the Code and are proven ways to convey knowledge. The Judges Association would be a suitable partner for ABA ROLI to collaborate on this training. Judicial Code Amendment Support: ABA ROLI is well placed to assist with the amendments of the Judicial Code contemplated in the Justice Action Plan to prevent the practice of informal consultations with higher courts and discipline for court decisions as “grave” violation of material or procedural law. Such amendments could have considerable impact on judges’ independence. Court Monitoring Report Follow Up: ABA ROLI should persist in following up with the results of the court monitoring reports with the Judicial Department to achieve the goals of this program. If the Judicial Department does not respond, ABA ROLI could partner with OSCE or other donors or programs involved with court monitoring to either use their influence or find ways to publicize the results of the monitoring. Improve Relations with the Judicial Department: As a useful instrument to improve relations with the Judicial Department and to exemplify the value ABA ROLI adds to the Department, ABA ROLI should consider a study tour for leadership of the Department. The tour should follow the six principles for organizing effective study tours as outlined in the award winning USAID documentary, “INSPIRING SUCCESS: Rediscovering the Marshall Plan Productivity Program,” available online. Precedent System Support: Although ABA ROLI assists the judiciary by providing the essential criminal cases that are precedents and points out the lack of citation of the ECtHR by the first instance courts, it needs to promote the application of the laws. Many judges do not know how to analyze and apply the cases in the manner prescribed by the Judicial Code. With its previous experience with the Judicial Code, training and development of curriculum, ABA ROLI is the ideal implementer for the Justice Action Plan’s activities on the precedent system. These include assistance on amendments to the Judicial Code for rules on application for appeal to the Court of Cassation, guidelines for judicial decisions and design of a new training course at the judicial school on application of judicial interpretation. Conduct a new Judicial Reform Index within the next year as four years has passed since the last one and the Index is an indicator of 30 separate critical factors useful for USAID and ABA ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 36 ROLI. Other donors and stakeholders also rely on this assessment for information and programming. Legal Profession Reform CLE Curriculum Design: ABA ROLI should continue to support CLE for advocates because its programs have had a positive impact on advocates and funding is considered problematic by the Chamber of Advocates. ABA ROLI should also persist with the larger donors in participating in curriculum design for both new advocates and CLE for the new School of Advocates. ABA ROLI could play an invaluable and constructive role as an advisor to the Chamber of Advocates as the Chamber makes many decisions about the School of Advocacy. Improved Regional Services: ABA ROLI should emphasize that the Chamber of Advocates provide more services and programs to advocates in the regions by providing distance learning at central points with computer facilities, for example. Pursuing Institutional Reform: ABA ROLI should make it a priority to work with the Chamber of Advocates to assure institutional reform with the addition of an executive board to facilitate decision making. Increased Public Awareness on Legal Aid: Based on its previous success with a public service announcement for the Public Defender, ABA ROLI should support the creation of a public service announcement to publicize the new increased legal aid for civil cases available at the Public Defender. Legal Education Reform Increased Legal Writing: ABA ROLI should assure that its legal writing course be taught at the undergraduate level to as many students as possible. More professors need to be trained in legal writing or a program needs to be created to train advocates to teach this subject. This will ensure sustainability. Develop Legal Ethics Course: A curriculum for an undergraduate course in legal ethics should be developed and supported as a required course in all law faculties with which ABA ROLI partners. With its experience in ethics codes and curriculum development, ABA ROLI has the expertise and the influence to persuade the schools to adopt such a program of study. There are faculty members who would support this new curriculum and probably some who would be willing to be trained to teach it. Expanding Legal Clinics: The legal clinics at Gyumri and Gavar should continue to receive support if the universities maintain the corresponding funding. Ideally, the legal clinics should be expanded if funding is provided for proper supervision. Other forms of practical skill building such as moot court should be extended to other students outside of the clinic system with volunteer advocate supervisors, if necessary. A substantive internship program with supervision should be created for those students who do not participate in the clinic program, concurrent with a required legal writing program. At Yerevan State University, which has a larger student enrollment and faculty, specialized clinics such as environmental law, children’s rights or other clinics developed after a training needs assessment would help expand this essential training tool. The more successful private universities such as the French University, Slavonic University and ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 37 American University of Armenia should be engaged in discussions as at least the first step for creating legal clinics at their law faculties. Curriculum Development: With its previous experience in curriculum development, ABA ROLI would be the appropriate program to assist in the creation of an administrative law textbook and the concurrent training for the professors for a course in administrative law. Human Rights Training Advocates on the ECtHR: ABA ROLI has proven itself the expert on training advocates on the ECtHR so it should take responsibility for that part of the curriculum development at the new School of Advocacy. This would require greater outreach on the success of the current program and intensive coordination with donors who will be funding the School. Support for the Strategic Litigation Fund: Support should continue for the Strategic Litigation Fund as it is seeing results in human rights enforcement in the Constitutional Court and advocates are becoming more expert through experience on this project. ABA ROLI should also consider training for advocates for alternatives to the ECHR such as United Nations conventions. Traveling or Mobile Legal Clinics: Traveling or mobile legal clinics should continue to receive support from ABA ROLI. They perform a unique, valuable and cost-effective service utilizing law students and pro bono advocates. If funding is available, they should be expanded as the demand is high for such services. Defining Human Rights: ABA ROLI should be specific on which human rights it will focus on so that it can further develop its expertise and credibility in this area. Those related to fair trial and more specific political rights such as elections and right to assembly are appropriate. Donor Coordination Strengthening Ties with Larger Donors: ABA ROLI’s work plan for the remainder of its program might overlap or conflict with the programs of donors such as the European Union. To insure formal cooperation, a Memorandum of Understanding could be considered to specify mutual tasks and areas of collaboration. ABA ROLI should analyze the Justice Action Plan and decide where it can intervene and assist. Management and Staffing Enhance Judiciary Reform Expertise: A long term consultant for the judicial reform activity, especially a judge, would offer greater access to judges and provide valuable expert assistance to ABA ROLI in this field. Transferring Responsibility to Partners for Small Activities: ABA ROLI should discontinue the ad hoc funding of small activities such as publications and meetings because they appear to disproportionately consume ABA ROLI’s management resources. Small fixed obligation grants combining these initiatives awarded at specific intervals may require less oversight. Future Programming ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 38 Using the Justice Action Plan for Work Programming: Because the future of rule of law programming seems inextricably bound to the Justice Action Plan for 2012 to 2016, ABA ROLI and USAID should use this plan as guidance for future programming. The Justice Action Plan encompasses many activities such as the promotion of an independent judiciary, a strong advocate system, and a general fair trial system in Armenia. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 39 ANNEX I: EVALUATION SCOPE OF WORK Evaluation of USAID/Armenia Armenia-Turkey Rapprochement (SATR), Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA￾ROLI), and the Small Scale Infrastructure Program (SSIP) Summary: USAID/Armenia requires evaluations of the following activities: Armenia-Turkey Rapprochement (SATR), IFES and NDI electoral and political process Associate Awards under the Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA-RO LI), and the Small Scale Infrastructure Program (SSIP). The purposes of this Task Order are to evaluate the success of these projects in their relevant areas and assess effectiveness of these in achieving set programmatic goals and the USAID/Armenia’s strategic objectives. Two of the five planned evaluations are designed as midterm evaluations (Armenia￾Turkey Rapprochement, and ABA-ROLI), while the other three are designed as end-of-project evaluations (IFES, NDI and SSIP). In the case of mid-term evaluations the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives. They will feed information into the future work plans. In the case of end-of-project evaluations, the findings will be used to inform design and development of future projects. Therefore, the evaluations will identify “lessons learned”; assess strengths and weaknesses of strategies and activities performed under these projects; and provide recommendations to USAID for project planning purposes for the next three to five years. The Contractor will seek to capture effective approaches; analyze the utility of performance monitoring efforts; consider respective outcomes and results; and assess the influence of internal and external changes on the achievement of results. Contractor Responsibilities and Projects: The evaluation should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that looks at how successful the programs have been in achieving their set targets, and how effectively USG resources have been used. Additionally, USAID/Armenia would like to measure the sustainability of the project results on respective beneficiaries where applicable and possible. The Contractor shall review each project’s implementation methodology and to the degree possible, verify the results achieved, the relevance of the project in addressing USAID priorities, and to what extent USAID can be accountable for achieving those objectives. Final recommendations to USAID will help improve program outcomes, weigh sustainability factors, and address program relevance as well as cost efficiency and effectiveness. The evaluations will serve to guide how similar projects, approaches and/or work plans can be improved. The evaluations will also validate (or not) the feasibility of the initial designs of the projects and of their respective development hypotheses. SATR ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 40 This two-year activity is in its first year of implementation. The main objective is to promote improved Armenia-Turkey relations by engaging civil society in the reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to government dialogue. The activity is being implemented in a rapidly changing environment of Armenia-Turkey relations. Therefore the appropriateness of the design of the project needs to be explored in addition to the actual project implementation issues. Thus, this evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project. The evaluation should aim at obtaining feedback from stakeholders and project partners both from Armenia and Turkey. CEPPS These are two separate Associate Awards under CEPPS III, implemented by IFES and NDI, both of which are in their third year of implementation and are expected to end in FY2012. Both activities are primarily aimed at improving political processes. The IFES activity supports the strengthening the administration of electoral processes and assists the electoral administration to meet international standards for free and fair elections through its support to the Central Election Commission and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR). The NDI activity aims to improve the ability of citizens to effectively participate in political processes as members of political parties and helps political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and in 2013 respectively. The findings and recommendations from this evaluation will be reviewed for results achieved, and effectiveness of program approaches. Best practices and lessons learned will be identified which will contribute to the Mission’s decisions on future election-related activities. ABA-ROLI This is in its third year of implementation and will end in FY2012. The main objectives are to support curriculum reform in the Judicial School; cooperate with the Judicial Department of the Republic of Armenia for enhancing the judicial reforms; and provide greater access to justice through Law School legal clinics with a particular focus on protection of human rights. The purpose of evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision￾making with regard to similar undertakings. SSIP This is a 27-month project ending in FY2012. The goal of the project is to mitigate the consequences of Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. This project deals with multiple communities and partners in jointly carrying out construction and work with target communities. Executing water projects has been challenging due to different factors, including multiple construction partners, delayed contribution from other donors, weather conditions, etc. However, civil construction has been progressing timely and with very good quality. Project has been modified twice to increase the number of projects (from ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 41 48 to 58), labor days and expand the geographic coverage. The final evaluation will look at few aspects of the program: how the targets have been met vis-à-vis original targets and benchmarks; how the partnership between all parties (IFAD, local NGO Shen, local construction companies, village Mayors, Local Supervisors, etc.) worked; what are mechanisms that would help communities sustain project gains after its completion; did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally. Evaluation Questions: The Contractor shall review and summarize the implementation and results achieved by all five activities to answer the following evaluation questions and additional questions that may be developed by the Contractor after reviewing the provided materials. For ALL projects: 11. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? 12. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? 13. What were the main achievements of the program? 14. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? 15. How did the implementer perform in terms of project management and how effective was the project leadership? 16. What lessons learned‟ can be provided for future USAID programming in this area? 17. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? 18. Did the agreement provide clear and achievable results against which progress and impact could be measured? 19. Analyze attribution of project successes to USAID involvement. 20. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Project-specific questions/Tasks: For SATR:  Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations?  Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.]  How flexible is the project in terms of adjusting activities to the changing political context?  How has this activity been able to build on the advances made under the previous “Days 2 and 3”project? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 42  What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given?  What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project?  What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect?  How appropriate is the decentralized project implementation approach (four local partners with their Turkish counterpart organizations) to the project objectives?  How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? For CEPPS:  How can the projects be better designed in the future to measure impact, given the political situation faced in Armenia? NDI  Have NDI youth and women’s Leadership Training Academies, the NDI Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party memberships and activities, political engagements, political public gatherings, etc.) among youth and women?  Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities.  To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations?  Have NDI/Yerevan Press Club public debates had value for participating political parties?  What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? IFES  Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code?  Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 43  Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms?  Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? For ABA-ROLI: 15. What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009? 16. What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary? 17. Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable? 18. What did the project do to ensure proper implementation and enforcement of new or existing laws? 19. What are the activities aimed to establish mechanisms for oversight of court proceedings  How do these mechanisms affect judicial performance?  Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector?  If yes, please name, if not, please mention gaps. 20. Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest? 21. How does the project support the country’s only Bar Association in helping to play a balancing role within the justice sector? 22. What has the project accomplished in regard to defending human rights through legal protection? 23. How did the project affect legal education in law schools? 24. Where is the most viable stakeholder support for rule of law reform likely to be found? 25. What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector? 26. How successfully does the project coordinate with the international community to promote human rights? 27. What recommendations can be made for a more effective, integrated project design? For SSIP:  What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)?  What are social, economic and other impacts of the project on target communities and beneficiaries?  How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability?  How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 44  How did the partnerships impact the project (timeliness, quality, and cost-benefit)?  In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met?  How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.?  Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented?  What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies?  If a similar activity is considered in the future, what should be changed in the design and technical approach? USAID’S Role in the Evaluation The USAID Mission in Armenia will: • organize a small USAID advisory group to support the Contractor in the implementation of this scope of work; • provide relevant programmatic and budgetary information to the Contractor (some relevant portions of contracts and assistance agreements are attached); • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Develop additional research and evaluation questions as needed based on the development hypothesis and on the above-mentioned evaluation questions; identify informants and stakeholders, samples and/or other relevant data sources. 3. Develop data collection tools based on the best possible methodology in accordance with the evaluation questions and feasibility considerations and provide to USAID prior to commencing field work. 4. Prepare a field work plan. 5. Conduct field research in Armenia. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 45 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports as requested by USAID and submit final reports to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and quantitative sources; and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. To the extent possible, data should come from facts, rather than be based on anecdotal evidence, and conclusions should be based on findings received from multiple sources. Clear, standardized data collection methodology should be described in detail to ensure reliability and consistency of the evaluation findings. Deliverables The Contractor’s deliverables shall include: 1. A written methodology plan (research design and operational work plan). 2. Provide a verbal debriefing at the end of the field work to Mission management and technical teams. 3. Prepare draft evaluation reports (electronic and hard copy) which will analyze data and summarize key findings, conclusions and recommendations. The Evaluation Report shall at a minimum contain 1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work, all evaluation tools, all sources of information. Submit these to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. The Contractor shall submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The final reports will meet the following quality standards: a) The reports will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports shall address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f)Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of people’s opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 46 5. The Contractor will submit Evaluation data to USAID/Armenia along with the final reports for warehousing and future use by the Mission. The data will be in easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Evaluations Timeline 1. Evaluations should be initiated around September-October 2011. The preliminary findings of the evaluations should be submitted to USAID/Armenia immediately after the completion of the field work. 2. The first drafts of the five evaluation reports shall be submitted to USAID/Armenia no later than three weeks following the completion of the field work. (For CEPPS USAID expects to have two separate reports – one on the IFES activity and one on the NDI activity). 3. The final Evaluation Reports shall be submitted no later than two weeks after final comments on the draft evaluation reports are submitted by USAID/Armenia. 4. Once finalized, the contractor is responsible for ensuring that the final approved reports are also submitted to USAID’s Development Experience Clearinghouse within three months of the completion of the reports. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 47 ANNEX II: ABA ROLI WORK PLAN Introduction The American Bar Association Rule of Law Initiative (ABA ROLI) began implementing a rule of law program in Armenia in 1996 and a criminal law program in 1998. This work plan describes the implementation of a midterm evaluation for ABA ROLI’s program from 2009 to 2012, which includes judicial reform, legal profession reform, legal education reform, and human rights. More specifically, ABA ROLI’s objectives are to support judicial independence through assistance with drafting of the Rules of Judicial Conduct, monitoring of cases, publishing criminal cases of the Court of Cassation, and assisting with the drafting and implementation of human rights legislation.58 Legal profession reform includes assisting the Chamber of Advocates with institutional development strategy in order for it to continue with the Qualification Examination, develop the Young Advocates, and a Continuing Legal Education program, along with assistance with revisions to the Law on Advocacy.59 ABA ROLI has also established legal clinics intended to provide practical legal education for students at three universities, and it is working with the law faculties on curriculum reform.60 The work plan proposed for the evaluation includes field work and the drafting of reports to take place between December 5, 2011 and February 22, 2012. Purpose of Evaluation The purpose of the evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision-making with regard to similar undertakings.61 For midterm evaluations such as this, “the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives.”62 Issues to be Addressed The literature review has identified the issues to be addressed and the strategic partners and individuals who could provide evidence and data during the field work. The ABA ROLI Performance Management Plan lists the indicators, units of measurements and targets that will be examined. The evaluation will determine whether ABA ROLI is meeting these targets and whether these indicators are useful in determining the effectiveness of the programs. The USAID Armenia Statement of Work formulates questions that should be answered by all the evaluations and those that should be answered by the evaluation of the ABA ROLI project (Annex I). 58 See ABA ROLI Armenia Rule of Law Program Work Plan for June 1, 2011 – May 31, 2012, p. 4. 59 Id. at p.12. 60 Id. at p. 21. 61 Statement of Work, Evaluation of USAID/Armenia Rule of Law Initiative (ABA ROLI) at 1. 62 Id. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 48 Methodology The research methodology is based on a review of the relevant reports, studies and articles listed below:  Revised Statement of Work for USAID/Armenia Assessment for ABA ROLI;  ABA ROLI Armenia Y2 and Y3 Work Plans with revised PMP;  ABA ROLI Armenia Quarterly Report January - March 2011;  ABA ROLI Armenia Quarterly Report April - June 2011;  Assessment of the Commentaries of the Rules of Judicial Conduct, March 2011;  ABA ROLI Judicial Reform Index for Armenia January 2008;  World Bank, Implementation Completion and Results Report on Credit to the Republic of Armenia for Judicial Reform Project, June 2007;  World Bank Judicial Reforms Project 2, Public Perception Survey, December 2009;  ABA ROLI Final Report on Results on Survey on Extent to Which ECHR and ECtHR are Cited in Courts of First Instance in Armenia;  ABA ROLI Legal Education Reform Index for Armenia;  European Higher Education Area Bologna Accords;  Heringa, “European Legal Education: The Maastricht Experience,” November 2010;  ABA ROLI Legal Profession Reform Index for Armenia, December 2008;  Law on Advocates, original; and  Draft revisions of Law on Advocates. Field Work The ABA ROLI Evaluation will include a series of interviews and focus groups during the period of December 5-21, 2011 in Armenia to respond to the questions posited in the SOW. The suggested interviews are listed below and should be scheduled initially through ABA ROLI prior to the arrival of the assessment team. Draft interview outlines for legal educators, legal clinics, court monitors and the deputy minister of justice are included. Proposed Interviewees General  Ron Mangum, ABA ROLI Country Director – first meeting;  Bella Markarian, Governance Sector Specialist, USAID – second meeting;  Steve Kessler, DOJ Resident Legal Advisor;  Deputy Minister of Justice, Mr. Aram Orbelyan, Ombudsman;  Professor Aida Iskoyan. Advocates and Lawyers  Chamber of Advocates;  Scientific Analytic Center Committee of the Chamber of Advocates;  Young Advocates Committee; ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 49  Bar Association of the Republic of Armenia;  AYLA – Armenia Young Lawyers Association;  Public Defenders Agency. Judges and Judicial Administration  Individual Judges according to recommendations of ABA and Judges’ Association;  RA Judicial Department;  Judges’ Association;  Council of Europe Access to Justice in Armenia Project - Gayane Hovhannisyan;  World Bank – Davit Melikyan, Public Sector Management Specialist. Legal Education  Dean Gazik Ghazinyan, Yerevan State University Law Faculty;  Legal Clinics – Yerevan, Gavar, Gyumri;  Other law faculties such as Gavar, Gyumri in association with the visits to clinics. Court Monitoring and Human Rights  OSCE – Court Monitoring;  OSI – Court Monitoring;  Human Rights Public Defender;  Strategic Litigation Fund/Armenian Institute for Development;  Other human rights NGOs working on fair trial issues. Focus Groups The assessment team will organize focus groups with lawyers, law students and young lawyers. Focus groups will comprise six to 12 participants, with those who have participated in ABA ROLI programs and a few who have not. Only three or four major open ended questions (with subparts) will be treated in a one or two hour session. To get as much data as possible, a short questionnaire will also be used especially if some focus group participants are less articulate or quiet. Questionnaires for lawyers and law students are included in Annex II of this proposal. These questionnaires will be translated and there will be a translator attending as well as the local rule of law expert. Some snacks will be provided and equipment such a whiteboard or flipchart paper and pens or pencils will be needed. Lawyers should be both advocates (criminal) and civil. Evidence and Data ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 50 The evaluation team will collect data as available63 from ABA ROLI and government agencies. Also, the team will collect additional laws or relevant drafts of laws, including the latest drafts of the Law on Advocacy and the Rules of Judicial Conduct. Deliverables 1. Draft Work Plan This draft work plan and evaluation design is the first deliverable and is sent to USAID/AM in advance of the first briefing with USAID at which time approval will be sought. 2. Preliminary Results The USAID mission in Armenia will be briefed after field work has been completed in Yerevan with feedback included from the mission, approximately December 20, 2011. 3. Draft Report A draft report, which will analyze data and summarize key findings, conclusions and recommendations, will be completed and submitted to USAID Armenia within three weeks upon completing field work. The format of the evaluation report will follow the standard format for USAID evaluation reports,64 and will not exceed 25 pages, excluding annexes. 4. Final Report Revisions of draft reports incorporating USAID’s comments will be submitted two weeks after receipt of these drafts from USAID, as a final report to USAID/Armenia for acceptance. 63 Data could include for example, the number of participants in ABA ROLI and other programs, the most recent court monitoring statistics, the total number of judges and their positions, education and training, the number of advocates and civil attorneys and data relating to the number of students at law faculties and future projections, if available. 64 IBCTI evaluation teams refer to USAID’s “EVALUATION Learning from Experience,” Bureau for Policy, Planning, and Learning and “Checklist for Assessing USAID Evaluation Reports” for guidance. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 51 ANNEX III: LIST OF INTERVIEWS AND CONTACTS # Name Organization Position 1. Amiryan David Open Society Foundation Deputy Director for Programming 2. Aproyan Arsen Legal Clinic at Gavar State University Head of Legal Clinic 3. Arustamyan Melanya Chamber of Advocates Board Member, Head of the CA Club 4. Gasparyan Narine ABA ROLI Senior Staff Lawyer Responsible for Judicial Reform 5. Ghazanchyan Masis Public Defender Office Head of Office 6. Ghazinyan Gagik Yerevan State University (YSU), Law School Dean 7. Gevorgyan Mariam USAID Monitoring and Evaluation Specialist 8. Givens Elizabeth ABA ROLI/INL 9. Hakobyan Ruzanna Gavar State University President 10. Haroyan Liana ABA ROLI Staff Lawyer Responsible for Legal Education Reform 11. Harutyunyan Armen YSU Legal Clinic Director 12. Hovhannissyan Gayane Council of Europe Access to Justice Project Project Manager 13. Kessler Steve DOJ Resident Legal Advisor 14. Karapetyan Nora Judicial Department Deputy Director 15. Magnum Ronald ABA ROLI Chief of Party 16. Margaryan Bella USAID Governance Sector Specialist 17. Margaryan Gor Chamber of Advocates Member, Active ABA Trainer 18. Melikyan Davit World Bank Public Sector Management Specialist 19. Mkrtchyan Gayane ABA ROLI Staff Lawyer Responsible for Legal Profession Reform ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 52 20. Ohanyan Victoria Independent Legal Expert 21. Orbelyan Aram Ministry of Justice Deputy Minister 22. Poghosyan Arman Chamber of Advocates, Strategic Litigation Fund Coordinator 23. Poghosyan Artur Administrative Court Judge 24. Sahakyan SIranush Protection of Rights without borders NGO President 25. Sargsyan Hrachik Association of Judges of the Republic of Armenia (AJRA) President 26. Stambultcyan Anahit Legal clinic in Gyumri, Progress University Head of Clinic, Chair of the Legal Department 27. Silvanyan Maria Organization for Security and Co￾operation in Europe Legal Advisor 28. Vanoyan Ishkhan Legal clinic in Gyumri, Progress University Mobile Clinic Attorney 29. Yayloyan Tamara Legal clinic in Gyumri, Progress University Supervising Attorney 30. Yengibaryan Vahe AJRA Executive Director 31. Zohrabyan Ara Chamber of Advocates Deputy President ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 53 ANNEX IV: QUESTIONAIRRES AND INTERVIEW GUIDES Sample Survey for Law Students  What do you see as the three biggest obstacles to promoting rule of law in Armenia?  Political interference from governmental decision makers (executive and legislative)  Corruption among governmental decision makers (executive and legislative)  Political will among governmental decision makers (executive and legislative)  Corruption among the judiciary  Corruption among prosecutors  Lack of laws  Poor enforcement of the laws  Poor enforcement of judicial decisions  Lengthy delays within the judicial system  Citizen’s public awareness of their legal rights  Lack of transparency within the justice system  Lack of ability to access the justice system and the courts  Inability to protect human rights and civil liberties  Inability to resolve contract disputes  Inability to resolve property rights disputes  Public apathy  Other (write in)  What are the three most important changes that could realistically be done to promote rule of law in Armenia?  Reduce corruption with the justice system  Increase the training of the judges  Increase training of judges on human rights  Increase the training of the prosecutors  Increase training of advocates  Increase salaries of justice officials  Raise public awareness of peoples’ rights  Provide more support to the Bar Association  Increase regulation of civil lawyers  Provide more support to legal aid clinics  Reform and update the curriculum in the law schools  Made the judicial appointment and selection process more independent  Increase the number of female judges  Increase enforcement of court decisions  Other  Have you participated in any programs of ABA ROLI? ___ yes ___ no ___ Legal clinics ___ Moot court ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 54 ___ Legal writing instruction ___ Intellectual property course Were they useful? ___yes ___ No Which ones? ______________________________________________________ (write in)  Have you participated in any other legal education programs provided by?  Chamber of Advocates  Bar Association  OSCE  British East West Institute  AYLA  US Department of Justice Were any of these programs useful? ____ yes _____ no If yes, in what way?  Increased practical skills such as interviewing clients and organizing cases (legal clinic)  Increased advocacy skills  Increased analytical and writing skills  Helped decide on specialty within legal profession  Satisfaction from helping others with their legal problems (legal clinics) Which ones? _____________________________ (write in)  What additional courses or skills training do you think should be offered at the law faculty? Contracts law Company laws and other business related courses European Court of Human Rights Law Advocacy skills Negotiation skills Insurance law Intellectual property Legal ethics and regulation of the legal profession ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 55 Others _____________________________ (write in)  Do lawyers have an obligation to support rule of law reform? ___yes ___no What should they do to support this? ___________________________ (write in) ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 56 Focus Group Guide – Lawyers 1) How should discipline of advocates be strengthened? Is it important? Has ABA ROLI contributed to the awareness and enforcement? How? (Help with LOA, training) 2) What is the role of the advocate in legal and human rights reform? Pro bono? Advocating for new human rights legislation? Adequate compensation and legislation to support the advocate’s practice 3) What kind of training do advocates need to be properly prepared? What kind have they had? Who did it? Did they participate in training by ABA ROLI? Was it useful and how? If not, why? What kind would they like in the future? Will they be willing to pay for continuing legal education? What should be included in the new School for Advocates? Focus Group Guide – Law Students 1) Is the current curriculum adequate to prepare students for the practice of law? 2) What do they need to prepare? 3) Has the programs provided by ABA ROLI improved the curriculum and increased practical skills? Have they participated in the legal clinic, moot court, Street law, legal writing courses and any other training? Was it helpful? What could be done better? 4) What should be included in the new School of Advocacy? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 57 Interview Questions for ABA ROLI  Names of judges we should see and why  Names of particular advocates or leaders of the legal community and why  Human rights NGOs?  What are the most notable accomplishments of the project? Judicial Accountability  What are the factors hindering?  How will you pick the judges for the self assessment/focus group?  Decisions are public but ABA ROLI does not have access to cases reviewed by the Ethics Committee—does that mean there are cases where there are no decisions. How does that work?  Training on the new Rules of Judicial Conduct  Is the request to shift training funds an indication that most judges have had training on judicial ethics?  How has the leadership of the Judicial School been a challenging partner? Court Monitoring  How do you expect to see the goal “Courts improve their compliance with principles embedded in the Rules of Judicial Conduct?  Did monitoring see whether there was compliance with judicial ethics? (Sometimes violations do not occur in the courtroom!)  Do OSCE and OSI monitor different cases? Do you meet with them to exchange information?  How do you pick the cases?  Why did you decide to use the number of times ECHR ECT HR judgments are cited as an indicator of accountability?  What if there are more economic or commercial cases or they apply the judgments wrongly? Publishing Court of Cassation decisions  In trying to get the JD to publish the decisions online, will there be a way to search the cases or an index or digest?  Would the judges have access to the online version the Chamber of Advocates have developed? Judicial Appointment and Exams  Why did the Judges decide to work with the European donors?  Money?  Same form of judicial system? Strengthening Human rights and other relevant legislation  What kind of legislation? Related to the judicial system? Not free speech or right to assembly, etc. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 58  Activity 1 Expected results: First gaps addressed through amendments, and then ABA ROLI assists with implementation strategies. They are listed in reverse.  But what is the human rights legislation – fair trial or all human rights or enforcing human rights grievances that come before the court?  What is status of amendments of Criminal Procedure Code and Law on Advocacy relating to speedy trial rights and right to free legal aid through Public Defenders Office?  What is the case of Silva Asatryan?  Has the JD been cooperative when discussing areas for improvement observed during court monitoring? Resistant? Revisions to the Judicial Code  What limited set of revisions are envisioned? What areas will they cover? Law on Advocacy  What is the status? Any factors hindering passage?  What needs to be done after passage? Can it be done in the time left in the project?  Case Law data base - different from what the Court puts out? Uses that?  Bar Exam – ever for those who only practice civil law? Bar Association and Civil Lawyers  Do you ever have contact with them?  Why not?  What do think about the issue of non-regulation of civil lawyers?  Should they submit to some form of regulation? Legal Education  What do you think is the ultimate number of students per year that can be enrolled in the clinical education programs?  Is this number sustainable?  What other universities want to participate? Is this feasible? How long does it take to set these up?  Civil legal clinics?  Street law – what are the “human rights” mentioned there?  Legal Writing  Now mandatory – 200 reflects all first year students at 3 universities?  Who teaches it?  Did you do any special training of professors to teach it?  How many professors did you train on the human rights curriculum?  What do you think are the greatest accomplishments of ABA ROLI?  What do you think are the impediments to legal reform?  What gaps do you see need to be filled? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 59 Interview Questions for Deputy Minister Of Justice  What have been the achievements of the rule of law in Armenia in the last 10 years?  What has been the role of this Ministry in these achievements?  What are the three biggest obstacles of achieving rule of law in Armenia?  What can be done to overcome these obstacles?  What specific projects would you like to accomplish in the next five years?  What kind of assistance would you need to do this?  Has ABA ROLI been helpful to you in achieving your goals?  If yes, how?  If not, why not?  Has the judicial ethics training been helpful?  Do you see an effect on the judiciary?  Has the training on the European Court on Human Rights been effective?  Do you keep any data on the effectiveness on any training, including this training? (Can we have it?)  What are the gaps that could be filled?  Are other donors providing assistance that you need? (Judicial appointment and examination reform now with the EU)  What kind of assistance? Is it helpful?  Should there be any changes in the way lawyers are trained? What are they?  Do you think the Chamber of Advocates does its job in regulating its members?  What changes could be made?  What about the other lawyers who are not members of the Chamber, i.e., civil advocates?  Do the prosecutors need more advocacy training?  Does court monitoring have an effect on the judges?  Do you think it is necessary?  Does it help with enforcement of laws?  Does it improve the public perception of the judicial system?  What other things could be done to improve the public perception?  Could ABA ROLI help with this?  Have they helped with this in the past? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 60 Interview Questions for Legal Education  What are the biggest achievements of the law faculty in the last 10 years?  What are the three greatest obstacles affecting legal education in Armenia?  What new courses would you like to see offered at the law faculty?  Is legal ethics offered?  If not, why not?  What new skills training?  Do you think the law students are currently adequately prepared for law practice?  What would it take to do this? Legal Clinics  Do you believe the legal clinics are useful for training law students?  What are the funding plans for the clinic?  Is it possible to increase funding or increase participation? New Curriculum of legal writing  Has it been successful?  Is it a required course? (YSU, yes but in other law faculties?)  If not required, are students taking the course?  Do you have any feedback or data on its effectiveness?  What kind of effect do you see?  Do professors see changes from previous classes of students? Human Rights curriculum  Do you offer such courses?  If not, why not?  (Funding, no one capable of teaching, don’t think necessary)  What would it take to offer such a course at your school? Teaching Methods  Do you believe there should be changes in teaching methods of the law faculty?  How should this be done? For current faculty, for new faculty?  Has the ABA been helpful in making progress in the law faculty?  MOE sets curriculum but has ABA had effect on the curriculum?  Has ABA had an effect on legal education generally?  How has it had an effect?  What has been the most useful?  What has been the least useful? What are the gaps in any of the assistance?  What more assistance would you want? From ABA, from anyone else? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 61 Effect of Bologna Process  What has been the effect on the law faculty?  How has the change in structure, credits and years of education requirements of the Bologna process affected the law faculty?  How has the “quality assurance” aspect of Bologna Process affected the law faculty?  Would you make any changes in the admission process?  Would you like to see any changes in the examination process?  If you wanted to make a change in the curriculum, how do you go about it with the Ministry of Education?  Are there changes you would like to make in the curriculum?  What are the obstacles?  Could ABA ROLI or other assistance help you? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 62 Interview Questions for Court Monitoring Partners  What does your program consist of?  How do you believe these activities affect judicial performance?  Are they sufficient?  Are there gaps?  What data and indicators do you use to gauge effectiveness?  What would you like to see done?  How long do you plan to continue this project?  What is the interaction between you and ABA ROLI, i.e., how do you cooperate?  Is this cooperation effective?  What do think of their program in this area?  Is it effective?  What do you base your opinion on?  Does ABA ROLI coordinate with others in the international community to promote human rights?  Do you feel their activities are effective?  What do you think your program and the ABA ROLI program have accomplished in regard to defending human rights through legal protection?  How long do you think court monitoring will have to go on? Are there other mechanisms that will replace it?  What do you see as the three biggest obstacles to promoting rule of law in Armenia?  What are the three most important changes that could realistically be done to promote rule of law in Armenia?  What are the incentives for legal reform in Armenia?  Is joining the EU an incentive for reform in Armenia? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 63 Interview Questions for Legal Clinics  How many students participate?  How are they chosen?  Is there future capacity planned to include more students?  What kind of training do the students get? From whom?  Do they get any training on legal ethics as part of their training?  Are training materials provided?  What is the supervision of the students? By whom?  Do you keep data on case management and resolution?  Do you use that as training material?  How often do the students meet with the supervisors? What is the purpose of such a meeting?  (feedback, teaching skills, answering questions)  Do you keep data on what specialties within the legal professions the students choose after participating in the clinic?  Is there any data or evidence on whether the students have learned particular skills by participating in the clinic?  How is the law clinic funded?  Is it currently enough to meet expenses?  If there are gaps, how do you plan to get funding?  Is the funding sustainable?  Are there possibilities of increases?  Has the clinic does a sustainability index?  If not, why not?  Does the clinic have a strategic plan?  If not, why not?  If yes, has this been helpful? Is the clinic able to follow it?  How many clients are served annually?  How do clients hear about the clinic?  What kind of cases does the clinic handle?  What cases does the clinic NOT take?  Why not? (funding, complexity)  Does the clinic have enough capacity to handle the cases that come in?  What kind of resources would be needed to handle more? (More students, funding, space)?  Is there any resistance to this clinic within the surrounding community? By local officials?  How has ABA ROLI assisted the clinic specifically?  For each program, how helpful was it?  Were there any ways that these programs could be improved? How?  Are there other programs or assistance you would like to see instituted?  Do you believe this legal clinic has improved the skills of law students?  What could be done either within the clinic or within the law faculty to improve the practical skills of the law students?  Has ABA ROLI addressed these issues? ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 64 ANNEX V: BIBLIOGRAPHY ABA ROLI, Legal Profession Reform Index for Armenia 2008. ABA ROLI Judicial Reform Index for Armenia 2008. ABA ROLI, Legal Education Reform Index for Armenia 2007. ABA ROLI Armenia Rule of Law Program Work Plan for June 1, 2011 – May 31, 2012. ABA ROLI, Assessment of the Code of Judicial Conduct of the Republic of Armenia and the Current Mechanisms for its Enforcement, January 2010. ABA ROLI Armenia Rule of Law Program, Annual Report, 2009-2010. ABA ROLI Armenia Rule of Law Program, Quarterly Reports (2009-2011). ABA ROLI 2011 Review of Chamber of Advocates of Armenia, May 2011. ABA ROLI, Assessment of the Revised Commentaries to the Rules of Judicial Conduct of the Republic of Armenia, March 2011. ABA Rule of Law Initiative, Final Report on the Results of the Survey Assessing the Extent to Which the European Convention on Human Rights and the Judgments of the European Court of Human Rights are Referenced in the Judicial Acts of the First Instance Courts of the Republic of Armenia, July 2011. Bologna Process web site: http://www.ond.vlaanderen.be/hogeronderwijs/bologna/about/index.htm Freedom House, Report on Nations in Transit 2011. P. Alston; M. Robinson, The Challenges of Ensuring the Mutuality of Human Rights and Development Endeavors, in Human Rights and Development, P. Alston, M. Robinson, Eds; Oxford Univ. Press,. 2005; p. 12 Protection of Rights without Borders and Counterpart, Implementation of Precedents of European Court of Human Rights by the Courts of RA, November 2011. Strategic Action Program for Judicial Reforms, rev. 110701. Transparency International Corruption Perception Index, December 2011. USAID, Office of Democracy and Governance, Guidance for Promoting Judicial Independence and Impartiality, Rev. Ed., January 2002, p. 27. USAID, Statement of Work, Evaluation of USAID/Armenia Rule of Law Initiative (ABA ROLI), 2011. ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 65 ANNEX VI: PROFESSIONAL SURVEY 1. Sample Survey for Lawyers Involvement in USAID/ABA ROLI and Other Donor Sponsored programs a) Have you participated in any programs of ABA ROLI? __ _ yes ___ no b) If yes, please mark those projects in which you were involved (as many as necessary)  _ Trainings  - Scientific Analytical Center  - Chamber Club discussions supported by ABA  - Development of materials (electronic, textbooks and others)  - Strategic meetings on institutional structure  - Other (please specify) c) On a scale of 1-5 (1 being the lowest and 5 the highest) assess how useful they were to you.  Trainings -1 2 3 4 5  Scientific Analytical Center -1 2 3 4 5  Club discussions supported by the Chamber -1 2 3 4 5  Development of materials (electronic, textbooks and others) -1 2 3 4 5  Strategic meetings on institutional structure -1 2 3 4 5  Other (please specify) -1 2 3 4 5 d) Did you use any of the knowledge gained in your practice? If yes, in what way? Please mark a maximum of three answers and use a scale of 1-5 to assess the level of usefulness (1 being the lowest and 5 the highest)  More involvement in the Chamber work-1 2 3 4 5  Improved practical advocacy skills including negotiations and client representation-1 2 3 4 5  Increased analytical and writing skills -1 2 3 4 5  Improved legislation analysis skills -1 2 3 4 5  Better involvement in pro bono work-1 2 3 4 5  Other (please specify) -1 2 3 4 5 e) Have you participated in any continuing legal education provided by?  Chamber of Advocates  Bar Association  OSCE  British East West Institute  AYLA ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 66  US Department of Justice f) Were any of these programs useful? ____ yes _____ no If yes, Which ones? _____________________________ (write in) f) On which topics would you be interested for the future compulsory legal education? Please mark a maximum of three answers and use a scale of 1-5 to assess the level of interest (1 being the lowest and 5 the highest)  Criminal procedure (new criminal procedure code)-1 2 3 4 5  European Court of Human Rights Law -1 2 3 4 5  Advocacy skills-1 2 3 4 5  Legal writing-1 2 3 4 5  Negotiation skills-1 2 3 4 5  Tax-1 2 3 4 5  Intellectual Property-1 2 3 4 5  Environmental Law-1 2 3 4 5  Civil Procedure-1 2 3 4 5  Economic law -1 2 3 4 5  Administrative procedures-1 2 3 4 5  Others (please specify ) -1 2 3 4 5 Legal and Institutional/Structural Limitations to Promoting Rule of Law in Armenia a) From the list below choose three key factors that limit rule of law in Armenia. Mark the most essential as 1 and the least essential as 3.  Lack of laws  Poor enforcement of the laws by the judiciary  Poor enforcement of judicial decisions  Lengthy delays within the judicial system  Citizen’s public awareness of their legal rights  Lack of transparency within the justice system  Lack of ability to access the justice system and the courts, including lack of representation ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 67  Inability to protect human rights and civil liberties  Inability to resolve contract disputes  Inability to resolve property rights disputes  Public apathy  Other (please specify) Political Limitations to Promoting Rule of Law in Armenia a) From the list below choose three key factors that limit rule of law in Armenia. Mark the most essential as 1 and the least essential as 3  Political interference from governmental decision makers (executive and legislative)  Corruption among governmental decision makers (executive and legislative)  Political will among governmental decision makers (executive and legislative)  Corruption among the judiciary  Corruption among prosecutors  Other (please specify) Improved Rule of Law situation in Armenia a) What are the three most important changes that could realistically be done to promote rule of law in Armenia? Please chose 5 answers and use a scale of 1-3 to assess their importance (Mark the most important as 1 and the lease important as 3)  Reduce corruption within the justice system  Increase the training of the judges  Increase training of judges on human rights  Increase training of judges on ethics  Increase the training of the prosecutors  Increase training of advocates  Increase salaries of justice officials  Increase training of the police  Raise public awareness of peoples’ rights  Provide more support to the Chamber of Advocates, including the new School for Advocates  Provide more support to Public Defenders  Increase regulation of civil lawyers  Provide more support to legal aid clinics  Reform and update the curriculum in the law schools  Make the judicial appointment and selection process more independent  Increase the number of female judges  Increase enforcement of court decisions  Increase regulation of civil lawyers  Improve legal education, including  Provide more support to legal aid clinics  Reform and update the curriculum in the law schools ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 68  Increase enforcement of court decisions  Other (Please specify) ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 69 ANNEX VII: CONFLICT OF INTEREST STATEMENTS ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 70 ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 71 ANNEX VIII: STATEMENT OF DIFFERENCE March 16, 2012 Bella Markarian Governance Sector Specialist Democracy and Governance Office USAID/Armenia (37410) 494 536 Re: ABA ROLI Statement of Differences to Evaluation Report Dear Ms. Markarian, Thank you for sending us the USAID evaluation of ABA ROLI’s Armenia Rule of Law Program for our review. In responses to the evaluator’s request to submit ABA ROLI’s Statement of Differences, please see below. 1. Judicial Reform a. The Judicial Department’s implementation of recommendations from ABA ROLI’s Court Monitoring Reports The evaluation team indicated that “ABA ROLI has not been as persistent as it should have been with the Judicial Department to obtain results on the court ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 72 monitoring reports” (page i) and that “the Judicial Department after it received the monitoring report has been slow to react and this is frustrating the purpose of the monitoring. The OSCE, in contrast, was able to obtain from the Judicial Department a timetable for compliance with the recommendations in its reports and also had a program for publicizing and training on the issues raised in the monitoring.” (page 32) In its efforts to ensure that ABA ROLI’s court monitoring report findings and recommendations are implemented by the Judicial Department, ABA ROLI follows up frequently with Judicial Department representatives and offers needed technical assistance. ABA ROLI is also starting the new practice of providing copies of the monitoring reports to the Ministry of Justice. It is reasonable to assume that Armenia’s status as an OSCE member state, along with the nature and size of its funding for the judiciary, may give it greater leverage to spur the Judicial Department to comply with its recommendations. In coordination with USAID, in the 2012-2013 extension year, ABA ROLI will monitor the Judicial Department’s progress in implementing our court monitoring recommendations and evaluate which future direction and resources would produce the best results. 2. Legal Profession Reform a. The Chamber of Advocates’ Executive Board The USAID evaluation stated that “ABA ROLI has been working with the Chamber to change the organization to include an executive board that would be more flexible in making decisions but the Chamber has not done this yet.” (page 14) Under the Chamber’s Charter, its Board -comprised of twelve members plus a Chair- is the executive body that manages its affairs. To date, the Board functions well. From ABA ROLI’s perspective through its close working relationship with the Chamber, there does not appear to be a challenge in the Board’s decision making. The recently passed Law on Advocacy expanded the Chamber’s responsibilities. To implement these, the Chamber may need to establish specialized committees such as executive, finance, or School of Advocacy committees. If so, ABA ROLI will support these efforts as appropriate. 3. Legal Education Reform a. Legal Writing Course The USAID evaluation stated: “the ABA ROLI work plan notes that they successfully lobbied four universities’ academic councils to make Legal Writing a mandatory for-credit course. Of the three major law faculties visited, only Yerevan State University (YSU) has legal writing as a required undergraduate course. … The evaluation team could not independently verify the existence of legal writing courses at two of the universities visited.” (page 20) ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 73 The evaluation team indicated that it visited three of ABA ROLI’s partner universities – YSU, Gavar State and Gyumri Progress Universities. Each of these has a mandatory legal writing course for LLM students, as does Open University. Please find the attached letters from the presidents of Gavar and Gyumri universities reflecting that the legal writing courses have been incorporated into their curricula. Please also find attached the Teaching Methodology Guideline for the “Legal Writing and Analysis” Course, developed in partnership with Armenian and American university counterparts. b. Legal Ethics Course and Practical Skills In its report, the USAID evaluation team wrote that “Legal ethics … are still not taught in law faculties as of the beginning of 2012” (page ii) and that “none of the law faculties visited was teaching legal ethics generally to undergraduates.” (page 21) Legal ethics is currently a component of the existing “Course on Advocacy”, a mandatory course for law undergraduates in Armenia. Legal ethics subjects comprise approximately 15% of the course material. ABA ROLI will translate into English and provide the Course on Advocacy syllabi for USAID’s review by March 23rd. In addition, per the attached letter from the Gavar State University rector, a master’s level “Ethics of Legal Profession” course has been recently introduced. In ABA ROLI’s work to improve the existing trial advocacy skills course by increasing students’ practical skills opportunities and legal ethics knowledge, in the 2012-2013 extension year, ABA ROLI will be responsive to the USAID evaluation. To push through a new and separate legal ethics course is a time- and funding-intensive process which necessarily involves engaging the Ministry of Education and securing its approval for revisions to the Education Law. If USAID wishes to prioritize this activity, contingent upon future available funding, ABA ROLI would be well-positioned to devote additional resources to this effort. We would be happy to answer any questions that you may have and to provide additional information to help USAID evaluate our program. We welcome the opportunity to improve the impact and effectiveness of our work. Best regards, Ronald S Mangum Ronald Mangum Country Director, ABA ROLI Armenia ABA ROLI Evaluation Report International Business & Technical Consultants, Inc. Page 74 cc: Corinne Smith, Irina Parshikova, Kristi Kontak Enclosures: 1) Letter from Gyumri “Progress” University Rector confirming existence of Legal Writing Course; 2) Letter from Gavar State University Rector confirming existence of Legal Writing and Ethics of Legal Profession courses; 3) Teaching Methodology Guideline for the Legal Writing and Analysis Course.