[April 2014] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Andrew Green and Marilyn Zelin of Social Impact, Inc. Mid-Term Performance Evaluation of the Judicial Strengthening Project in Macedonia MID-TERM EVALUATION OF THE JUDICIAL STRENGTHENING PROJECT IN MACEDONIA April 3, 2014 Contract No. AID-165-C-12-00001 Evaluation Mechanism No. GS-10F0294V Task Order: AID-OAA-M-13-00011 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms .................................................................................................................................................................................................. 1 Executive Summary................................................................................................................................................................................. 3 Evaluation Purpose ............................................................................................................................................................................ 3 Project Background ........................................................................................................................................................................... 3 Evaluation Questions, Methods and Limitations ........................................................................................................................ 3 Conclusions ......................................................................................................................................................................................... 4 Recommendations ............................................................................................................................................................................. 5 Introduction .............................................................................................................................................................................................. 7 Evaluation Purpose & Evaluation Questions .................................................................................................................................... 7 Project Background ................................................................................................................................................................................ 8 Macedonia: Political Development and Judicial Sector Reform ............................................................................................. 8 Judicial Strengthening Project ......................................................................................................................................................... 8 Evaluation Methods & Limitations ...................................................................................................................................................... 9 Evaluation Methods ........................................................................................................................................................................... 9 Methodological Limitations ...........................................................................................................................................................10 Findings ....................................................................................................................................................................................................11 Result 1: Strengthened Advocacy and Citizen Participation in Judicial Sector Reforms ...............................................11 Result 2: More Independent, Efficient and Consistent Application of Judicial Sector Policies and Practices...........14 Result 3: Effective Legal Personnel and Efficient Processes ..................................................................................................18 Conclusions ............................................................................................................................................................................................22 Analysis of Progress and Effectiveness of Project Interventions .........................................................................................22 Evaluation Question #1: What is the progress to date on the following stated project results? ...............................22 Evaluation Question #2: How effective has been the project approach and interventions in achieving the expected results to date? ...............................................................................................................................................................22 Result 1 ..........................................................................................................................................................................................23 Result 2 ..........................................................................................................................................................................................24 Result 3 ..........................................................................................................................................................................................24 Evaluation Question #3: How well is the project communicating and collaborating with beneficiaries and counterparts in the implementation of project interventions? ............................................................................................25 Evaluation Question #4: Based on results to date, is the Project likely to engender sustainable and systematic development impacts after USAID funding has stopped? What should the implementer do to ensure sustainability and transfer of know-how? ...................................................................................................................................26 Recommendations ................................................................................................................................................................................27 Recommendations for Modifications to Improve Process Elements ..................................................................................27 Evaluation Question #5: Given the progress to date, provide a recommendation if the optional one year extension of JSP should be exercised. ........................................................................................................................................27 Evaluation Question #6 and #8: Does the project approach need to be modified in order to reflect the current justice sector environment? If so, how? What alternative approaches exist which could lead to better results and greater cost efficiency? ...................................................................................................................................................................28 Evaluation Question #7: Which selected actions and cross-cutting themes and corresponding activities should be further emphasized, modified or eliminated and why? .....................................................................................................29 Annexes ...................................................................................................................................................................................................31 Annex I: Evaluation Statement of Work ....................................................................................................................................31 Annex II: Team Composition and Study Methods ..................................................................................................................38 A. Team Composition ..........................................................................................................................................................38 B. Study Methods ...................................................................................................................................................................38 Annex III: Data Collection Instruments .....................................................................................................................................39 A. Evaluation Matrix ..............................................................................................................................................................39 B. Semi-Structured Interview Questions ......................................................................................................................... 41 Annex IV: Sources of Information ...............................................................................................................................................45 A. Persons Interviewed ........................................................................................................................................................45 B. Documents Reviewed......................................................................................................................................................47 1 ACRONYMS ACCMIS Automated Court Case Management and Information System ABMS Automated Budget Management System ADR Alternative Dispute Resolution AFFT All for Fair Trials Coalition AO Administrative Office CAA Court Administration Association CBC Court Budget Council CIRa Center for Institutional Development CoP Chief of Party COR Contracting Officer’s Representative CPC Criminal Procedure Code CSC Court Services Council CSO Civil Society Organization DCM Differentiated Case Management DCoP Deputy Chief of Party DG Democracy and Governance ECHR European Convention on Human Rights or European Court on Human Rights EDR Event Data Recorders ELSA European Law Students Association ERS Electronic Recording System ESE Association for Emancipation, Solidarity and Equality of Women EU European Union FOIA Freedom of Information Act GBV Gender-based Violence GDP Gross Domestic Product GoM Government of Macedonia GRECO Groupe d’Etats Contre La Corruption / Group of States Against Corruption IDEAS Investment Development and Export Advancement Support Project INPROL International Network to Support the Rule of Law IPA Instrument for Pre-Accession Assistance IQC Indefinite Quantity Contract IT Information Technology JBF Judicial Branch Forum 2 JC Judicial Council JCCMIS Judicial Council Case Management Information System JRIP Judicial Reform Implementation Project JSP Judicial Strengthening Project JTA Academy for Training of Judges and Public Prosecutors (also known as Judicial Training Academy) KI Key Informant LCS Law on Court Service LPA Legal Professional Association MBA Macedonian Bar Association M&E Monitoring and Evaluation MJA Macedonia Judges Association MLA Macedonian Lawyer’s Association MoJ Ministry of Justice MYLA Macedonia Young Lawyers Association NGO Non-governmental Organization OSCE Organization for Security and Co-operation in Europe OPDAT Office of Overseas Prosecutorial Development, Assistance, and Training PPA Public Prosecutors Association RCT Randomized Control Trial ROLC Rule of Law Council SC Supreme Court SI Social Impact, Inc. SSI Semi-Structured Interview SWOT Strengths, Weaknesses, Opportunities, and Threats ToT Training of Trainers USAID United States Agency for International Development USG United States Government VMRO-DPMNE International Macedonian Revolutionary Organization – Democratic Party for Macedonian National Unity 3 EXECUTIVE SUMMARY EVALUATION PURPOSE Social Impact conducted a mid-term performance evaluation for USAID/Macedonia of the Judicial Strengthening Project (JSP) during March 2014. The purpose of this evaluation is to analyze the progress and effectiveness of the project interventions toward achieving its stated program results to date and recommend potential modifications for improvement. It will also inform the Mission’s decision on exercising the optional one year extension. PROJECT BACKGROUND The Judicial Strengthening Project (JSP) is the third in a series of USAID-funded projects focused on the judicial sector, all implemented by Tetra Tech DPK. The second project, the Judicial Reform Implementation Project (JRIP), built on the first project’s emphasis on court modernization. JRIP was designed to deepen the prior reforms, reduce case backlogs (primarily through installation of an Automated Court Case Management and Information System (ACCMIS)), introduce additional automation, and enhance the work of the courts and all court personnel. JSP, in turn, built on JRIP to a large extent. JSP is designed to address both the demand and supply sides of judicial independence, effectiveness, and accountability. The project is organized in three components, one to support the development of organizations outside the judicial branch to advocate for policies to enhance the rule of law and the independence of the judicial sector (demand); one to partially build on JRIP to work with leading judicial branch actors on developing policies to strengthen judicial independence and improving operations to raise the quality of court processes (supply); and one to continue court room automation and judicial training (supply). The project operates in a highly politicized environment dominated by a single party in which continued judicial branch reform is a challenge. EVALUATION QUESTIONS, METHODS AND LIMITATIONS The Evaluation Team applied a mixed methods approach using standard rapid appraisal methods of semi￾structured interviews (SSIs) of key informants (KIs) and materials review. The Team conducted over 36 SSIs and site visits in three cities. A mixed method approach and reliance on SSIs often involve methodological limitations, including recall, response, and selection bias, which are often exacerbated by the technical nature of rule of law programs. Nonetheless, the Evaluation Team took several steps to mitigate such bias. Through these efforts, the Team sought to provide answers to the following questions: 1. What is the progress to date on the following stated project results? 2. How effective has been the project approach and interventions in achieving the expected results to date? 3. How well is the project communicating and collaborating with beneficiaries and counterparts in the implementation of project interventions? 4. Based on results to date, is the Project likely to engender sustainable and systematic development impacts after USAID funding has stopped? What should the implementer do to ensure sustainability and transfer of know-how? 5. Given the progress to date, provide recommendation if the optional one year extension of JSP should be exercised? 4 6. Does the project approach need to be modified in order to reflect the current justice sector environment? If so, how? 7. Which selected actions and cross-cutting themes and corresponding activities should be further emphasized, modified or eliminated and why? 8. What alternative approaches exist which could lead to better results and greater cost efficiency? CONCLUSIONS The Team concluded that:  Overall, JSP met the targets under the evaluation’s specified requirements, as seen in the project’s monitoring and evaluation (M&E) Plan. Nonetheless, the indicators included in the Plan do not fully recognize or capture key project activities relevant to the overall goal of the project. The Team concluded that targets were not very ambitious, or were tailored to match actual conditions. By contrast, the Results are over-ambitious and at a much higher level of effect than the underlying activities could ever achieve. This is due in large part to the specification of results in the Request for Proposal and subsequent Contract, which are quite difficult for USAID to change, but also to the lack of an explicit and systematically applied theory of change.  JSP has indeed been effective in its project interventions, but the overlying political context and weak judicial independence limits the effect these have on the higher goals of the project.  JSP is in regular and collaborative communication with the various judicial branch institutions and relevant legal professions associations (LPAs) and civil society organizations (CSOs). Partners and beneficiaries have tremendous respect for JSP, which has not been affected by the transition of the Deputy Chief of Party (DCoP) to Chief of Party (CoP).  In terms of utility, JSP has definitely raised the capacity of its partner LPAs and CSOs through its tailored program of capacity-building training. The JSP promoted Rule of Law Council (ROLC), which brings together key LPAs and CSOs for discussions of judicial sector reform issues, is the only such discussion opportunity that exists. Assistance for the Judicial Council (JC), Administrative Office of the Judicial Council (AO), Court Budget Council (CBC), Court Services Council (CSC), and Judicial Training Academy (JTA) continued from JRIP and was demonstrably valuable to leadership in those institutions. The maintenance and upgrade of automated systems was particularly welcomed by the courts, as was the addition of electronic recording systems (ERS) in courtroom hearings. In terms of efficiency, activities such as customized websites and printed brochures are a passive and costly a means to raise public awareness of the judicial branch and various actors in the sector. By contrast, the JBF is viewed as the best means for bypassing the dysfunctional Supreme Court in order that judicial branch actors can discuss and prioritize judicial reform issues. In terms of sustainability, the LPAs and CSOs have indeed become more capable, as seen in the reproduction of learned skills and knowledge. The Court Administration Association (CAA) and Macedonia Young Lawyers Association (MYLA) have directly benefitted from the assessment and resulting capacity-building training designed for them. The European Law Students Association (ELSA) has benefitted too, but as an all-volunteer student organization, it faces inherent challenges to sustainability. The ROLC is potentially unsustainable, but it is too early at this point to make a definitive conclusion.  At this point, there is a mixed picture with regard to sustainable and systematic development of the broader judicial sector. 5 The JC, CSC, AO, and CBC may be more capable than before, but they are still highly dependent on JSP for technical assistance and financial support. JSP’s support to the JC in the strategic planning process is critical to not just the quality of the outcome, but also to the judicial branch’s need to actively consider the perspectives of all judicial branch actors. The establishment of the JBF is an innovative way to bypass the inactive Supreme Court, but again this forum is wholly dependent on JSP facilitation. Other operational interventions (e.g., ACCMIS, Automated Budget Management System-ABMS, etc.) will continue to be used in the future. Much of the equipment and systems have become outdated, however, and it was not clear to any KI where funding for upgrading could be obtained. JSP could improve the prospects for sustainability by working more closely with the Supreme Court, collaborating with efforts to implement the Criminal Procedure Code (CPC), and coordinating with other donors to identify funding resources. RECOMMENDATIONS The Team recommends that:  USAID/Macedonia exercise the one-year option to extend JSP. There are several interventions that should be continued during the extension period: ROLC start-up; capacity-building for key LPAs and CSOs at critical points in their development; support to the JBF; engagement of the Supreme Court; efforts to address deficiencies in the recent amendments to the LCS; and further consideration of differentiated case management (DCM).  Relatively minor changes to the approach and activities should be made. The overall political and judicial context has not changed significantly since the design and start-up of JSP. The dominance of the executive branch has continued and may even be worsening, as exemplified by the recent flood of expedited legislation, of which the new Law on Court Service was a part. Further, the Government persists in using the law for its own purposes, as evidenced by the use of defamation suits for punishment of political opposition and the media. The judicial sector is more capable in operational and administrative respects, primarily as a result of JSP assistance, but it lacks the budgetary resources to meet its own internal needs. The Government of Macedonia (GoM) has not complied with the Law on Court Budget’s provisions to raise the court budget over time, and the budget has actually decreased in absolute terms. Judicial sector institutions are unsure about the availability of resources to maintain improvements initiated and supported under JSP and previous projects. At this time the judicial sector faces problems in advocating for its own interests and appears unable to sustain the supply side of reform as foreseen in the JSP design. The demand side is limited by the small number of LPAs/CSOs interested in promoting reform. Although organizational capacity has improved as a result of JSP assistance, the missions of the LPAs are to serve their members. Advocacy efforts to raise public awareness of judicial sector issues involve passive modes (websites) and limited-audience distribution of printed materials. Neither of these could reasonably be expected to change broader public perceptions, which are shaped much more by media coverage of high-profile cases that are proxies for political competition. However, the JBF and ROL Coalition, both new initiatives, may have the latent capacity to support the supply/demand approach of JSP. Therefore, continued support should be provided, especially support for implementation of the nearly completed Strategic Reform Plan 2014-2016, 6 which merges the interests of the demand and supply sides. The Strategic Plan should inform JSP’s work during the extension period, with rigorous monitoring of implementation.  The three cross-cutting issues of sustainability, EU alignment, and serving gender and minority communities receive more focused attention in JSP’s Year Three Work Plan. As a result, there has been only a partial period of activity to analyze in this evaluation. As noted above, the ownership and sustainability of judicial branch independence, effectiveness, and accountability is questionable, and should be emphasized more in the remaining option year and beyond. The judicial branch actors rely on JSP to help them coordinate, discuss, and even act collectively, and do not foresee this happening in the future without JSP’s assistance. This is due in part to the budgetary situation, but also to a lack of vision among themselves that up to now has only been supplied by JSP. It is clear to the Team that the overall political context is the primary factor limiting judicial branch independence, as seen in the budgetary allocations and revisions in the framework legislation, i.e., outside the judicial branch itself. This calls for a deliberate, concerted effort on the part of the international community to pressure the GoM to allow the judicial branch the authority and resources needed for meaningful independence. A slowdown of EU activities in Macedonia (and in other states moving toward EU integration) has decreased the impetus for the GoM to engage in judicial reform actions that would strengthen this branch of the state. For JSP, supporting activities beyond the minor ECHR￾related efforts by CSOs and occasional JTA-based training is sufficient, other than the higher diplomatic actions discussed above. JSP’s engagement with gender or minority community issues has not been significant or integrated well within the project. Although the GBV work with a local CSO was valuable and would not have happened without JSP support, engagement with minority community issues have been marginal, due in large part to the existence of quotas within the judiciary for minority communities. 7 INTRODUCTION USAID/Macedonia, through the USAID Center of Excellence on Democracy, Human Rights and Governance, contracted Social Impact to conduct a mid-term performance evaluation of the USAID￾funded Judicial Strengthening Project (JSP) project. Implemented by Tetra Tech DPK, the overall JSP objective is to strengthen the role of the judiciary in the separation of powers and to develop broad based support for rule of law and judicial sector reform in Macedonia. The three-year project began in November 2011 and concludes in November 2014, with a possible one-year extension through November 2015. The purpose of the evaluation was to assess progress to date according to the project’s Monitoring and Evaluation Plan (M&E Plan), recommend modifications, and assist in decision￾making by USAID/Macedonia in determining whether or not to exercise an option year. Evaluation planning began in February 2014, and field work was conducted in Macedonia over two weeks, from March 3 through March 17, 2014. During this two-week period, the Evaluation Team was based in Skopje, and also conducted field work in Veles, Štip, and Bitola. EVALUATION PURPOSE & EVALUATION QUESTIONS The purpose of this evaluation is to analyze the progress and effectiveness of the project interventions toward achieving its stated program results to date and recommend potential modifications for improvement. It will also inform the Mission’s decision on exercising the optional one year extension. The performance evaluation was to answer eight questions in two clusters: Cluster A: Analysis of Progress and Effectiveness of Project Interventions  EQ #1: What is the progress to date on the following stated project results1: o Result 1 –Strengthen Advocacy and Citizen Participation in Judicial Sector Reforms o Result 2 – More Independent, Efficient and Consistent Application of Judicial Sector Policies and Practices o Result 3 - Effective Legal Personnel and Efficient Processes  EQ #3: How well is the project communicating and collaborating with beneficiaries and counterparts in the implementation of project interventions?  EQ #4: Based on results to date, is the Project likely to engender sustainable and systematic development impacts after USAID funding has stopped? What should the implementer do to ensure sustainability and transfer of know-how? 1 Subsequent guidance from USAID/Macedonia clarified that this evaluation question should be interpreted to refer to progress according to the M&E Plan. 8 Cluster B: Recommend Potential Modifications for Improvement  EQ #5: Given the progress to date, provide recommendation if the optional one year extension of JSP should be exercised?  EQ #6: Does the project approach2 need to be modified in order to reflect the current justice sector environment? If so, how?  EQ #7: Which selected actions and cross-cutting themes and corresponding activities should be further emphasized, modified or eliminated and why?  EQ #8: What alternative approaches exist which could lead to better results and greater cost efficiency? PROJECT BACKGROUND MACEDONIA: POLITICAL DEVELOPMENT AND JUDICIAL SECTOR REFORM Macedonia gained independence from Federal Republic of Yugoslavia in 1991, forming a parliamentary democracy. After suffering through several episodes of intense ethnic violence, the 2001 Ohrid Accords prevented further deterioration in ethnic relations and made explicit the role of ethnic minorities in political life. Macedonia was granted candidate status by the EU in 2005, but Greek objections to the country’s name have been an intractable obstacle to deeper and more active EU engagement. The International Macedonian Revolutionary Organization–Democratic Party for Macedonian National Unity (VMRO-DPMNE) party has been in power since the 2006 parliamentary elections, always in coalition with an Albanian party. During this time period, several governance related challenges and concerns have emerged. The coalition’s majority in parliament has been used on multiple occasions to alter the standing rules of order and allow expedited passage of controversial legislation. Corruption is a significant factor of life at all levels, and politically-connected favorites have easy access to valuable state contracts, concessions, and privatization processes. Media critical of the Government of Macedonia (GoM) face targeted harassment, including the strategic use of defamation lawsuits that carry significant financial penalties. Large portions of civil society are highly politicized, in opposition to the coalition government. JUDICIAL STRENGTHENING PROJECT Important steps were taken in 2001-06 to strengthen the judiciary. These include development of a national strategy, expanded budget authority, and the creation of the Judicial Council, the Public Prosecutor’s Council, a bailiff’s system, and the Judicial Training Academy. The JSP is but the third in a series of USAID-funded projects focused on the judicial sector, all implemented by Tetra Tech DPK. The second project, the Judicial Reform Implementation Project (JRIP), built on the first project’s emphasis on court modernization. JRIP was designed to deepen the prior reforms, reduce case backlogs (primarily through installation of an ACCMIS), introduce additional automation, and enhance the work of the courts and all court personnel. JSP, in turn, built on JRIP to a large extent. JSP is designed to address both the demand and supply sides of judicial independence, effectiveness, and accountability. The project is organized in three components: (1) to support the development of organizations outside the judicial branch to advocate for policies to 2 Subsequent guidance from USAID/Macedonia defined ‘approach’ to mean the explicit demand- and supply-side design of JSP. 9 enhance the rule of law and the independence of the judicial sector (demand); (2) to partially build on JRIP to work with leading judicial branch actors on developing policies to strengthen judicial independence and improving operations to raise the quality of court processes (supply); and (3) to continue court room automation and judicial training (supply). JSP is structured to address five key results, although this evaluation is only concerned with the first three: Result 1: Strengthened advocacy and citizen participation in judicial sector reform Result 2: More independent, efficient, and consistent application of judicial policies and practices Result 3: Increased fairness and efficiency of the administration of justice through more effective legal personnel and efficient processes Result 4: Crisis-modifier and material support provided Result 5: Rapporteur services with justice sector provided The first Result is intended to be a “demand” result, as the activities under it are to strengthen the demand outside government institutions for an independent, effective, and accountable judicial sector. To achieve this result, TetraTech DPK was to identify the strongest civil society organizations (CSOs) with direct interest in judicial sector issues, build their capacity to function as organizations, and provide support for advocacy and public awareness. The second and third Results are intended to be “supply” results, as the activities under them are designed to strengthen the supply of independent, effective, and accountable justice in Macedonia. The second result could be further subdivided into activities addressing policy and practice issues. Under policy, the JSP team has worked with leading judicial branch actors to engage the other branches of government, conduct analyses of case management and workloads, conduct strategic planning, and develop means to increase interaction between judicial branch actors on issues of common concern. Under practice, JSP worked to increase the capacity through training or other facilitative assistance, develop manuals and standard processes, and improve the transparency and perception of the courts. The third result focuses on the administration of justice in the courts themselves. To achieve this result activities included supporting training, leadership development, exploration of comparative models, systems and processes for backlog reduction and faster case processing, and other measurement and equipment needs. EVALUATION METHODS & LIMITATIONS As seen in the set of evaluation questions, this mid-term evaluation combines both performance and process evaluations. Social Impact (SI) strongly believes in the use of mixed methods for answering these evaluation questions, a value reiterated through USAID’s Evaluation Policy. Therefore, the Evaluation Team applied standard rapid appraisal methods of semi-structure interviews (SSIs) of key informants (KIs) and materials review. The Team did not conduct a mini-survey of court administrative staff due to concerns that recent legislative amendments to a key framework law on court service would unduly bias responses. As we note below, however, a mixed method approach and reliance on SSIs often involve methodological limitations. EVALUATION METHODS The Team reviewed a large number of project documents and other secondary materials, including but not limited to periodic reports to USAID, materials for direct beneficiaries produced with project support, and Department of State and European Union (EU) analyses of political and judicial reform. 10 The SSI protocols were finalized by the Evaluation Team after in-depth discussions with USAID personnel and JSP staff. Each SSI differs depending on the KI’s role and “causal distance” from activities, as well as the extent of the KI’s involvement in JSP activities and the time available for interviewing. Thus, the Team developed questions to ask of (a) the implementer and its contractors (i.e., JSP, CIRa, and EMBRA); (b) direct beneficiaries that participated in various activities (e.g., capacity-building trainees, Judicial Council members, President judges in pilot courts, and similar); (c) indirect beneficiaries who did not participate in a particular activity (e.g., President judges on ACCMIS or changes in interaction with court administrators, Macedonian Bar Association (MBA) on electronic court recording, and similar); and (d) external experts who have technical expertise in judicial sector issues but are outside of the sector (e.g., OSCE and the EU). Illustrative questions are included in this report as Annex III. As can be seen, the questions address not just knowledge and general perceptions, but they more importantly probe for specific examples of attitude and behavior change. The Team interviewed over 36 individuals, which included 4 US government (USG) personnel, 3 implementer staff, 23 direct beneficiaries, 2 indirect beneficiaries, and 4 external experts. It is important to please note that some direct beneficiaries of some activities were also indirect beneficiaries of other activities. After consultation with USAID/Macedonia, the Team conducted site visits to direct and indirect beneficiaries in the Skopje area, Veles, Štip, and Bitola, but the sheer number of JSP activities and the limited evaluation field time precluded a more extensive effort. Parallel analysis was used to analyze the evidence from SSIs and materials review. In this analytical approach, each type of data for an activity is analyzed in parallel, and then across data type. For example, in the case of Result 1, the Evaluation Team first analyzed SSIs with JSP staff, CIRa director, and the EMBRA director for evidence of the effect of capacity-building on CAA and MYLA; second, analyzed SSIs with leadership and other staff of those organizations; and third, analyzed the data from SSIs with indirect beneficiaries and external experts to develop preliminary findings; we then analyzed relevant materials to develop preliminary findings; and finally, analyzed preliminary findings across the types of data to develop activity-level findings. METHODOLOGICAL LIMITATIONS As with any research methodology, there are methodological limitations that should be recognized by readers. In addition, the complex nature of rule of law projects and consecutive implementation by the same organization added to these limitations. Recall bias is always an issue for multi-year projects, as KI’s memories fade or are commingled with those of similar activities by other donor-funded projects. This can be mitigated by asking KIs about specific activities clearly, and asking for details to ascertain which activity they have in mind. In the case of JSP, this was a more significant problem than the norm given that this is the third consecutive project by the same implementer, with much of the same project personnel, and more importantly a project design that continued elements of the previous project. As a result, it is difficult to isolate opinions about JSP vis-à-vis previous USAID funded efforts. Response bias is perhaps the biggest single limitation of evaluation methodologies that rely on SSIs, because many KIs have strong incentives to provide positive responses about the project in order for their organizations or themselves to continue benefitting in the future. One way to mitigate this bias is to talk to more indirect beneficiaries and external experts. In the case of judicial strengthening or other rule of law projects, this is typically a greater challenge because the number of such indirect beneficiaries and experts is limited by the highly technical nature of this work. Selection bias is a general problem with performance evaluations, as the exercise depends at least initially on contacts provided by the project itself. A strategy for mitigating the effect of this form of bias is to identify additional indirect beneficiaries and external experts, or even take advantage of direct beneficiaries of one activity who might be indirect beneficiaries of another activity. As with response 11 bias, the technical nature of judicial strengthening projects results in a small number of potential KIs with substantive information for interviews. Attribution is the most difficult problem for non-randomized control trial (RCT)-based evaluations, as evaluation teams are unable to establish a rigorous counter-factual and cannot isolate the programming effects from other donor-funded projects. For this evaluation, JSP is the only significant project in these activity areas, but key activities under Results 2 and 3 were continuations of the previous project. FINDINGS This section presents the findings of fact to answer the eight evaluation questions. Following the structure of the evaluation scope of work and the guidance from USAID/Macedonia, the Team presents findings of fact. In the sections that follow, the Team presents its conclusions for the first cluster of four evaluation questions that address performance aspects, and then conclusions and recommendations for the second cluster of four questions that address process aspects. The first evaluation question asks what has been the progress to date in achieving the project result. In the following paragraphs we consider progress for each result and their respective requirements in turn. RESULT 1: STRENGTHENED ADVOCACY AND CITIZEN PARTICIPATION IN JUDICIAL SECTOR REFORMS Requirement 1.1: A short focused assessment, baseline survey, and a resulting action plan on justice sector professional associations and civil society organizations and the role they can play in strengthening the rule of law in Macedonia. Requirement 1.2: Per the findings of Requirement 1.1, the capacity of at least two judicial sector professional associations are improved so that they can better serve their members and are better equipped to play an advocacy role in justice sector policy-making, and assume the ownership of reforms by generating the demand for change among its members and in justice sector institutions themselves. In general, the Team found that JSP had strong communication and collaboration with the three organizations that primarily benefitted from activities under this Result, the Court Administrators Association (CAA), the Macedonia Young Lawyers Association (MYLA), and the European Law Students Association (ELSA). While the project also selected the Macedonia Judges Association (MJA) for inclusion in project activities, as will be discussed in greater detail below it ended its participation in the middle of the training process, and as such it is discussed in this section. KIs from CAA and MYLA were able to provide specific examples of how JSP worked with them directly and through the two contractors, CIRa and EMBRA, to identify, design, and then provide training modules that met their needs. While ELSA, which was brought on later in the project, did not participate directly in the curriculum development, ELSA was included in the assessment and the training modules its representatives attended met their needs despite not being designed specifically for the organization. One implementer KI mentioned that JSP has a better relationship with these organizations than other donors, because the organizations respect JSP’s collaborative approach. CIRa and EMBRA also gave positive responses about working with JSP. This was perhaps not surprising given the revenue-producing nature of their relationship with the project; however, the Team extracted specific examples of collaborative interaction. For example, KIs reported the finalization process of the organizational capacity assessment tool with CIRa as a discussion among equals. 12 JSP met its M&E Plan targets for this Result. For the first Requirement, an assessment of potential organization partners was conducted through CIRa. The second Requirement stated that at least two organizations must have their capacity strengthened, and the follow-up assessment by CIRa showed strengthened capacity among CAA, MYLA, and ELSA (not MJA). JSP also provided assistance to organizations for advocacy campaigns to foster public understanding, non-governmental organization (NGO) support, and media coverage for judicial independence and accountability issues, as seen in its work on websites and brochures for CAA, MYLA, ELSA, and MJA. Work in the third year also included All for Fair Trials Coalition (AFFT) and ESE. As an additional measure of outcomes resulting from improved capacity, JSP tracks CSO promoted reform laws or policies changed or defeated; however, this may prove to a problematic indicator in light of the expedited amendment of the Law on Court Service. For Requirement 1.1, the Team found that: The requirement was completed. KIs at CAA, MYLA, and ELSA explicitly said that the assessment process was inclusive and collaborative, and that the work by CIRa was quite professional. For Requirement 1.2, the Team found that: CAA, MYLA, and MJA were originally selected to be direct beneficiaries of capacity building activities; however, the selection of MJA was not in line with the original selection criteria. Instead, MJA was selected for reasons more closely tied to other JSP goals and needs, and the association eventually withdrew from the program. As a result, several other organizations were invited to participate in training modules, including ELSA, which better fit the needs identified in CIRa’s assessment. Following legal professional association (LPA) selection, EMBRA worked closely with JSP and CIRa to develop a set of training modules and met with the selected organizations to tailor its training topics and content to meet identified needs. Setting aside the MJA, KIs from participating associations valued the training and EMBRA's expertise. All KIs from CAA, MYLA, and ELSA provided specific examples of skills and knowledge they gained from the trainings that are still being used and reproduced within the organizations. To illustrate:  MYLA: MYLA KIs pointed to the development of skills in strategic planning, human resource and volunteer management, EU funding strategies, public relations and media, and financial budgeting and planning. The organization also increased its membership based from 30 to 55 people, and it organized a national conference on the “Legal Profession in Macedonia— Challenges, Problems and Perspectives” to discuss key issues in legal education and the bar exam. MYLA also noted that the training of trainers they received helped them collaborate with and mentor ELSA. MYLA is about to embark on a new three-year strategy, and reported that they would be capable of conducting the process on its own, albeit with the helpful facilitation of an outside group to avoid group-think.  CAA: KIs from the implementing partners and indirect beneficiaries reported that CAA in particular has demonstrated capacity for membership management, and a document review of CIRa’s analysis of organization capacity change showed that CAA showed the greatest overall improvement. All interviewed CAA members reported that the organization can plan meetings, maintain an association structure of branches, has leadership selection bylaws, can communicate with members, and was actively involved in an analysis of the 2009 (LCS) to identify and propose amendments in collaboration with the CSC.  ELSA: The ELSA KI reported improved capacity to prepare proposals for EU funding and management and leadership skills applicable to the work of the Board, although the ELSA KI noted that training primarily benefits the members that participate as ELSA is a student organization with an ever-changing membership. ELSA valued the networking opportunities 13 provided by joint trainings as evidenced by stronger cooperation with MYLA as well as financial support from the Public Prosecutors Association (PPA) and Notaries Association for the ELSA regional conference on “The Fight Against Human Trafficking” hosted by the Macedonian branch in Dec. 2013. In addition, JSP supported the development of customized websites for selected organizations, and the creation and distribution of brochures. The CAA website is a key means for communication with its members, but MYLA and ELSA primarily rely on social media despite their existing websites. The ELSA brochures, one on Macedonia’s judicial system structure and process and the second on juvenile justice, were distributed to secondary and elementary school students and complemented by visits to judges and courts. Both were viewed as successful by ELSA and participating judicial personnel, but the Team lacks corroborating evidence of the effect on participating students. CAA and MYLA brochures were distributed in appropriate places, but the Team again lacks evidence of effect on targeted audience; none of the three organizations knew how they would finance the production of another round of brochures. One of the capacity building activities included a well-received study tour to Estonia and Sweden, which highlighted models of state-CSO interaction.3 KIs at the main participating organizations noted that this trip was the impetus for collaborating with JSP to form the Rule of Law Council (ROLC). MYLA reported this tour as another manifestation of the training program’s positive unintended consequence on rule of law related networking among CSOs. While the ROLC has potential to serve as a vehicle for CSOs and LPAs to participate in and promote judicial sector reform; it is only a nascent initiative and suffers from some potential weaknesses. There are a limited number of LPAs and other CSOs interested in and capable of demanding judicial independence, effectiveness, and accountability, which will in turn limit expanding the ROLC’s membership (MJA and MLA stated that ROLC is not within their focus areas). Furthermore, among those participating there was no clear consensus on the nature and activities of the ROLC that could be conveyed to the Team. What was clear was that different organizations have different viewpoints. For example, MYLA KIs thought that CAA may be able to generate public demand for reform, but CAA KIs hoped to use the ROLC as a vehicle for promoting its own more narrow objectives. JSP engaged Kaizen Company, a US consulting firm, to facilitate initial events and planning for the ROLC. Kaizen recently facilitated a planning retreat attended by 11 representatives of MYLA, CAA and ELSA. The Kaizen KIs reported that the organizations had agreed to focus on citizen education and that they appeared to be committed to the ROLC. The KIs noted that the challenge for JSP would be limiting its role and allowing ROLC participants do the work. If this could be accomplished, they felt ROLC had the potential to be part of the JSP sustainability plan. However, the Team notes that this view may be influenced by the revenue producing nature of the consultancy and that the Team did not have the opportunity to interview the organizations after the retreat. As one final point, it is worth nothing that the Association for Emancipation, Solidarity and Equality of Women (ESE) valued the support of JSP to promote awareness of gender-based violence (GBV) issues. In particular, the KI from ESE noted that JSP’s credibility gave them entrée to judicial branch actors. As a result, ESE will create a benchbook to provide guidance to judges for GBV cases. 3 Representatives of AFFT, MYLA, Macedonian Lawyer’s Association (MLA), CAA and ELSA participated in the study tour. 14 RESULT 2: MORE INDEPENDENT, EFFICIENT AND CONSISTENT APPLICATION OF JUDICIAL SECTOR POLICIES AND PRACTICES Requirement 2.1: Legal framework and judicial branch policies strengthen independence, effectiveness and accountability of the judicial system. Requirement 2.2: Administration and management rules, policies, procedures, systems and practices support a modern court system. As part of this requirement, the Contractor will work with the judicial sector authorities and actors to establish effective governance and operational systems for managing court resources – budget, human, facilities, equipment, etc.; strengthen managerial capabilities and introduce performance standards. The distinction between activities under Requirement 2.1 and 2.2 was not always clear and that some activities were reported in more than one place or in different places in different reports. The Team understands the distinction to essentially be that Requirement 2.1 deals with judicial policy and Requirement 2.2 with operations and this evaluation report reflects this distinction. The Team also found overlap between Result 1 and 2 (e.g. strategic planning for the judiciary is presented in both places). For Result 2 overall, the Team found that: All KIs reported excellent cooperation with JSP under Result 2. The Head of the Administrative Office (AO) of the Court Budget Council (CBC) also reported continuous cooperation with JSP and said she considered JSP part of the team. Both the President of the Judicial Council (JC) and Head of the AO said that the high level of cooperation was not interrupted when Ms. Ivanovska took over as CoP. The President of the Veles Basic Court, which served as a pilot court for several initiatives, noted a continuous and productive relationship with JSP (again over the course of several projects). He mentioned that JSP staff visited the court on a regular basis. The Ministry of Justice (MoJ) KI noted that the MoJ is not in direct contact with JSP as its cooperation with the project is through membership on the JC. The M&E Plan targets were met. JSP supported its target number of legal institutions and associations, although the latter should be counted under Result 1. The target for the number of courts with improved case management was achieved, but the indicator and related narrative do not indicate what data were used to arrive at this figure. The target number of policies or laws were passed or changed to comply with the agenda of judicial sector authorities (or proposed negative changes defeated); however, in addition to the lack of a substantive M&E narrative, it would appear that the same policies or laws are being used for the similar and complementary indicator for CSOs under Result 1. The target for number of courts that developed needs-based budgets was achieved. For Requirement 2.1, the Team found that: The JSP Year One Work Plan4 envisioned establishing a working committee(s) composed of representatives of the judiciary and external stakeholders to systematically review laws and regulations, identify gaps and inconsistencies, and make recommendations for amendments. JSP supported discussion groups on legal issues identified by the MJA5, but this was not a systematic review. Laws and regulations were also reviewed in the assessment report that is discussed 4 JSP Year One Workplan, p. 9. 5 These include Decriminalization of Macedonian Law on Defamation and Responsibility of Administrative Bodies for the Duration of Administrative Practices. 15 under requirement 2.2, but it is not clear to the Team whether that assessment was intended to be a substitute to the planned systematic review by the working committee(s). All KIs in the judicial branch recognized the gap of leadership due to the Supreme Court’s inactivity and the JC’s limited mandate to issues of selection, evaluation, and discipline of judges. Because of its limited mandate, the JC legally cannot fulfill a leadership role for the judicial branch. Instead certain leadership responsibilities are within the mandate of the Supreme Court (e.g. harmonization of legislation and procedures) and all relevant KIs acknowledged that the Supreme Court has not fulfilled such responsibilities. Providing statutory authority for the JC to assume leadership of the judiciary would require an amendment to the Constitution and KIs noted reluctance to initiate such an amendment for a variety of reasons. The MoJ KI did, however, indicate that changes to the JC’s mandate may occur during the next year. The Team found that JSP’s attempt to address the lack of Supreme Court leadership with activities like the Judicial Branch Forum (JBF) was valued precisely for that reason.6 JSP convened the first JBF in November 2012, where participants agreed to hold the forums quarterly. The fifth JBF was held in December 2013. The Team found that during its short existence, the JBF has discussed key judicial issues, including judicial ethics, uniform court statistical reporting, Differentiated Case Management (DCM), staffing guidelines and needs based budgets, an ICT development plan for the judiciary, the need for a stronger role by Appellate Courts in backlog reduction and harmonization of court decisions, and the role and activities of the Judicial Training Academy (JTA). All KIs strongly valued the role of JSP as a third-party convener of meetings and discussion. KIs reported that the JBF and JSP’s role gave the participants the courage to discuss difficult issues that they would otherwise avoid. Further, all relevant KIs recognized not only that these discussions would not have happened without JSP but that they will not occur in the future without JSP. JSP has supported the JC in preparation of a new strategic plan for the judiciary for 2014-2016. (The previous plan prepared under JRIP covered 2010-2012). JSP contracted with CIRa to facilitate the strategic planning process, which included members of the JC as well as LPAs and CSOs supported by JSP under Result 1. The strategic planning process included a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis to identify strengths and weaknesses. All KIs reported that the strategic planning process was collaborative and stressed the importance of focusing on the identified weaknesses. The evaluators were informed that the plan is close to completion, but the Team was given neither specific details nor materials. The JC is composed of 15 members that serve 6 year terms with the right to be elected for a second term. The President is elected by the members for a single two-year term.7 The current President will complete her presidential term in December 2014, though in accordance with the Law on the Judicial Council, she will remain on the JC to serve the balance of her 6 year term. The JC KI reported that it struggles to engage with other branches of the GoM on an equal basis. The Team learned that the GoM is becoming increasingly centralized and dominated by 6 The JBF includes the top policy and decision makers of the Macedonian judiciary; the President of the JC, the President of the Supreme Court, the Presidents of the four appellate courts, the Presidents of the Higher Administrative and Administrative Courts, the President of the Court Services Council, the Head of the Administrative Office of the Court Budget Council, the Director of the Academy, and the Presidents of the MJA and CAA representing the judges and court administrative staff, respectively. 7 Law on Judicial Council, 2010, Arts 8-9. 16 one party. To illustrate, the Parliament recently adopted a package of more than 100 laws on an expedited basis. Furthermore, the government leaders have used defamation suits against figures perceived as political opposition as well as the media. The package of laws rushed through the Parliament included a new Law on Court Service, which was prepared without any stakeholder consultation. KIs reported that the recent Law on Court Service came as a surprise to the JC, CSC and CAA. Prior to the unanticipated legislation, the CSC and CAA, with JSP support, had been reviewing the previous law and drafted their own amendments. They had attempted to present these drafts to the MoJ but without success. The MoJ KI maintained that the newly passed law would not undermine court administration, but rather apply the same terms to court staff as to civil servants. KIs noted that the GoM’s disregard of the judicial branch is also evidenced by the court budget. The judicial branch does not receive its legally-mandated budget allocation. Despite the requirement to increase the court budget from 0.4% to 0.8% of gross domestic product (GDP), it decreased to 3.9% in 2013.8 The budget also decreased in absolute terms as a result of lower GDP. KIs mentioned that the court budget will decrease further because the budget reflects the number of expected cases, which is declining due to the reduction in backlog. For Requirement 2.2, the Team found that: The JSP Year One Workplan called for an assessment of the status of judicial administration and management systems, procedures and capacity to identify procedural impediments and performance weaknesses, recommend changes, and develop a plan for implementation of changes over the course of the Project.9 The assessment, prepared by an international consultant, was completed in June 2012, nine months after project initiation. The assessment was based on a desk review, prior consultancies and general professional expertise.10 The key findings and conclusions of the assessment were: o There is no clear constitutional or statutory authority responsible for the administration of the Judiciary, which makes it exceedingly difficult for the Judiciary to plan, manage and control the delivery of justice and the rule of law in an efficient and effective manner. o Organization, staffing and operations of judicial administration are insufficient and incomplete, with significant gaps in covering the basic functions of a modern judicial administrative office at the central level. Existing offices are significantly short staffed, and there are low levels of competency with little training available.11 As the judiciary’s leadership challenges are discussed under Requirement 2.1 above, the following discussion covers progress under Requirement 2.2. The Court Services Council (CSC), established under the Law on Court Services, functions as an appeal body for decisions on hiring, termination, discipline and other rights of court service employees. Members of the CSC also sit on employment committees that make decisions on hiring court staff. The CSC is based in the Supreme Court and has 11 members: the Court Administrators for the Supreme Court (SC), Administrative and High Administrative Court, the four appellate courts and four rotating members from the basic courts with extended jurisdiction. The President is selected 8 Law on Court Budget, 2014, Art. 20. 9 JSP Year One Workplan, p. 9. 10 JSP Assessment of Administrative and Management Systems prepared by Dr. Robert Stout, June 28, 2012. Dr. Stout had several consultancies between 2006 and 2008 under JRIP (p.8). 11 Id, p. 2-3. 17 from among the members for a non-renewable two-year term, with the current president’s term expiring in 2014.12 The President of the CSC reported that there is only one administrative staff supporting the council. The President of the CSC recognized the critical role JSP has played in the development of the CSC and CAA (the President was a founding member of the CAA) and that continued support from JSP is critical to the functioning of the CSC and CAA. CSC activities that JSP supported include: o A court staffing study prepared by an international expert, which the CSC used to develop “Staffing Guidelines for the Courts of the Republic of Macedonia.” The Staffing Guidelines have been presented to the JBF,13 but the Team has no further information on the status of the Staffing Guidelines. o Preparation and revisions of job descriptions for information technology (IT) specialists, court administrators, and public information officers and a Manual for Hiring Court Employees. Relevant KIs acknowledged the value of the Manual and reported following the outlined procedure when hiring new employees. o Workshop on “Successful Transitioning, Transfer of Institutional Memory and Building Plan for Future Activities” for current and previous CSC members. The CSC President reported that the need for amending the Law on Court Services (LCS) was discussed during the workshop. However, before proposed amendments could be submitted to the MOJ, the Parliament passed a new LCS. Concerns were raised with several provisions, and the CSC President reported that the CSC and CAA are considering how to amend the most troublesome provisions before the new LCS goes into effect in Feb. 2015. They will look to JSP for support in this process. The CBC has a President (the President of the JC) and 10 members: the MoJ; Presidents of the Supreme Court, the Administrative courts, the Appellate courts, and two courts with extended jurisdiction on a rotating basis; and the Director of the JTA. The MoF is a non-voting member. The AO of the CBC carries out professional and administrative affairs and is a unit of the JC.14 CBC activities supported by JSP include: o Needs-based budgets supported by data from the ACCMIS and Automated Budget Management System (ABMS). Needs-based budgeting was piloted in six courts during Year 1 of the project, and the Team learned that it is now used in all courts. A JSP international expert conducted training for CBC members as well as other court presidents and administrators. The AO uses the individual court budgets to prepare the aggregate court budget, which is submitted to the Ministry of Finance. However, despite the requirement in the Law on Court Budget to increase the percentage of the GDP allocated to the judiciary’s budget, as mentioned above, the amount of the budget has declined both as a percentage and in absolute terms. Within this tight fiscal environment, JSP is supporting the AO’s efforts to rationalize the use of scarce 12 Law on Court Services, 2014, Section III. 13 JSP Monthly Report, Jan. 2014. 14 Law on Court Budget, 2014, Arts. 7 and 10. 18 resources through workshops to share best practices in organization of departments and re-distribution of personnel to balance workloads.15 o Standardize accounting practices and produce unified financial reports. JSP supported a workshop for selected accountants that produced “Guidelines for Unified Accounting Practices” and training for accountants at all courts on “Entering Court Assets in the Automated Budget System” and “Preparation of the Final Annual Account”. The AO now produces a consolidated annual account report. The Head of the AO reported that the AO is also now capable of conducting training of its staff and in the courts, although JSP support for organizing such events is still needed due to limited financial resources. Courts report different practices for dealing with public requests for information. Not all courts have filled their Public Information Officer positions, and 4 KIs reported that their courts still rely on a single judge who often handles high profile cases and interacts with the media. JSP and the JTA conducted training on “Free Access to Public Information/Communication with Clients in the Court” during Year 2, which was attended by court employees and judges. KIs reported that responding to Freedom of Information Act (FOIA) requests is not a problem if the request is presented in writing, which it normally is; however, they have experienced some challenges with oral requests, including from the media. Other JSP supported activities to improve transparency and communication with the public include: publication of a brochure for the JC, which was released in April 2013 and publication of the Annual Reports of the four appellate regions and the SC. At the request of the MJA, JSP is supporting preparation of a revised Code of Judicial Ethics that will reflect comments made by the Group of States against Corruption (GRECO) and provisions of the Laws on the Prevention of Corruption and Prevention of Conflict of Interests. The draft code was prepared by JSP and is begin reviewed by a working group composed of representatives of the MJA and JC and the Director of the Academy. JSP’s Year 3 Workplan includes continued support to the JC and its AO, continued leadership training for motivated judges, support for development of strategic plans for Basic Courts Skopje 1 and 2, and support for the establishment of an analytical unit within a Court Operations Support Office; analysis to be part of role to perform applied research and analysis with the aim of enhancing judicial efficiency and effectiveness with 3 pilots created in the JC, Basic Court Skopje 1 and an Appellate Court. No JC KI mentioned anything about the plan to establish the Analytical Office. RESULT 3: EFFECTIVE LEGAL PERSONNEL AND EFFICIENT PROCESSES Requirement 3.1: Develop and implement targeted specialized training to judges, court administrative executives, and other court staff. The trainings should be developed in cooperation with and input from justice sector professional associations and coordinated with the Academy for Judges and Prosecutors, and other relevant stakeholders and beneficiaries. Requirement 3.2: Improve caseload processing and reduce backlog of cases. 15 The model budget was part of the Functional Analysis for Improvement of the Effectiveness of the Courts in Macedonia conducted by the World Bank and Ministry of Justice in 2011. 19 All KIs reported excellent cooperation with JSP under Result 3. The Team found that judicial branch institutions consider JSP as a partner that is flexible, responsive to needs, and results-oriented. Furthermore, KIs reported that their institutions can rely on JSP to work with them to find solutions. JSP has been very responsive with support in the form of computer and other equipment, and financial support for events or working meetings. Examples include computers for the Judicial Training Academy (JTA), roundtables for IT staff and an upgrade of ACCMIS to accommodate processing of Higher Administrative Court cases. The Team found that JSP met or exceeded its targets under Result 3. The ratio of new cases to disposed cases exceeded its target.16 The number of trained justice sector personnel also exceeded its target. The number of legal courses or curricula developed met its target. For Requirement 3.1, the Team found the following: JSP has supported training activities and conferences for judges and court organized in cooperation with the JTA. o At the request of the AO of the CBC and the JC, a tailored training on court management for court presidents and court administrators was designed and conducted by Dutch expert Judge Bert Maan during Year 1. o A Judicial Leadership Conference attended by over 90 president judges, heads of departments and court administrators was held in Sept. 2013. Presentations were made by local and international experts from the US (JSP CoP Judge Traficanti and Judge Kavanagh) and the Netherlands (Judge Maan). KIs reported that this was a good opportunity for courts to share best practices. A court president and administrator also reported that the separate training on Alternative Dispute Resolution (ADR) was useful. o JSP collaborated with a local USAID project, Investment Development and Export Advancement Support Project (IDEAS), to implement commercial law training. The Team does not have any additional information on these trainings. JSP is also supporting the JTA’s role as coordinator and publishing home of appellate court jurisprudence. The JTA conducts a large number of trainings (272 in 2013) given the small staff size (Director and 16 full time staff) and relies heavily on external trainers. The JTA has a memorandum of cooperation with the MJA, MBA, CAA and PPA. These organizations provide trainers and content. JTA also maintains a roster of approved trainers. The MJA also conducts its own trainings under authority from the JTA, for which participating judges can receive continuing education credit. These trainings are discussed under Result 2. The JTA's portfolio expanded to cover the complex and varied training needed for court staff. The Director reported that although this training is conducted in cooperation with the CAA, the additional responsibility for training the large court staff as well as lay judges, is an excessive burden on the JTA. The Director also expressed concern about the quality of training. The only qualification for trainers is six years of directly-relevant position experience. The Director also stated that it is difficult to bring in high-quality experts because, in the case of judges, there is no release time or continuing education credit, and in the case of other experts (e.g. attorneys and 16 JSP should change either the word order within the indicator name or its ratio calculation. As the ratio is being reported currently, the number of new cases is increasingly greater than the number of disposed cases, which would mean increased backlog and less efficient justice. 20 professors), the honorarium is low. The JTA reported that it does not receive its full allocation of 2.5% of the judicial branch budget17 and that it relies heavily on donor support. KIs expressed mixed opinions on the quality and utility of the training at the JTA. The Team found that KIs were also concerned about the amount of obligatory training for judges (20 days per year for new judges and 3 days per year for experienced judges). Court presidents stated that 20 days is too much time for newly appointed judges to be out of the court. The JTA is considering reducing the requirement. The training methodology was a concern as many trainers do not practice adult learning techniques and often do not incorporate relevant cases in their training materials. The JTA does conduct its own Training of the Trainers (ToT) (two in 2013), but the Director expressed concern that the quality of the ToT experts had declined and that the JTA would benefit from a “refresher” ToT conducted by US experts. (Initial ToT courses were conducted by US experts under other USAID programs. Two ToTs were organized in cooperation with the Office of Overseas Prosecutorial Development, Assistance, and Training (OPDAT), the Organization for Security and Cooperation in Europe (OSCE) and the British Embassy in 2013, but this was in preparation for the upcoming training on the new Criminal Procedure Code.) While the JTA uses a post-training evaluation questionnaire on the organization of the training and trainers, which JTA does review in order to revise trainer practice (e.g. requiring training materials to be prepared and distributed in advance), the JTA does not currently test for learning or verify application of learning on the job. The JTA is, however, preparing to pilot post-training tests as part of its curriculum on the criminal procedure code. JSP has supported the JTA’s investigation of methods for evaluating effectiveness of judicial training through the translation of a Manual on Measuring the Impact of Judicial Training prepared by the International Network to Support the Rule of Law (INPROL). As noted in JSP’s Year 3 Workplan, the Program will support training programs at the JTA in priority areas (judicial writing, commercial law, domestic violence, leadership training), explore the usage of e-learning and facilitate preparation of a strategic plan. In addition to its support of JTA’s activities, JSP organized a study tour for a small group of court administrators, IT specialists and the Head of the AO to Slovenia and the Netherlands in Year 2, which focused on the main JSP interventions (court administration and management, case processing and backlog reduction, budgeting and finance, and IT). According to the JSP Year 2 Annual Report, JSP staff assisted participants in developing an Action Plan based on the information gained, which was presented to the JBF. The team does not have any additional information on the status of that Action Plan. As noted in JSP’s Year 3 Workplan, two additional study tours are planned for Year 3. For Requirement 3.2, the Team found the following: All KIs reported that case backlog has decreased and the time to disposition has been reduced drastically.18 As noted in the JSP Year 1 Annual Report,19 JSP supported improved case management by establishing a working committee of local experts to develop a model monthly report and model internal procedures that were tested at pilot courts selected by JSP. These models were approved 17 Law on Court Budget, 2010, Art. 4. 18 The President of the Stip Basic Court reported that the time to preliminary hearing was reduced from 12-18 months to 2-3 months. Basic Court Skopje 1 achieved clearance rate of 112% in 2013. 19 P. 15. 21 by the JC and presented to president judges, heads of department and court administrators at events supported by JSP. As noted in the Year 2 Annual Report, as follow-up, JSP has participated in working meetings with courts that encountered difficulties implementing the models.20 JSP is supporting efforts to introduce a differentiated case management (DCM) system for the Macedonian courts. JSP retained court administration expert, Marcus Zimmer, to prepare a detailed analysis and report on developing and piloting DCM in the Macedonian court system.21 The report includes a proposed framework for implementing a pilot DCM in two civil basic courts. The report also identifies key elements of a court system to sustain and build on DCM pilot efforts. These include a case management system with case tracking functionality (which ACCMIS could accommodate provided some upgrading), a culture of case settlement rather than tried cases, strong judicial support staff, and developing central case management and statistical research and analysis capacity. The report was presented and discussed at the JBF in June 2013. The Team found the report to be excellent; it is clear and thorough. Based on KIs responses, it is likely that some form of DCM will be adopted. JSP engaged a local expert to prepare an IT assessment report on utilization and satisfaction with ACCMIS, ABMS, ERS, and Judicial Council Case Management Information System (JCCMIS).22 The KIs at the JC found this report to be very good, timely, and with actionable recommendations, which, as noted below, JSP is supporting. The Team also found that the report was very high quality. All KIs had strong positive comments on the value of ACCMIS, although all mentioned the need to upgrade and revise the system. The KIs noted that JSP has financed the maintenance agreement with EduSoft, the AACMIS vendor, which runs through January 2016. The Team found that KIs were concerned about future maintenance without support from JSP or another donor, as the court budget is insufficient. The IT Assessment Report recommended several improvements in ACCMIS that have been supported by JSP (e.g. developing a database warehouse and statistical interface software, data backup, development of uniform nomenclature). As noted in the Year 2 Annual Report,23 JSP also participates in meetings of the Supreme Court Committee for Improving Use of ACCMIS and has financed upgrades to ACCMIS to enable processing of cases at the Higher Administrative Court. The Report noted the need for incremental replacement of old equipment, but the Team is not aware of funds promised by the GoM for the court budget for this purpose. JSP retained the developer of the ABMS to upgrade the system in line with legislative and regulatory changes. ABMS is now integrated with ACCMIS. Electronic Recording Systems (ERS) were installed in 80 courtrooms under JRIP, but were not utilized for a variety of reasons. This problem was addressed in the IT Assessment Report referred to above, which found that there were no problems with the hardware or software, that the system was user friendly, and that court staff should be able to operate the system without the presence of the IT administrators. The report recommended additional training for court staff so JSP retained a vendor to train staff in two pilot courts (Veles and Bitola Basic Courts), which has been extended to other courts. However, KIs in other courts still reported that IT staff must be present to ensure the system is on and functioning. KIs also reported that EDR devices are not installed in all courtrooms. 20 See p. 22. Basic courts Skopje I, Kumanovo and Ohrid. 21 Case Management Practices in the Macedonian Courts: A Proposal for Designing and Implementing Differentiated Case Management and Time Standards, March 2013. 22 USAID Macedonia/Judicial Strengthening Project, Assessment of Court IT Utilization (ACCMIS, ERCS, JCCMIS and ABMS), April 2012. 23 P. 22-23. 22 As noted in the Year 3 Workplan, JSP plans to review case processing activities by region and support improved productivity; enhance court ICT utilization, including through ToT; continue activities to enhance use of ERS and promote adoption of DCM. CONCLUSIONS ANALYSIS OF PROGRESS AND EFFECTIVENESS OF PROJECT INTERVENTIONS The Team synthesized the above large set of findings of fact to address the first cluster of four evaluation questions to draw conclusions about the effects of the set of project activities. This section addresses those conclusions. EVALUATION QUESTION #1: WHAT IS THE PROGRESS TO DATE ON THE FOLLOWING STATED PROJECT RESULTS? Achievement of results could be measured against either the M&E Plan or the Results Framework. Per instructions from the Mission, the evaluation has measured progress against the M&E Plan. Overall, JSP met the targets under the evaluation’s specified requirements, as seen in the project’s M&E Plan. JSP has maintained its M&E Plan, but narratives using in reporting documents sometimes place activities in different Results, as mentioned in the Result 2 discussion above. The indicators do not fully recognize or capture key project activities relevant to the overall goal of the project, however. The Team concluded that targets were not very ambitious, or were tailored to match actual conditions. Moreover, the Results themselves are at a much higher level of effect than the underlying activities could ever achieve. This is due in large part to the specification of results in the project contract, which are quite difficult for USAID to change, but also to the lack of an explicit and systematically applied theory of change. In contrast to the under-ambitious nature of the M&E Plan, the Results are over-ambitious with regard to the actual project activities. EVALUATION QUESTION #2: HOW EFFECTIVE HAS BEEN THE PROJECT APPROACH AND INTERVENTIONS IN ACHIEVING THE EXPECTED RESULTS TO DATE? At the project level, JSP has indeed been effective in its project interventions, but the overlying political context and weak judicial independence limits the effect these have on the higher goals of the project. In this evaluation, it was agreed that the Team would judge effectiveness in terms of three components: utility, efficiency, and sustainability. In terms of utility, JSP has definitely raised the capacity of CAA, MYLA, and even ELSA, but the MJA was not a cooperative partner. Training received by the first three organizations have been integrated into their management and activities. The brochures and other printed materials cannot be reproduced by the organizations for financial reasons, and had limited reach to the public. The project plays a vital role in connecting and ensuring collaboration among various judicial branch actors; it has fostered the development of bodies such as the JBF and ROLC to overcome persisting problems among the institutions with respect to coordination and planning caused by the inactivity of the Supreme Court. The CSC and AO of the CBC have become technically competent, and can fulfill their functions with JSP assistance. Despite JSP support, the JTA is struggling to meet training demands and quality assurance. JSP’s continued support for case management and IT systems (ACCMIS, ABMS, JCCMIS, and ERS) initiated under JRIP have allowed for necessary upgrades and better use of these systems by the courts and other judicial branch institutions. JSP’s activities to introduce EDR in some courtrooms is widely viewed by judges and court administrators as a needed advancement, but usage by court staff remains problematic. The DCM report was welcomed by the JC, and judicial system actors are now considering whether and how to implement it for judges. 23 In terms of efficiency, the modes of public engagement by LPAs are in turn passive (websites) or of such limited distribution (brochures, etc.) that there is little actual effect on public understanding or perceptions. As such, these do not represent the most cost-effective means of engagement. The selection of MJA for capacity-building training under Result 1 distracted attention and financial resources that could have been devoted to other, more appropriate organizations. JSP’s facilitation of the JBF is a faster way to support coordination among judicial branch actors in light of the Supreme Court’s dysfunction. The IT Assessment and DCM report were written by highly qualified experts with strong familiarity with the Macedonian judicial sector, and so brought together both international expertise and context sensitivity without the need for larger teams. The Judicial Leadership Conference, however, is a costly, labor-intensive, and limited impact activity. In terms of sustainability, the skills and knowledge from training provided to CAA, MYLA, and even ELSA has been adopted and reproduced by the organizations themselves. JSP is and will continue to be needed as a convener or facilitator for both judicial branch actors and LPAs, and for the JBF and ROLC efforts. CSC is completely dependent on JSP assistance and financial support. The CBC is capable of reproducing the skills and knowledge required for needs-based budgeting and other accounting analyses. Systems and equipment installed under the previous project and upgraded by JSP face budgetary challenges in maintaining and upgrading the system. Result 1 The utility of the activities under this result is high. The CIRa assessment and EMBRA training were accurate and relevant to the organizations' needs. Setting aside MJA, the associations participating in training found the trainings relevant, and the skills and knowledge gained from them were integrated into the organizations’ management. An unintended consequence was the trainings also provided a venue for networking and mentoring. CAA has become a capable organization, and would not have been as capable without JSP. MYLA has continued its prior path of building capacity, and it benefitted coincidentally from the timing of strategic planning during a period of leadership transition. Similar to MYLA, ELSA was undergoing a leadership change and thus embraced an unexpected opportunity to learn new skills for organization management. The MJA is not viewed as an effective and pro-active organization by any KI among the judges or other LPAs. The efficiency of the activities under this result is high, with two exceptions. First, the work on customized websites and printed brochures is a relatively expensive and one time contribution. Rather than custom design websites, sources like WordPress have thousands of free website templates that only require content. It is also evident that social media is much more relevant to MYLA and ELSA and potentially to CAA. As with the passive nature of websites as a communication tool, brochures that are distributed as takeaways on counters or at one-off events are similarly passive. Second, the selection of MJA for organizational capacity-building came at the expense of working with another organization that could have participated in trainings across the set of modules that also could have been more tailored for their needs. The sustainability of the activities under this result is high, with a few notes of caution. It was very clear to the Team that CAA and MYLA had integrated the skills and knowledge learned in training modules, and could and were reproducing them independently of JSP. The ELSA KI and MYLA KIs noted that ELSA had integrated some skills and knowledge, but the nature of ELSA as an all-volunteer student association with high turnover presented real challenges to true reproduction and sustainability. The Team concluded that caution is also warranted over the financial sustainability of CAA and ELSA. CAA may have approximately 900 members, but the membership dues are quite low. The organization does 24 receive revenue from its agreement with the JTA, which is positive given that professional associations around the world cannot rely on dues for financial survival.24 In addition, CAA’s ability to engage in larger, more substantive donor-funded projects is limited due to its necessary focus on policies critical to its members. ELSA is an all-volunteer group of students with extremely limited funding, and must work collaboratively with other organizations for free or in-kind contributions to its activities. Only MYLA is actively strategizing on how to develop market-oriented (fee for service) activities to diversity its funding base, largely by offering other CSOs basic capacity-building training. This is a promising innovation, and it is unusual for a rule of law or human rights organization; however, the true viability of such a strategy needs to be explored further. Given the questions regarding sustainability of the member organizations, sustainability of the ROLC is clearly uncertain. Result 2 The utility of the activities under this result is high. The JBF is a direct response to the lack of judicial branch leadership, itself a consequence of legislative and other changes that have taken place in recent years. By supporting the JBF, JSP performs a vital role in bringing together functionally separate actors for discussions on key judicial sector issues. The new strategic plan for the judiciary (2014-2016), developed in consultation with judicial branch institutions, LPAs and CSOs, will draw attention to weaknesses and provide a platform for developing solutions. It is an important step in unifying the institutions, LPAs and CSOs. JSP support to the individual judicial branch institution (CSC and CBC) was critical to building the capacity of these institutions to fulfill their mandates. Products of JSP activities are being used (e.g., procedures and models for needs-based budgets, models for reporting to the JC and SC, guidelines and a manual for hiring court staff). For example, the CSC and court administrators follow the regulations and manual in its human resources management role for court management. Activities designed specifically to build the ability and willingness of non-IT court staff to use ERS is necessary to ensure the usefulness of that system. The efficiency of the activities under this result is high. The JBF is an efficient mechanism for bring key policy makers together. Supporting small working groups or committees is also an efficient mechanism for drafting regulation and manuals and designing procedures in a collaborative manner. Support for preparing the new strategic plan for the judiciary was efficient as it extends beyond the scheduled end of JSP. Additionally, CIRa, the facilitator, utilized a well-structured approach. The Team notes that it is difficult to ascertain the efficiency of the project’s dominant activity of facilitation and direct assistance, as no benchmarks exist as reference points. The sustainability of the activities under this result is moderate. As the JBF is completely dependent on JSP’s support, which participants expect to continue for the foreseeable future, it is unclear if the JBF would continue without JSP. Further, the success of JBF depends, at least in part, on the individuals involved. However, with JSP’s continuing support for another year or more, combined with a concerted effort to shift responsibility to participants, the JBF might become sustainable. The same is true for strategic planning for the judiciary, although this activity could be adopted by JBF if that initiative succeeds. However there is no assurance the JBF would include LPAs and CSOs in the planning process. As mentioned above under utility, the results of other interventions are in use. Result 3 The utility of the activities under this result is a mixture of high to moderate. The IT Assessment Report on utilization and satisfaction with ACCMIS, ABMS, ERS, and JCCMIS was thorough, accurate and included recommendations acted on by JSP and the judicial branch institutions. The Report provided a 24 Cite JHU/CCSS figures here. 25 road map for interventions that built on the progress under JRIP and JCMP. System upgrades and modifications noted in the Assessment Report and supported by JSP have enhanced utility and user satisfaction, although usage of the ERS by court staff remains problematic. The report on implementing DCM was also of a very high quality, with a thorough discussion of the benefits of DCM generally and how it could be introduced in the Macedonian courts. The report was presented to the JBF and will be used to inform the development of a DCM for Macedonia. In contrast to these focused interventions, activities, such as the Judicial Leadership Conference have less direct impact. Instead, they provide opportunities for Macedonian judges and court administrators to learn from and share best practices among themselves and international experts. This is also the case for the study tour to Slovenia and the Netherlands for court administrators and IT personnel. Focused roundtables and coordination meetings for top IT personnel had greater impact as they produced tangible benefits such as developing model monthly reports and procedures. The efficiency of the activities under this result is a mixture of high to low. In the case of the IT Assessment and report on implementing DCM, utilizing highly qualified experts that were familiar with the Macedonian courts was an efficient way to collect large amounts of information in a format that could readily be presented to judicial branch leaders and decision makers, such as the JBF. Using pilot courts to test new systems and procedures and make necessary modifications and then using those courts as models for the rest of the court system was also efficient. Promoting in-house training offers additional efficiency gains although such measures require a means to monitor quality. Other activities, such as the Judicial Leadership Conference require a great deal of planning and organization, are costly, and have limited impact. The sustainability of the activities under this result is moderate. JSP addressed gaps and deficiencies in ACCMIS, JCCMIS, ABMS and ERS, and, except for continuing problems with the use of ERS, the systems are functioning at a high level. As noted by KIs and in the IT Assessment Report, the problem with ERS is that despite training, court staff resists using ERS in the absence of IT staff. In addition, maintenance and upgrades to these systems, particularly ACCMIS, are a major concern, as the maintenance contract with EduSoft, which JSP funds, ends in Feb. 2016. Further, as noted in the IT Assessment Report, a great deal of equipment used in the judicial branch is old and already outdated. Replacement of equipment can only be delayed for so long. It is unlikely that the court budget will be sufficient to cover system maintenance, upgrades, and replacement of equipment. The Team does not have any information regarding whether another program (such as IPA II) will cover these costs. EVALUATION QUESTION #3: HOW WELL IS THE PROJECT COMMUNICATING AND COLLABORATING WITH BENEFICIARIES AND COUNTERPARTS IN THE IMPLEMENTATION OF PROJECT INTERVENTIONS? JSP is in regular and collaborative communication with the various judicial branch institutions and relevant LPAs and CSOs. Partners and beneficiaries have tremendous respect for JSP, which has not been affected by the transition of the DCoP to CoP. The LPAs and CSOs explicitly valued JSP’s collaborative approach, saying that they did not view them as funders, but as partners. MYLA and ELSA pointed to the networking opportunities that JSP activities gave them, and ESE emphasized that JSP actively worked with them to engage judges on GBV. JSP’s cooperation and communication with the President of the JC and the Head of its AO as well as the President of the CSC are excellent. These individuals consider JSP as a partner. This is also true of the Presidents of the Appellate Court Presidents and pilot courts. JSP has helped develop two ‘talking shops’ to promote the interests of the judicial branch, the JBF, primarily intended for official institutions, and the new ROLC, primarily intended for LPAs and CSOs. The only participant overlap between the two is the CAA. KIs were unanimous in the opinion that important judicial sector topics would never be discussed or various judicial sector actors would never 26 meet without JSP facilitation. The current strategic planning process that began with a large set of interviews by CIRa is designed to enhance collaboration on judicial branch issues across judicial sector institutions and LPAs and CSOs, which essentially merges both demand- and supply-side interests. The breadth of JSP and its strong collaborative practice enables actors from within the judicial branch to act collectively, and allows for networking and sharing of information between the judicial branch and various NGOs. JSP is in regular contact with the President of the JTA with regard to specific training and other events organized in collaboration with the JTA. JSP has been responsive to specific requests, e.g., providing computers for training of candidates for judges and prosecutors and translating materials on measuring the impact of judicial training. EVALUATION QUESTION #4: BASED ON RESULTS TO DATE, IS THE PROJECT LIKELY TO ENGENDER SUSTAINABLE AND SYSTEMATIC DEVELOPMENT IMPACTS AFTER USAID FUNDING HAS STOPPED? WHAT SHOULD THE IMPLEMENTER DO TO ENSURE SUSTAINABILITY AND TRANSFER OF KNOW-HOW? At this point, there is a mixed picture with regard to sustainable and systematic development of the broader judicial sector. On one hand, there are key LPAs that are capable organizations thanks to JSP interventions. CAA and MYLA have directly benefitted from the assessment and resulting capacity￾building training designed for them. ELSA benefitted too, but as an all-volunteer student organization, it faces inherent challenges to sustainability. The ROLC is potentially unsustainable, but it is too early at this point to make a definitive conclusion. The JC, CSC, and AO CBC are technically capable, but clearly need JSP financial and/or technical support in order to fully meet their missions. JSP’s support to the JC in the strategic planning process is critical to not just the quality of the outcome, but also to the judicial branch’s need to actively consider the perspectives of all judicial branch actors. The establishment of the JBF is an innovative way to bypass the inactive Supreme Court, but again this forum is wholly dependent on JSP facilitation. Other operational interventions (ACCMIS, ABMS, etc.) will continue to be used in the future. Much of the equipment and systems have become outdated, however, and it was not clear to any KI where funding for upgrading could be obtained. JSP could improve the prospects for sustainability by working more closely with the Supreme Court, collaborating with efforts to implement the CPC, and coordinating with other donors to identify funding resources. Because augmenting the JC’s mandate would require legislative, if not constitutional reform, the Supreme Court needs to fulfill its role as the jurisprudence center for the judicial branch. Given that no judicial body has policy leadership authority, JSP could more actively engage both the JC and the Supreme Court to full this gap in framework legislation, although it should be noted that this would be a difficult and highly-political activity to undertake. The upcoming implementation of CPC represents a potential opportunity to draw on the significant funding resources committed by a variety of foreign donors. The criminal courts use the same systems as the civil courts, and the JC, CSC, and AO CBC have the same responsibility for both sets of courts. Given that the criminal courts benefit as much from the existing systems as the civil courts, it would be appropriate for these latest judicial reform initiatives to contribute to their ongoing functioning. IPA II funding could be used for systems upgrades, except that their availability is unknown, the money would flow through the MoJ and not the JC, and Macedonia has a poor rate of utilization of IPA funding. 27 RECOMMENDATIONS RECOMMENDATIONS FOR MODIFICATIONS TO IMPROVE PROCESS ELEMENTS This section details the Team’s recommendations for the second set of four evaluation questions, Evaluation Questions 5 – 8. These questions and the corresponding recommendations address process aspects of the program. EVALUATION QUESTION #5: GIVEN THE PROGRESS TO DATE, PROVIDE A RECOMMENDATION IF THE OPTIONAL ONE YEAR EXTENSION OF JSP SHOULD BE EXERCISED. The Team strongly recommends that USAID/Macedonia exercise the one-year option for this project. There are several interventions that should be continued during the extension period:  Support for the ROLC could come at an important point in its development, although it is important to note that this is a new and emerging activity for JSP. As a result of the planning retreat facilitated by the Kaizen consultants, the ROLC has agreed on a common purpose (citizen education) and advocacy activities for its first year. JSP should follow Kaizen’s advice and allow the members to do the work, so as not to create another unsustainable, donor-supported initiative. JSP should act as an advisor and join the advisory body if and when the ROLC establishes one.  Continue focused capacity building and mentoring for selected LPAs and CSOs, which are also at an important point in their development. The training provided under Result 1 during the first two years of JSP was at a relatively basic level, and some LPAs and CSOs recognize that they would benefit from more advanced training and mentoring from JSP. The ROLC members are clearly candidates. With the 2013 leadership change, the MBA may also be a good candidate, inasmuch as they participated sporadically in the past under the previous leadership. MYLA will undertake a new round of strategic planning in the summer and would likely benefit from further capacity-building in support of new or expanded goals. CAA would benefit from advanced topics, as it faces potential turnover in its leadership and a potentially difficult struggle to mitigate the most negative effects of the recent amendments to the Law on Court Service. The MJA may experience a leadership change in its June 2014 assembly, so JSP should be prepared to engage this organization anew.  Continue support for JBF. The JBF serves an important purpose by providing an opportunity for open discussion between judicial branch institutions. Nonetheless, it is completely dependent on JSP – in part because of tensions between the JC and Supreme Court – and JSP should initiate a frank discussion of sustainability and shifting shared responsibility to participating institutions in future years.  Engage the Supreme Court. As previously discussed, the Supreme Court has not played a leadership role in the judicial branch for several years due to vacancies on the court and other reasons. However, as the President now has her permanent appointment and vacancies have been filled, there is an opportunity to expand cooperation with the Supreme Court in activities such as harmonization of judicial decisions, the DCM working group mentioned below, and the JBF.  Support the CSC and CAA in advocating for amendments to the new LCS. The new law, which was passed under expedited procedure without any stakeholder consultation, could undermine the important role of JSP and JRIP supported court administrators and negatively impact court administration and court employees as a whole. JSP should use its position to rally support from 28 the JBF, international organizations, and the MBA to apply pressure on the MoJ to respect stakeholder consultation. Successful pressure would offer a good test of the ability strengthened advocacy and citizen participation to influence judicial sector reforms.  Follow through on the recent DCM analysis. The report prepared by Marcus Zimmer describes implementation of a DCM pilot and the groundwork that must be completed to ensure the likely success of the DCM pilot.25 Although the groundwork as well as establishing the DCM pilot is included in JSP’s Year 3 Workplan, this will be a lengthy process. Given that introducing DCM will be another fundamental change for the Macedonian judiciary, it is not clear whether the pilot project could be completed within the extension period. USAID and JSP should consider whether a donor project (USAID or otherwise) will be available to continue to support DCM after JSP finishes. Therefore, in order to build ownership within the judicial branch from the outset, JSP should immediately follow-up on Zimmer’s recommendation to create an ad hoc DCM steering committee or working group. EVALUATION QUESTION #6 AND #8: DOES THE PROJECT APPROACH NEED TO BE MODIFIED IN ORDER TO REFLECT THE CURRENT JUSTICE SECTOR ENVIRONMENT? IF SO, HOW? WHAT ALTERNATIVE APPROACHES EXIST WHICH COULD LEAD TO BETTER RESULTS AND GREATER COST EFFICIENCY? Overall political and judicial context has not changed significantly since the design and start-up of JSP. The dominance of the executive branch has continued and may even be worsening, as exemplified by the recent flood of expedited legislation, of which the new Law on Court Service was a part. Further, the Government persists in using the law for its own purposes, as evidenced by the use of defamation suits for punishment of political opposition and the media. The judicial sector is more capable in operational and administrative respects, primarily as a result of JSP assistance, but it lacks the budgetary resources to meet its own internal needs. The GoM has not complied with its commitment to raise the court budget over time, and the budget has actually decreased in absolute terms. Judicial sector institutions are unsure about the availability of resources to maintain improvements initiated and supported under JSP and previous projects. At this time the judicial sector faces problems in advocating for its own interests and appears unable to sustain the supply side of reform, as foreseen in the JSP design. These represent constraints that the JSP has only limited ability to address. One potential modification to the project approach would be to engage more with the Supreme Court to strengthen the supply side aspects of the JSP. One of the key constraints noted above was the lack of clarity over who has responsibility for policy leadership within the judicial branch. As such, it would be desirable for the JSP to participate in and/or promote the drafting of legislation that explicates policy leadership within the judicial branch. The demand side is limited by the small number of LPAs/CSOs interested in promoting reform. Although organizational capacity has improved as a result of JSP assistance, the missions of the LPAs are primarily to serve their members. Larger advocacy efforts to raise public awareness of judicial sector issues involve passive modes (websites) and limited-audience distribution of printed materials. Neither of these could reasonably be expected to change broader public perceptions, which are shaped much more by media coverage of high-profile cases. Furthermore, piecemeal efforts at public outreach will be further drowned out by 25 See Zimmer Report, Section 4. 29 future messaging as Macedonia transitions from an inquisitorial to accusatorial system. As such, the project should eliminate or completely revamp its activities in this area. The JBF and ROLC, both new initiatives, may have the latent capacity to support the supply/demand approach of JSP. Therefore, continued support should be provided, especially support for implementation of the nearly completed Strategic Reform Plan 2014-2016, which merges the interests of the demand and supply sides. The Strategic Reform Plan should inform JSP’s work during the extension period in two ways. First, the Plan will likely produce new opportunities that would benefit from JSP support, and second, JSP activities should include rigorous monitoring of the Plan’s implementation. Overall, the supply/demand approach is still viable, but more effort is needed to improve the supply of justice, and more modest expectations should be adopted with respect to creating demand for justice. EVALUATION QUESTION #7: WHICH SELECTED ACTIONS AND CROSS-CUTTING THEMES AND CORRESPONDING ACTIVITIES SHOULD BE FURTHER EMPHASIZED, MODIFIED OR ELIMINATED AND WHY? The three cross-cutting issues of sustainability, EU alignment, and serving gender and minority communities receives more focused attention in JSP’s Year Three Work Plan. As a result there has been only a partial period of activity to analyze in this evaluation. Ownership/Sustainability As noted in multiple discussions above, the ownership and sustainability of judicial branch independence, effectiveness, and accountability is questionable, and should be further emphasized more in the remaining option year and beyond. The judicial branch actors rely on JSP to help them coordinate, discuss, and even act collectively, and they do not foresee this happening in the future without JSP’s assistance. This is due in part to the budgetary situation but also to a lack of clear constitutional authority on judicial branch leadership (See recommendation above). It is clear to the Team that the overall political context is the primary factor limiting judicial branch independence, as seen in the budgetary allocations and revisions in the framework legislation without judicial branch participation. These challenges go beyond the scope of what a project alone can achieve and call for a deliberate, concerted effort on the part of the international community to pressure the GoM to augment the authority of the judicial branch and ensure it the resources needed for meaningful independence. EU Alignment A slowdown of EU activities in Macedonia (and in other states moving toward EU integration) has decreased the impetus for the GoM to engage in judicial reform actions that would strengthen this branch of the state. While the EU will have significant financial resources in the upcoming IPA II funds targeted on judicial sector issues, the importance of successful implementation of the CPC will be the main area. For JSP, no modifications in its existing activities are necessary. Support for activities such as the minor ECHR-related efforts by CSOs and occasional JTA-based training is sufficient. Elimination of this cross-cutting theme is not recommended, for both diplomatic reasons and practical reasons (strong interests by some of JSP’s direct beneficiaries). Further progress on EU alignment would have to result from the higher diplomatic actions discussed above. Gender and Minority Communities JSP’s engagement with gender or minority community issues has not been significant or integrated well within the project. The ESE work has been critical for that organization’s ability to interact with judges on key GBV issues and would not have happened otherwise. It is not clear to the Team how or even whether it was desirable for gender or minority community activities to be expanded within the current structure of the project. Engagement with minority community issues have been marginal, due in large 30 part to the existence of quotas within the judiciary for minority communities. Without a more coherent rationale or theory of change to support inclusion of this cross-cutting theme, the Team recommends eliminating these activities. 31 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK FOR THE EVALUATION OF USAID/MACEDONIA’S JUDICIAL STRENGTHENING PROJECT I. SUMMARY The United States Agency for International Development (USAID) in Macedonia seeks the services of a Contractor to perform a mid-term evaluation of the USAID-funded Judicial Strengthening Project (JSP) project in February/March 2014. II. BACKGROUND Since 1993, USAID has been providing assistance for the strengthening of rule of law in Macedonia. While the early efforts mainly tackled the establishment of legal professional associations, street law and clinical education at the law faculties, in 2002 USAID shifted its assistance focus to the reform of the judiciary. Macedonia began an ambitious program to reform its justice system nine years ago. Considerable donor assistance was made available to facilitate these reform efforts. In November 2004, the Government of Macedonia enacted the National Strategy for Reform of the Justice System. The comprehensive assistance provided, as well as the tangible and measurable results accomplished in meeting the milestones envisioned in the Strategy to date, established the USAID as the key donor supporting the judicial branch reforms in Macedonia. USAID’s key accomplishments so far include the creation of a legal framework for the judicial reform; development and deployment of a system for automated processing of all types of cases from filing through disposition in all courts; establishment of a reliable system for enforcement of civil court decisions; civil procedure reform that streamlined the processing of civil cases and increased the court efficiency in the civil area; reduction of backlogs of old cases; and renovation and furnishing of court facilities. Despite the achievements, there remain a number of challenges, which, if not addressed, can undermine the progress that has been made to date. The key issues include: Independence of the judiciary: The institutions within the judiciary are still vulnerable to politicization by the executive branch of government and to personal agendas of powerful individuals. The independence of judicial decision-making is not free of political and other influence. The judges are perceived as susceptible to various types of pressure in deciding cases. There is a lack of proper institutional safeguards in the system to protect judges from punishment based on the substance of their rulings. Weak, fractured and splintered legal professional associations and civil society: The key justice sector professional associations have not engaged in a meaningful advocacy activities nor have they played a proactive role in the justice system reform efforts. Additionally, citizens’ interest in and engagement in demanding, monitoring and supporting justice sector reform and rule of law is low. 32 Court administration and management: The governance systems and management capacities required to operate a modern court system, with its many demands are still underdeveloped. Lack of participation of justice sector authorities and professional associations in formulation of laws and policies: As in many other sectors, the laws are drafted and brought to Parliament without a substantial participation of crucial stakeholders; in other words the process by which the laws governing the judiciary are enacted, administered, and enforced is not accessible, fair, and inclusive. Lack of communication/coordination within the judicial sector: The court system suffers from a lack of communication and collaboration among the several bodies that have been established by law to deal with the work of the courts. Lack of public trust: According to most national opinion polls the trust in the legal system is low. The lack of public confidence seriously jeopardizes the legitimacy of the judicial branch institutions and the overall judiciary. Public information and transparency: The lack of openness and transparency of the Macedonian judicial system has been another factor undermining public trust in the judiciary. Implementation of laws: Legal reforms of the past years have created a comprehensive legal framework. However, Macedonia has been plagued by the inability to implement and enforce its laws. In a legal culture where every act needs to be regulated by law, the system breaks down when the law is not enforced. To address these issues, the Mission implements the Judicial Strengthening Project, administered by Tetra Tech DPK. The project was authorized with the Contract No. AID-165-C-12-00001. The authorized period of performance is three years at $3.7 million, with a completion date of November 14, 2014. The Contract provides an option for an additional one year extension at $790,000, with a term option November 15, 2014 – November 14, 2015. The JSP builds on and furthers the accomplishments of two previous USAID judicial reform projects – Macedonia Court Modernization Project (2002 – 2007) and Judicial Reform Implementation Project (2007 – 2011). The overall objective of the JSP is to strengthen the role of the judiciary in the separation of powers; and to develop broad based support for rule of law and judicial sector reform. Its three key focus areas are: Strengthen Advocacy and Citizen Participation in Judicial Sector Reforms – The project supports the development of justice sector professional associations’ capacities to play a leadership role in justice system reform efforts, to advocate for the interest of their members; and to increase public awareness and participation in the justice sector reforms and the rule-of-law issues. The ultimate goal is to establish a broad coalition of NGOs, legal professional associations and reform minded individuals that will advocate for and monitor judicial branch independence. More Independent, Efficient and Consistent Application of Judicial Sector Policies and Practices – The goal is to increase the independence of the judicial branch by developing effective court governance systems and practices; increasing the role of the judiciary in preparing laws and policies governing the branch; and establishing regular coordination and consensus building on key issues among judicial institutions. The project promotes more accountable and transparent operations of judicial sector institutions and courts. 33 Effective Legal Personnel and Efficient Processes – To improve efficiency and quality of justice the project provides specialized trainings to judges and supports on developing and applying national time standards for case processing. The JSP also builds the capacities of court personnel to provide efficient services to court users. III. PURPOSE OF THE EVALUATION The purpose of the evaluation is to provide USAID with an external assessment of the JSP that will be used to determine if this program concept is working and meeting its expected results. It will also inform the Mission’s decision on exercising the optional one year extension. The evaluation will: 1. Analyze the progress and effectiveness of the project interventions toward achieving its stated program results (as defined in the Contract) to date 2. Recommend potential modifications for improvement IV. SCOPE OF WORK A. Analyze the progress and effectiveness of interventions toward achieving its stated program results (as defined in the Contract) to date 1. What is the progress to date on the following stated project results: Result 1 –Strengthen Advocacy and Citizen Participation in Judicial Sector Reforms Requirement 1.1: A short focused assessment, baseline survey, and a resulting action plan on justice sector professional associations and civil society organizations and the role they can play in strengthening the rule of law in Macedonia. Requirement 1.2: Per the findings of Requirement 1.1, the capacity of at least two judicial sector professional associations are improved so that they can better serve their members and are better equipped to play an advocacy role in justice sector policy-making, and assume the ownership of reforms by generating the demand for change among its members and in justice sector institutions themselves. Result 2 – More Independent, Efficient and Consistent Application of Judicial Sector Policies and Practices Requirement 2.1: Legal framework and judicial branch policies strengthen independence, effectiveness and accountability of the judicial system. Requirement 2.2: Administration and management rules, policies, procedures, systems and practices support a modern court system. As part of this requirement, the Contractor will work with the judicial sector authorities and actors to establish effective governance and operational systems for managing court resources – budget, human, facilities, equipment, etc.; strengthen managerial capabilities and introduce performance standards. Result 3 - Effective Legal Personnel and Efficient Processes 34 Requirement 3.1: Develop and implement targeted specialized training to judges, court administrative executives, and other court staff. The trainings should be developed in cooperation with and input from justice sector professional associations and coordinated with the Academy for Judges and Prosecutors, and other relevant stakeholders and beneficiaries. Requirement 3.2: Improve caseload processing and reduce backlog of cases. 2. How effective has been the project approach and interventions in achieving the expected results to date? 3. How well is the project communicating and collaborating with beneficiaries and counterparts in the implementation of project interventions? 4. Based on results to date, is the Project likely to engender sustainable and systematic development impacts after USAID funding has stopped? What should the implementer do to ensure sustainability and transfer of know-how? 5. Given the progress to date, provide recommendation if the optional one year extension of JSP should be exercised? B. Recommend potential modifications that can enhance the effectiveness of the project for the remaining performance period. 6. Does the project approach need to be modified in order to reflect the current justice sector environment? If so, how? 7. Which selected actions and cross-cutting themes and corresponding activities should be further emphasized, modified or eliminated and why? 8. What alternative approaches exist which could lead to better results and greater cost efficiency? V. METHODOLOGY Given the short time frame and mid-program nature of the evaluation, USAID expects the methodologies will be heavily qualitative. For example:  review and analyze the existing performance information in the subject area;  conduct site visits;  meet and interview USAID project beneficiaries, partners, host government counterparts at appropriate levels;  interview USAID staff and a representative number of experts working in the sector. USAID / Macedonia will provide the below documents to assist the team members in familiarizing themselves with JSP and USAID’s previous assistance in the judicial sector.:  USAID Request for Proposals and Tetra Tech DPK’s Project Proposal;  USAID Contract with Tetra Tech DPK; 35  The JSP reports and materials: annual, quarterly and monthly reports; annual work plans; consultancy reports, assessment reports and other relevant materials;  Final Reports of two previous USAID judicial reform projects;  EU Progress Reports - chapters regarding the judiciary VI. TEAM COMPOSITION Evaluation Team Members: The lead evaluation team member(s) should have at least 10 years of experience evaluating or working on development assistance programs and significant experience evaluating judicial and legal reform programs. Prior experience working in Central and Eastern Europe is desired. The team should contain a Macedonian team member. The local team member should have excellent understanding of the Macedonian court system, judicial structure and functioning, as well as the overall legal system and be fluent in English. S/he should be able to establish contacts and communicate effectively with judicial branch authorities, government officials and non-governmental sector representatives. Prior assessment and research experience in the subject area is desirable. Knowledge of foreign assistance program goals and aims in Macedonia would be helpful. In addition, the team will be joined by a USAID person from a different mission to both contribute to the evaluation and use it as a learning curve. All team members should have a very strong understanding of rule of law and judicial sector issues. Knowledge of USAID and other donor assistance in the justice sector is desired. The translator should be included in the evaluation budget (will not be provided by the mission). VII. SCHEDULE AND DELIVERABLES The evaluation effort should commence in late February 2014. USAID/Macedonia will provide the team with input and guidance in setting up a schedule of interviews and site visits, but the responsibility for the schedule resides with the Contractor. The schedule should be defined before the Team Leader arrives in country and should be finalized as soon as possible after the full Team is assembled in Macedonia. The draft schedule should be ready for review and discussion at the initial team planning meeting with USAID, which should take place within two days of when the team first convenes in Macedonia. Prior to beginning of the field work in Macedonia, all team members will review background program documents to gain a firm understanding of the situation in Macedonia and the USAID Judicial Strengthening Project. The team will interview key USAID and Project staff, beneficiaries of USAID’s assistance, representatives of the judiciary, legal professional associations, rule of law NGOs, other appropriate donor organizations providing assistance in the sector, and other program stakeholders. The following is an illustrative list of those to be interviewed by the team:  Relevant USAID staff and project staff  Judicial branch officials (President of the Judicial Council; President of the Court Service Council; Director of the Academy for Judges and Public Prosecutors; Head of the 36 Administrative Office (AO) of the Judicial Council; Head of the Financial Sector of the AO)  Courts (President and Court Administrator of the Supreme Court; presidents of all four appellate courts; presidents and court administrators of selected basic courts)  Legal professional associations and justice sector civil society organizations (Macedonian Judges Association, Macedonian Young Lawyers Association, Court Administration Association, European Law Students Association, Macedonian Lawyers Association, Chamber of Enforcement Agents, BAR Association, NGO Coalition All for Fair Trials, etc.)  Other donors providing assistance in the area including INL, OPDAT, ABA/ROLI, and EU Mission.  Ministry of Justice representatives (Deputy Minister of Justice and Head of Justice Sector) The Contractor is encouraged to identify and visit additional Macedonian organizations and groups, both formal and informal, based on its review of materials. Proposed schedule: Week of February 24th – 1) conduct literature review, schedule upcoming meetings, and arrange logistics in Macedonia; 3) Draft schedule submitted to USAID COR. (Deliverable 1) Week of March 3rd – 1) Start of the field work; 2) Kick-off meeting with USAID with the goal to establish clear expectations about the outcomes of the assessment and go over the goals, schedule and methodology of the assessment; 3) At the end of the week brief USAID orally on the progress and findings to date. (Deliverable 2) Week of March 10th – 1) Second week of field work; 2) At the end of the week the Contractor is required to submit a detailed outline of the evaluation to USAID for discussion and comment, as well as brief USAID orally on the key findings and recommendations. o/a March 21st – Contractor shall finalize the Evaluation and submit a penultimate draft to USAID. (Deliverable 3) a. Evaluation Report. The following sections shall be included in the document: i. Table of Contents ii. An Executive Summary – (3- 5 pages) containing a clear, concise summary of the most critical elements of the report, including the recommendations. iii. Evaluation Findings (no more than 15 pages), which provide analysis and answers the questions listed above in Section IV Scope of Work. iv. Detailed Recommendations and their potential impacts v. In addition, the Report has to meet the high quality standards from the Checklist for Assessing USAID Evaluation Reports 37 vi. Report Appendices, including: o A copy of the evaluation Statement of Work; o Cross-reference guide listing the evaluation questions from Section IV and specifying on which page the questions are answered in the report. o Team composition and study methods (1 page maximum); o A list of documents consulted, and of individuals and organizations interviewed; and o More detailed discussions of methodological or technical issues as appropriate. o/a March 27th - USAID will provide the Contractor with final comments. o/a April 1st - The Contractor shall incorporate all comments and submit the final Evaluation report to USAID. (Deliverable 4) The USAID/Macedonia and USAID/DRG will be responsible for review and approval of the final document. The Contractor shall be responsible for providing the final deliverables to USAID/DRG by email. The Contractor shall also provide an electronic copy to DEC, the database of the USAID Development Experience Clearinghouse (DEC) in accordance with normal AID/W requirements. http://dec.usaid.gov 38 ANNEX II: TEAM COMPOSITION AND STUDY METHODS A. Team Composition The interdisciplinary Evaluation Team consisted of Team Leader and Lead Performance Evaluator, Dr. Andrew Green; Lead ROL Evaluator, Ms. Marilyn Zelin; and Local ROL Evaluator from the USAID Mission, Ms. Jasna Kilalic. Together they bring expertise in: (1) Macedonian and the region, (2) justice sector reform and rule of law promotion, and (3) performance evaluation methodologies. Andrew Green, PhD, MA, BS (Lead PE Evaluator, Team Leader) brings nearly 20 years of experience in democratic development, civil society, political parties, rule of law and elections with a deep background in post-communist democracy and governance (DG) development from both the academic and applied worlds. He has a long history of engagement with monitoring and evaluation (M&E) projects in rule of law and in the region. Dr. Green is Social Impact’s DRG Practice Leader, the Indefinite Quantity Contract (IQC) Manager for DG Analytical Services III IQC, the IQC Manager for Strengthening Deliberative Bodies IQC, and the SI point of contact under DAI’s award for the Rule of Law IQC. Dr. Green holds a Ph.D. in Political Science from the University of Illinois at Urbana￾Champaign. Marilyn Zelin, JD, BA (Lead ROL Evaluator) is an attorney with over 25 years of experience, including 15 years of field experience in the Europe/Eurasia and MENA regions. She has specialized experience in justice sector reform, legal profession reform, legal education reform, advocacy and access to justice, and anti-corruption. Ms. Zelin served as the Political Scientist Evaluation Team Member on the 2013 mid-term performance evaluation of USAID/Serbia’s Judicial Reform and Government Accountability Program, which utilized a combination of key informant interviews, focus group discussions, and document review to identify assistance gaps and opportunities to improve the JRGA program. Previously, Ms. Zelin served as Country Director for ABA/ROLI projects in Georgia, Macedonia, and Kosovo and has additional rule of law program implementation experience in Tajikistan, Lebanon, Slovakia, Poland, Hungary, and Croatia. Jasna Kilalic, (Local ROL Evaluator) brings more than seventeen years of professional experience in the USAID Democracy and Governance Office, working on substantive Democracy programs in both rule of law and media. In that capacity, Ms. Kilalic has served as a Cognizant Technical Officer (CTO) for DG activities; provided technical direction to Chiefs of Party and DG implementing partners; provided expertise on rule of law and media issues to USAID and State Department colleagues; and monitored, evaluated, and assessed DG activities and sub-sectors. She has extensive experience and knowledge of the local rule of law climate. Ms. Kilalic is a graduate of Sarajevo Law Faculty. B. Study Methods For a detailed description of the study methods, please see the Evaluation Methods and Limitations section in the body of this report. The text provided in the body of the report provides a full description. 39 ANNEX III: DATA COLLECTION INSTRUMENTS A. Evaluation Matrix Documents & Other Materials Semi-Structured Interviews of Key Informants USG Implementing Direct Beneficiaries Indirect Beneficiaries External Proj Secondary JSP Partners Experts Result 1: Strengthen Advocacy and Citizen Participation in Judicial Sector Reforms Req1.1 Assistance to other selected organizations X X EMBRA MJA, MBA, PPA, CAA, CEA, MLA, MYLA, AFFT Judges Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB Req1.2 Build Capacity of 2+ judicial sector professional associations X X CIRa Council (MYLA, ELSA, AFFT, CAA); MBA; CEA, NGO ESE Members of MBA, MYLA; CSO members of AFFT Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB Result 2: More Independent, Efficient and Consistent Application of Judicial Sector Policies and Practices Req2.1 Legal framework & judicial branch policies strengthened X X Judicial Branch Forum; CBC AO, CSC, CAA Judges; budget leaders; court administrators Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB Req2.2 Administration and management support a modern court X X CIRa Judicial Council, CBC AO, Basic/Appellate courts; JTA; judges, court administrators, study tour Media; judges Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB 40 system participants, public information officers Result 3: Effective Legal Personnel and Efficient Processes Req3.1 Specialized training to staff of judicial institutions X X JTA, BC Sk 1&2; president judges, judges, study tour participants Judges; MBA members Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB Req3.2 Improve caseload processing and reduce case backlog X X ACCMIS Cmte; AC president judges, judges, court administrators, other/ICT staff, CBC/AO, JC; appellate court judges Judges; Members of MBA, MYLA; court administrators Former policymakers; law experts; Swiss, UNDP, OSCE, EU, WB 41 B. Semi-Structured Interview Questions JSP/Partners Direct Beneficiaries Indirect Beneficiaries External Sources EQ1: What is the progress to date on the following stated project results? [Questions tailored to activities under each result/requirement on examples, facilitated/assisted use of skills/experience, and independent use/adaptation of skills/experience] [Questions tailored to activities under relevant result/requirement on examples, facilitated/assisted use of skills/experience, and independent use/adaptation of skills/experience] [Questions tailored to activities under each relevant result/requirement on facilitated/assisted use of skills/experience, and independent use/adaptation of skills/experience] [Questions tailored to activities under each relevant result/requirement on independent use/adaptation of skills/experience] EQ2: How effective has been the project approach and interventions in achieving the expected results to date? What internal challenges or obstacles affected the project’s approach? What external challenges or obstacles affected the project’s approach? What risk assumptions were foreseen and which were not? Could you provide specific examples of ways in which JSP’s approach or interventions/activities did not address professional realities? Could you provide specific examples of challenges or obstacles outside JSP that affected its activities? Could these have been foreseen? Could you provide specific examples of challenges or obstacles that would have affected a program designed to strengthen the judicial sector? Could you provide specific examples of ways in which JSP’s approach or interventions/activities did not address professional realities? Could you provide specific examples of challenges or obstacles outside JSP that affected its activities? Could these have been foreseen? EQ3: How well is the project communicating and collaborating with beneficiaries and counterparts in the implementation of project interventions? Please provide specific examples how JSP engages with [key partners] and [direct beneficiaries]? What about with any [direct beneficiary] members or constituencies? Please provide specific examples how JSP engages with [key partners] and [direct beneficiaries]? What about with any [direct beneficiary] members or constituencies? Please describe how your [institution] or [organization] solicited your engagement in designing training, planning, or other activities? Please provide specific examples how JSP engages with [key partners] and [direct beneficiaries]? What about with any [direct beneficiary] members or constituencies? 42 JSP/Partners Direct Beneficiaries Indirect Beneficiaries External Sources EQ4: Based on results to date, is the Project likely to engender sustainable and systematic development impacts after USAID funding has stopped? What should the implementer do to ensure sustainability and transfer of know￾how? Could you provide examples of what skills/expertise JSP has imparted to direct beneficiaries that is still in use, has been adapted for other uses, or is being used in stand-alone training? Could you provide examples of processes or systems still in use? What do the direct beneficiaries still need from JSP? Could you provide examples of what skills/expertise JSP has imparted that is still in use, has been adapted for other uses, or is being used in stand-alone training? Could you provide examples of processes or systems still in use? What do feel you still need from JSP? Has it been your direct experience that [examples from JSP/Partners and Direct Beneficiaries] could be continued in the future? What do feel judicial sector institutions or LPAs still need from JSP? Has it been your direct experience that [examples from JSP/Partners and Direct Beneficiaries] could be continued in the future? What do feel judicial sector institutions or LPAs still need from JSP? EQ5: Given the progress to date, provide recommendation if the optional one year extension of JSP should be exercised? [Derivative of findings of fact and conclusions in EQ 1-4, 7] EQ6: Does the project approach need to be modified in order to reflect the current justice sector environment? If so, how? [Partially derived from findings of fact and conclusions from EQ2 and 4] In what ways has the sector changed to make current JSP activities more or less effective? Desired activities? [Partially derived from findings of fact and conclusions from EQ2 and 4] In what ways has the sector changed to make current JSP activities more or less effective? Desired activities? [Partially derived from findings of fact and conclusions from EQ2 and 4] In what ways has the sector changed to make further professionalization and reform more or less effective? [Partially derived from findings of fact and conclusions from EQ2 and 4] In what ways has the sector changed to make further professionalization and reform more or less effective? 43 JSP/Partners Direct Beneficiaries Indirect Beneficiaries External Sources EQ7: Which selected actions and cross￾cutting themes and corresponding activities should be further emphasized, modified or eliminated and why? [Ownership/Sustainability partially derived from findings of fact and conclusions from EQ2, 4, 6] Are there JSP activities that have not generated sufficient ownership or sustainability? Are there activities that are not currently but could be taken over by direct beneficiaries? What EU requirements have been or need to be met? What specific tasks could benefit from JSP technical assistance? To what specific extent have gender and minority community interests been incorporated in the judicial sector? What specific tasks could benefit from JSP technical assistance? [Ownership/Sustainability partially derived from findings of fact and conclusions from EQ2, 4, 6] Are there JSP activities that are not currently but could be taken over by [judicial institutions or LPAs]? What EU requirements have been or need to be met? What specific tasks could benefit from JSP technical assistance? To what specific extent have gender and minority community interests been incorporated in the judicial sector? What specific tasks could benefit from JSP technical assistance? [Ownership/Sustainability partially derived from findings of fact and conclusions from EQ2, 4, 6] What EU requirements have been or need to be met? What specific tasks could benefit from foreign technical assistance? To what specific extent have gender and minority community interests been incorporated in the judicial sector? What specific tasks could benefit from foreign technical assistance? [Ownership/Sustainability partially derived from findings of fact and conclusions from EQ2, 4, 6] Are there JSP activities that are not currently but could be taken over by [judicial institutions or LPAs]? What EU requirements have been or need to be met? What specific tasks could benefit from JSP technical assistance? To what specific extent have gender and minority community interests been incorporated in the judicial sector? What specific tasks could benefit from JSP technical assistance? 44 JSP/Partners Direct Beneficiaries Indirect Beneficiaries External Sources EQ8: What alternative approaches exist which could lead to better results and greater cost efficiency? [Partially derived from findings of fact and conclusions from EQ2 and 6] What specific ideas has JSP considered to extend the effect of its assistance? Are there local or regional organizations that could be providing training or technical assistance under JSP? [Partially derived from findings of fact and conclusions from EQ2 and 6] What specific ideas have you had to extend the effect of JSP’s assistance? Are there local or regional organizations that could be providing training or technical assistance under JSP? [Partially derived from findings of fact and conclusions from EQ2 and 6] What specific ideas have you had to extend the effect of judicial sector assistance? [Partially derived from findings of fact and conclusions from EQ2 and 6] What specific ideas have you had to extend the effect of JSP’s assistance? Are there local or regional organizations that could be providing training or technical assistance under JSP? 45 ANNEX IV: SOURCES OF INFORMATION A. Persons Interviewed Name Position Organization Names provided in draft report only. President/ General Secretary Judicial Council President ELSA Director Academy for Judges and Public Prosecutors JSP COR USAID Deputy Minister of Justice Ministry of Justice CEO EMBRA Corporation President CAA President Appellate Court Bitola President Basic Court Stip Legal Assistant INL Advisor to the Deputy Prime Minister/Director of MDC Media Development Center Judge/Acting President Basic Court 2 Skopje Court Administrator Appellate Court Bitola Deputy Executive Director ESE- Association for Emancipation, Solidarity & Equality of Women President Basic Court Bitola Ex-president of the Basic Court 2 Skopje Basic Court 2 Skopje President Appellate Court Stip Court Administrator Basic Court Stip President of the Court Appellate Court Skopje Task Manager EU Delegation Senior Justice Advisor INL President MYLA DG Office Director USAID Director Coalition All for Fair Trials Constitutional Court Judge Constitutional Court Chief of Party USAID/JSP 46 Court Administrator Basic Court Skopje 2 President of the Bar Bar Chamber President/ Judge MJA- Macedonian Judges Association Court Administrator Basic Court Bitola Head of Judicial Law Department OSCE Head of AO of CBC Court Budget Council Court Administrator of the Supreme Court; President of the Court Service Council Court Service Council Court Administrator Basic Court Veles President Macedonian's Lawyers Association Executive Director CIRa 47 B. Documents Reviewed Contract Documents - USAID Request for Proposal - USAID Solicitation Modification - Tetra Tech DPK's Project Proposal and Annexes - USAID Contract with Tetra Tech DPK Related Projects – Program Documents - Macedonia Court Modernization Project (MCMP) – Final Report - Judicial Reform Implementation Project (JRIP) – Final Report JSP Monthly Reports - December 2011 - January 2012 - February 2012 - April 2012 - May 2012 - July 2012 - August 2012 - October 2012 - November 2012 - January 2013 - February 2013 - April 2013 - May 2013 - July 2013 - August 2013 - October 2013 - November 2013 - January 2014 JSP Quarterly Reports - January - March 2012 - April - June 2012 - July - September 2012 - October - December 2012 - January - March 2013 - April - June 2013 - July - September 2013 - October - December 2013 JSP Annual Workplan Documents - JSP Year I Workplan - JSP Year 2 Workplan - JSP Year 3 Workplan EU Progress Reports – Macedonia - EU Progress Report 2011 (p. 11-13, 58-64) - EU Progress Report 2012 (p. 10-12, 49-51) - EU Progress Report 2013 (p. 10-11, 38-41) Consultant and Trip Reports - ACCMIS, ERCS, JCCMIS and ABMS. USAID Macedonia/Judicial Strengthening Project, Assessment of Court IT Utilization, April 2012. - Case Management Practices in the Macedonian Courts: A Proposal for Designing and 48 Implementing Differentiated Case Management and Time Standards, March 2013. - CIRa. Assessment Report - Initial Screening of LPAs and CSOs. - EMBRA and MESACONS. Capacity and Sustainability Building Progress. - Kavanagh, Michael. Consultant Trip Report. - LPAs and CSOs Assessment - Phase II Report by CIRa - Stout, Ron. Assessment of Administrative and Management Systems, June 2012. - Vasilevska, Rozalija Karchicka. Assessment Report - Court Automation and IT. - Vasilevska, Rozalija Karchicka. ICT Development Plan. - Watve, Amy. Broad Based Coalition of LPAs and CSOs. - Zimmer, Marcus. Differentiated Case Management Report. Legal Documents - Law on Judicial Council, 2010, Arts 8-9. - Law on Court Budget, 2014, Art. 20. - Law on Court Services, 2014, Section III. - Law on Court Budget, 2014, Arts. 7 and 10. - Law on Court Budget, 2010, Art. 4. 49 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523