This publication was produced for review by the United States Agency for International Development by Tetra Tech ARD, through Task Order 002 under the Analytical Services II April 2014 April 2014 This publication was produced for review by the United States Agency for International Development. It was prepared independently by the team of David Cohen, Michael Miner, Dian Rosita, Melinda MacDonald, Lily Purba and Aviva Nabiban. Mid-term Performance Evaluation of the Educating and Equipping Tomorrow’s Justice Reformers (E2J) Program FINAL REPORT MID-TERM PERFORMANCE EVALUATION OF EDUCATING AND EQUIPPING TOMORROW’S JUSTICE REFORMERS (E2J) FINAL REPORT Contract Number: AID-497-O-14-00034 USAID/INDONESIA: OFFICE OF DEMOCRATIC GOVERNANCE APRIL 2014 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i Contents 1. Executive Summary .................................................................................................. 1 2. Evaluation Purpose & Evaluation Questions.......................................................... 7 3. Program Background................................................................................................ 8 4. Evaluation Methods & Limitations......................................................................... 13 5. Findings, Conclusions & Recommendations ....................................................... 16 Annexes ....................................................................................................................... 36 Annexes - Annex I: Evaluation Statement of Work....................................................................... 38 - Annex II: Data Collection Instruments......................................................................... 43 - Annex III: Sources of Information – Bibliography of Documents Reviewed ................ 45 - Annex IV: 127 People Interviewed and the schedule of Evaluation Meetings............. 46 Tables Table 1: Factors producing difference between and within Universities.......................... 9 Table 2: Stakeholders met during the Evaluation of E2J............................................... 14 Table 3: Factors Impacting on Students Learning......................................................... 23 ii ACRONYMS AGO Attorney General's Office AIPJ Australia Indonesia Partnership for Justice BAPPENAS Ministry of National Development Planning C4J Changes for Justice CDCS Country Development Cooperation Strategy CSO DFAT Civil Society Organizations Department of Foreign Affairs and Trade E2J Educating and Equipping Tomorrow's Justice Reformers FJI Formal Justice Institutions GOI Government of Indonesia INP Indonesian National Police LBH Legal Aid Foundation LLM MoU Master of Laws Memorandum of Understanding NGO Non-Governmental Organizations PERADI Bar Association PMP Performance Managing Plan SOP Standard Operating Procedures SOW Statement of Work TAF The Asia Foundation TOEFL Test of English as a Foreign Language UI University of Indonesia, Depok UGM University of Gajah Mada, Yogyakarta UNAIR University of Airlangga, Surabaya UNHAS University of Hasanuddin, Makassar UNPAD University of Padjajaran, Bandung UNSRI University of Sriwijaya, Palembang UNUD University of Udayana, Bali USAID United States Agency for International Development USU University of Sumatera Utara, Medan UW University of Washington WCC Women’s Crisis Centre iii 1. EXECUTIVE SUMMARY The USAID program for “Educating and Equipping Tomorrow’s Justice Reformers” (E2J)1 is built around 4 main goals. • Provide a new generation with the knowledge, skills, opportunities and incentives to pursue a career in public service and contribute to justice sector reform efforts; • Strengthen the capacity of higher education institutions to provide legal education, research, and service that contributes to justice sector reform efforts; • Strengthen the capacity of Civil Society Organizations (CSOs) to support, advocate and monitor justice sector reform; and • Increase collaboration among law schools, CSOs, and key justice sector institutions to advance justice sector reform and develop the next generation of justice sector practitioners and reformers. In conducting this evaluation at the beginning of the 4th and final year of E2J activities the evaluation team aimed to assess: (1) progress that has been made towards attaining these goals; (2) factors that have to contributed to successes and the obstacles and challenges that have emerged; (3) the reorientation of the final year of the program as reflected in the challenges identified in Year 3 and addressed in the year 4 Workplan; and (4) what measures and initiatives might contribute to the future success and sustainability of the legal clinics supported through E2J. This assessment was organized around the 3 questions below which USAID asked the evaluation team to address. Analysis of responses to the 3 questions forms the basis of this report. 1. How effectively has the E2J project been able to increase the capacity of law faculties to channel well trained / educated graduates into public services? 2. How effectively has the E2J project enhanced the capacity of law schools, professors, students, CSOs and the formal justice sectors to improve practice￾oriented teaching of law? 3. How effectively has the E2J project been in improving the capacity of CSOs and Universities to support, advocate, and/or monitor justice sector reform? In addressing these 3 questions the evaluation also focused on analyzing the challenges identified in the E2J Q11 Report and the E2J Year 4 Workplan. Through interviews, focus group discussions, and a review of documents, the evaluation considered each of these challenges and addresses them in this Evaluation Report. In addition, the evaluation also focused on the extent to which gender mainstreaming has 1 USAID Indonesia, Educating and Equipping Tomorrow’s Justice Reformers in Indonesia Program, E2J. USAID Grant Agreement Number: AID-497-A-11-00005. 1 been integrated into E2J clinics and other activities. The evaluation team visited 4 of the 8 E2J universities and met with administrators, faculty, students, CSO partners, and implementers. The evaluation team interviewed 127 people from the 6 stakeholder groups. Details as to the methodologies employed, and the limitations of the evidence gathering is included in Section 4 of this Evaluation Report. As a preface to summarizing the specific findings and conclusions of the evaluation in regard to the questions and criteria set out above, some brief general comments may be helpful. First, it is important to underscore that in visiting 4 universities and their CSO partners, the evaluation team found that they implemented and experienced E2J in different ways. Moreover, it was not simply that there were differences between the E2J clinics at the 4 universities, but there were also different approaches and perceptions within universities. This resulted, for example, in differences between clinics at the same university. These differences appeared to be due to different faculty understandings of the nature and goals of clinical legal education as well as to varying levels of faculty commitment. Within each university context we also found different perspectives and conceptions of E2J among faculty, Deans, students and CSOs. We found that many factors are responsible for the wide range of performance and sustainability of the clinics supported by E2J. The most important of these involves the placement of E2J in curricula and budgets, the commitment of Deans and faculty for the sustainability of the clinics, the design and implementation of the clinical experiences, and the challenges created by systemic obstacles within the universities to implementation of E2J. Despite these differences, however, we found considerable uniformity in the motivation, commitment, and aspirations of students and CSOs to engage in serious practice-oriented clinical activities that would enhance their capacity and potential as legal professionals, advocates, and justice reformers. The strength of E2J and the greatest opportunities for its success resides in the talents and dedication of the impressive students and CSOs met by the evaluation team. On the basis of our analysis, as informed by the questions and criteria described above, the evaluation arrived at the following findings and conclusions: Finding 1: Impact on Public Sector Careers Although there is certainly evidence of actions being taken by and through E2J to inform and encourage students to consider public service careers, there is no evidence that this has had an effect on their entering the public service. This is primarily because justice sector institutions, including the prosecutors’ offices, are not hiring at this time. Finding 2: Law School Capacity Improvements The institutional capacity of all 4 law schools was substantially increased by E2J because all 4 implemented multiple legal clinics that brought far greater numbers of students into clinical settings than had previously been the case. Finding 3: Integration of Clinics into the Curriculum Unless clinics are fully integrated into the curriculum as a compulsory subject (whether in the manner adopted in UNSRI (University of Sriwijaya) or in some other way) their sustainability will be in doubt. The lack of such integration in a manner that is designed 2 to allow students to participate in real clinical practice is also an institutional issue of paramount importance. Finding 4: Training Approach The fact that only limited numbers of faculty were brought to training sessions in Jakarta and other cities limited the impact that training has on the clinical faculty as a whole. E2J is addressing this in the Year 4 Workplan by organizing training workshops at each university so that more faculty can participate. This development is well-advised. Finding 5: Learning Clinical Skills The students universally attributed this perceived lost opportunity to learn more clinical skills to the structural and systemic impediments created by the lack of integration of the clinical courses into the curriculum in a manner that would have encouraged full clinical practice. Finding 6: Contribution of CSOs The increased capacity of students to engage in effective legal practice appears to be largely due to the contribution of CSOs to the clinical experience offered students rather than to their learning in the classroom part of the clinic. Finding 7:Improving Capacity on Research and Advocacy E2J has improved the capacity of CSOs in research but has not been able to support development of strong linkages among CSOs, faculties and FJIs. Improvements to the curriculum, faculty and the overall pedagogical approach are necessary to produce law students who will be more effective advocates and monitors that are motivated to pursue justice sector advocacy careers. Finding 8: Linking with Formal Justice Institutions Despite the Memorandum of Understanding (MoU) between E2J and Formal Justice Institutions (FJI) being developed, networks and relationships among faculty, CSOs and the FJI, has been weak. Finding 9: Sustainability CSOs and law students demand the continuation of the legal clinics. CSOs are very committed and supportive of the clinics which indicates a strong potential for sustainability. Levels of commitment and ownership at the 4 universities are varied which suggests different levels of potential sustainability, and only 1 university (UNPAD), has secured funding for clinic continuation. Finding 10: Mainstreaming Gender to Develop Capacity in Graduates A widened integration of gender into all aspects of programming and clinics would increase the capacity of graduates in terms of understanding and applying gender equality principles. Finding 11: Gender Tools to Identify Issues and Opportunities Gender assessments and gender profiles that identify gender gaps, and suggest solutions to overcome them, don’t appear to have been used throughout E2J. Issues 3 such as differential access to justice and violence against women are only included in the women’s and children’s clinic except for the criminal law clinic in 1 location managed by the same CSO. There is no evident gender mainstreaming strategy and planning in E2J. Sex disaggregated data however has been collected effectively in E2J. Finding 12: Deepening Understanding of the Importance of Gender Some interviews reflected a misunderstanding about the role of gender mainstreaming by some faculty and students interviewed- i.e. in some interviews the local gender equality expert found that when she asked about gender the most common response was that it was about men and women and not about how the legal system affects men and women differentially. In addition, gender assessments should be better utilized to identify issues and opportunities. Findings could then be used to create an improved gender strategy. Finally, lessons learned should help the project maximize its gender equality results, e.g., when male students were exposed to gender in the Women’s and Children’s clinic, they claimed it was eye-opening and changed their views. CONCLUSIONS 1. Successes for Students and CSOs: E2J has been successful in creating opportunities to channel the talent and energies of highly motivated students into clinical settings at highly receptive CSOs. Students universally value such opportunities and request deeper engagement in practice￾oriented activities. 2. Structural and Systemic Challenges: Maximum benefit from these opportunities has not been achieved due to a variety of structural and systemic challenges. These include: faculty competence and engagement; effective integration of clinics into law curricula; lack of gender mainstreaming; students not acquiring substantive legal knowledge and analytical skills before they begin their work with CSOs; and, systemic limitations of time that students can commit to practice-oriented work with CSOs. 3. Effectiveness of E2J: CSOs in general have taken the lead in structuring meaningful practice-oriented educational activities for student participants. As a result the CSOs rather than the faculty classroom settings have emerged as the locus where E2J goals are most effectively being achieved. 4. Integration of Gender Equality: Overall, although sex disaggregated data has been collected effectively, gender has not been mainstreamed throughout all E2J programming and clinics except when it is the focus of Women’s and Children’s clinics where it is well integrated and includes a comprehensive understanding of how the legal system affects men and women differently. 4 5. Future Direction of E2J: Based on addressing the challenges identified by E2J and the findings in this evaluation, the workplan for year 4 of E2J is on the right track and needs to focus on the planned activities and on addressing the findings identified in this report. RECOMMENDATIONS On the basis of these findings and conclusions the Evaluation recommends: Recommendation 1 – Overall Sustainability E2J should prioritize systemic integration of legal clinics into the compulsory curriculum of law faculties as a necessary condition for the sustainability of legal clinics. Gender should be mainstreamed throughout the system including within all legal clinics. Recommendation 2 – Individualized Strategies E2J should identify the particular opportunities for effective implementation at each university and adopt individualized strategies for achieving ownership, sustainability, and greater faculty engagement. If and when appropriate, universities, clinics, or CSO partners should be dropped from E2J so as to focus resources most effectively. Recommendation 3 – Internships and Careers Some CSOs and clinics have demonstrated the value of internships or internship-like engagement in leading students into public service careers. E2J should explore and develop internship and externship components of its programs and their integration into law faculty curricula as the best available vehicle to channel students into career opportunities in CSOs and FJIs. Recommendation 4 - Strengthening CSO Advocacy Capacity Support strengthening the capacity of CSOs to conduct research based advocacy. This should include capacity building to produce policy oriented research products, capacity building on advocacy strategies particularly with justice sector institutions, and engagement with media. Research products should be focused on the primary targets of the CSOs. In most cases the primary audience will be justice sector institutions and government policy-makers. Capacity building can be addressed in a comprehensive training of ‘Advocacy Strategy’ which consists of a variety of relevant materials and supports linkages among stakeholders. In conducting this activity, E2J should draw on the strengths of partner organizations and external expertise to encourage cross learning and transfer of skills and knowledge. Approaches to capacity building need to consider skills development relevant to CSOs’ area of advocacy, so that training materials can be applied and practiced in day-to-day activities. Therefore 1 training session should be dedicated to improving the capacity of CSOs in different core areas (civil, criminal, environment, women & children). Recommendation 5 – Working with the FJIs The focus should be on the core objective of E2J which is providing students with knowledge, skills, opportunities and incentives to pursue careers in the public service and contribute to justice sector reform efforts. As a potential opportunity, currently the 5 Supreme Court is seeking organizations that can operate court-based legal consultation services in district courts throughout Indonesia as implementation of Supreme Court Regulation No. 1/2014 on Legal Services for the Poor. CSO offices could be well placed to implement this and develop student interest in justice reform issues. E2J should initiate discussions involving the Supreme Court, legal aid partners and university partners on this opportunity. Recommendation 6 – Gender Mainstreaming Data needs to continue to be sex disaggregated. Gender mainstreaming needs to be fully integrated from the wording of initial contracts throughout project design, gender analysis and in gender responsive programming in all legal clinics, including gender sensitive textbooks, teaching and other E2J materials. Recommendation 7 – Continuing USAID Support USAID should take advantage of the aspirations, talents and energies of Indonesian law students who express dissatisfaction with business as usual coursework and demand a more practice oriented approach that will better prepare them for careers as legal professionals. USAID should continue to support clinical legal education as the best means to direct the most highly motivated and capable students into public service careers. Recommendation 8 - Continuing USAID Support for Faculty Development USAID should also enable broader opportunities for faculty to engage in international collaborations and to participate in LLM or Summer Programs in American law schools, especially those, like UW or the University of Hawaii, with an orientation towards Asian law. Unless substantial numbers of faculty can be involved effectively in such programs it is difficult to see how the demand for practice oriented, skills-based clinical education can be met. To be effective this will undoubtedly require additional opportunities for required intensive English language preparation of the kind offered by some US based institutions (e.g., the collaborative program between the East-West Center and the University of Hawaii) so as to permit more faculty to be eligible to participate. 6 2. EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE Overall for USAID, as noted in the USAID Evaluation Policy2 , the primary purposes of evaluations are: (1) accountability which measures project effectiveness, relevance and effectiveness and (2) learning which can be gleaned from a well designed and executed evaluation. As noted in the Statement of Work (Annex A)3 , the purpose of this specific evaluation of E2J is to assess the performance of the E2J program. This evaluation will measure the degree to which the program goals have been met, and the contributing factors that have been responsible for or detracted from the achievement of these goals. In addition, this evaluation will provide an initial assessment of the sustainability of program achievements and the factors that have contributed to, or detracted from, the sustainability of program achievements. The evaluation is intended to provide USAID and their local partners, including the 8 participating universities, the 19 CSOs and the 3 implementing partners, The Asia Foundation (TAF), KEMITRAAN and the University of Washington (UW), with an objective assessment of program performance, lessons learned and recommendations to inform final activities before the end of the project. The results of the evaluation are also intended to inform possible future programming in the area of legal education, justice sector reform and justice sector CSO capacity strengthening. EVALUATION QUESTIONS The following 3 questions identified by USAID4 will be addressed in this evaluation: 1. How effectively has the E2J project been able to increase the capacity of law faculties to channel well trained / educated graduates into public services? 2. How effectively has the E2J project enhanced the capacity of law schools, professors, students, CSOs and the formal justice sectors to improve practice￾oriented teaching of law? 3. How effectively has the E2J project been in improving the capacity of CSOs and universities to support, advocate, and/or monitor justice sector reform? 2 USAID Evaluation Policy, 2. Purposes of Evaluation, January, 2011, p.3. 3 Statement of Work – Evaluation of E2J, Indonesia EXO – 13-002, p.2. 4 Ibid, p. 2. 7 3. PROGRAM BACKGROUND The purpose of the “Educating and Equipping Tomorrow’s Justice Reformers” (E2J)5 project is to: • Provide a new generation with the knowledge, skills, opportunities and incentives to pursue a career in public service and contribute to justice sector reform efforts; • Strengthen the capacity of higher education institutions to provide legal education, research, and service that contributes to justice sector reform efforts; • Strengthen the capacity of Civil Society Organizations (CSOs) to support, advocate and monitor justice sector reform; and • Increase collaboration among law schools, CSOs, and key justice sector institutions to advance justice sector reform and develop the next generation of justice sector practitioners and reformers. Although these 3 types of institutions are arguably the 3 pillars of justice reform, they often have minimal contact with 1 another in practice. By promoting communication and interaction, and institutionalizing cross-relations, law reform will have a greater likelihood of success. E2J has 4 components: (1) research projects relevant to issues of law reform, (2) curriculum development, (3) increased capacity of civil society organizations, and (4) delivery of 10 University of Washington LLMs. A main program focus has been law clinics, focusing on anti-corruption, criminal law, civil law, environmental law and women and children. In Oct. 2012, USAID conducted a Democracy, Rights and Governance Assessment which concluded that the 2 main issues facing Indonesia requiring priority attention from USAID were: 1) poor public service delivery and 2) poor and weak justice sector. The assessment pointed to continued need in the area of legal education, legal analysis, capacity of justice sector CSOs and institutional reforms. The Context for the Evaluation of E2J A. Differences and Commonalities It is important to underscore that in visiting 4 universities and their CSO partners6 , the evaluation team found that they implemented and experienced E2J in different ways. Moreover, it was not simply that there were differences between the E2J clinics at the 4 universities. Rather, there were differences both between the universities and within them. We found, for example, marked differences between clinics at the same university. Some clinics were more oriented towards attempting to provide a meaningful 5 Ibid. p. 1. 6 Schedule of Activities and Meetings included in Appendix IV of this document. 8 clinical experience for student participants (i.e., actual practice-oriented activities) despite systemic impediments, while others were essentially regular classroom based courses with some observational visits to other institutions. This appeared to be due to different faculty understandings of the nature and goals of clinical legal education as well as to different levels of faculty commitment. Within each university context we also found different perspectives and experiences between faculty, Deans, students and CSOs. At times it seemed as if these different groups which we met in separate focus groups were describing completely different programs. This appears to be in part due to the lack of knowledge on the part of faculty and Deans as to what the students were actually doing in the clinics and how the students felt about their clinical experience. For example, at UNHAS, the faculty reported that the student participants in the Environmental Law Clinic were spending extensive amounts of time in remote island villages where the CSOs mangrove regulation project was being implemented. According to the CSO partner, however, none of the students had ever visited a village because of the time constraints imposed by their competing class schedule that limited them to visiting the CSO office for only 2 hours per week. It was in fact, the 2 interns who had worked at the village sites. Such examples could be multiplied and appear to arise from lack of mentoring, attention to student evaluations, and communication with the CSO partners on the part of faculty and Deans. To give another example, while Civil Law Clinic faculty at USU reported that they were providing a real clinical experience for their students, the students stated that they found the activities to be of minimal value because they had no opportunity to engage in practical activities. We might summarize the factors that produce or influence these differences between and within universities as follows: Table 1: Factors producing difference between and within Universities 1. The clinic as an elective vs a compulsory course 2. Commitment of budget and resources to clinics 3. Selection, competence, and numbers of faculty 4. Commitment of Deans and Vice-Deans 5. Unit credit and structural conflicts in course scheduling 6. Relations to existing Legal Aid and other clinics 7. Cooperation and Interaction with CSOs 8. Commitment to Students by Faculty 9. Nature, design, and scope of clinical work 10.Efforts to engage formal justice institutions 11.Numbers and recruitment of student participants 9 We must also emphasize, however, that apart from these differences we also observed very important commonalities. The most significant of these was the quality, interest, and commitment of the students we met. At every university, and particularly at the UNSRI, USU, and UI, we were deeply impressed by the bright, articulate, ambitious, and highly motivated students who were participating in the clinics. These students are thirsting for, and expecting, just the kind of clinical experience and practical opportunities that E2J aims to provide. But at every university, albeit to different degrees in each, we also found systemic factors, listed above, that were preventing E2J from fully realizing its potential. The second significant commonality we observed was the commitment, competence, and motivation of most CSO partners to provide high quality practical experience to their student participants. As will appear in greater detail below, CSO partners appear to understand and appreciate the importance of training young lawyers through participation in legal practice and through development of strong analytical and advocacy skills. B. Challenges One of the tools that we used to inform our interaction with the 4 universities were the challenges that E2J itself identified when it undertook its own review of the clinics as reflected in the Year 3 Report (especially Annex 2),7 the Q11 Quarterly Report8 , and the Annual Workplan for Year 49 . We found these challenges10 to be useful and well￾grounded as analytical lines of inquiry. In this section we will summarize our analysis of the challenges identified by E2J for our evaluation of the performance of the clinics. Challenge 1: Lack of a structured program The faculty we met described a structured program, but what they, in almost all cases, were referring to was the formal 3 step structure of 4 weeks of classroom lecture, 3-4 weeks of clinical observation, and 1 week of evaluation. They represented this to us as the structure they were given by E2J and it appeared that many thought that it was an inflexible requirement. We heard little from faculty that reflected engagement with E2J about how best to structure clinical courses and there was little evidence of a substantive structure designed for each clinical setting to develop practice-oriented knowledge and skills. As a result, the clinics followed the 3 step structure and didn’t appear to have focused on how best to integrate classroom learning with the goals of clinical education. Many CSO partners, on the other hand, had developed a substantively structured program designed to take students through the various stages of a case, trial, or investigation by engaging them in meaningful practical activities. With notable exceptions, faculty as a whole do not seem to systematically and fully engage with their students. While the CSOs strive to engage students in a full range of practical activities, all reported that the structurally imposed constraints on students’ time due to 7 The Asia Foundation, Year 3 Report on E2J. 8 The Asia Foundation, Quarterly Report – October- December, 2013, Program Q11. 9 The Asia Foundation, Annual Workplan, Year 4: 15 March 2014 to 14 March 2015. 10 Ibid, the challenges are noted in several documents including being specifically listed in the Q11 Report, pp 11-12. 10 their heavy course loads, impeded them from doing so. Students with whom the team met during the evaluation in all locations made the same observation. Challenge 2: Staffing clinics remains a difficulty In our meetings with faculty and Deans, challenges which surfaced included: selection criteria for faculty (especially in regard to expertise in the subject matter of the clinic), the amount of time they needed to commit to clinics, and managing the competing obligations of faculty. These issues appeared to be more problematic than the number of faculty assigned. If the faculty are not committed to making time for their students or do not have the competence to teach the clinics both in terms of area of substantive expertise and ability to promote the development of analytical skills, then increasing numbers will not likely address the problem. The underlying issues are, first, in some schools, the lack of integration of the clinics into the regular curriculum which results in the course being taught as an add-on to an already full teaching load. The second, and related underlying problem, is whether the course is offered as a compulsory course or as an elective and where in the curriculum it is placed (e.g., in relation to other practice requirements). We also note that CSOs appear on the whole to provide better mentoring, supervision, and practice-oriented instruction than do the law faculty. Challenge 3: Insufficient supervision impacts quality E2J notes that many faculty lack time and/or commitment to properly supervise the students who are participating in their clinics. The underlying issue here is whether the way in which clinics are organized and staffed by the law schools facilitates or inhibits faculty engagement. Because of the factors noted in Challenge 2, as well as because of the pressures of other professional activities, faculty may not have the time to fully engage with and supervise their students. If they lack the substantive knowledge of the area of law covered by the clinic, or if they lack knowledge of actual legal practice in that area, they may also not be in a strong position, or may lack confidence to supervise their students. On the other hand, several CSOs reported that they have developed their own procedures for supervising students, including for example, weekly reports and evaluations or assigning individual mentors. They also recommend grades for students to the faculty which is understandable since they are the ones who are doing the supervision and mentoring of the students at the clinics. Challenge 4: Lack of financial planning None of the universities we visited have budgeted for continuing operation of the clinics. UNSRI, however, has integrated the clinics into the compulsory curriculum which suggests that funding will be included in the regular budget. At UI and USU law schools, the Deans assured us that they will find a way to fund the clinics because of their importance. Despite these assurances, however, questions remain, especially because it was at these schools that the Deans told us that it would be extremely difficult or impossible to make the clinics compulsory. As the Dean at UNHAS was unable to keep his appointment with us we were not able to speak to him about this issue. It should also be recognized that some CSOs are providing modest funding to support the clinics in the form of providing lunches for the students, paying for their transportation to the clinics and fieldwork sites, and in 1 case providing honoraria to encourage law faculty to come and speak at the CSO. 11 Challenge 5: Partner relationships While it is true that in most cases the links between faculty and the CSO partner are weak, it is important to observe that where they are strong it appears to be due to a pre￾existing relationship between faculty members and the CSO. In some cases, for example, a faculty member is on the board of the CSO and actively promotes cooperation. The relationship that appeared strongest to the team seemed to be between CSOs and student participants whereas in most cases it was the faculty who was less informed about students’ activities with clinics. Students almost universally attributed the substantial benefits they felt they were deriving from the clinics to the strength of their engagement with the CSOs. They strongly and eloquently expressed their dissatisfaction at the systemic impediments that prevented them from achieving the maximum benefit from the clinics (i.e., not a compulsory course, insufficient units, and course scheduling conflicts preventing them from participating in CSO activities). The CSOs also strongly indicated these same factors as well as pointing in many cases to the lack of engagement of faculty with them. Challenge 6: CSO partners in environmental and women’s clinics lack knowledge We found that in the case of 1 environmental clinic the CSO was ill-suited for E2J because it did not engage in any law related environmental work. On the other hand, the 3 other environmental law CSOs seem to have adequate and substantive expertise. CSOs do, however, express the desire for additional training and engagement with faculty experts in order to enhance their legal knowledge. In terms of gender, at the women and children’s clinics there was excellent knowledge and understanding but in most other clinics there was no link made with gender equality issues. As noted above, in some cases faculty themselves appear to lack the relevant substantive knowledge in either area. The underlying issue here is the lack of faculty oversight of the students’ experiences and sufficient faculty engagement to ensure that their students have a positive and valuable education experience in the clinics. Other Challenges In addition to the challenges identified by E2J we point to related challenges that are having an impact on the quality and sustainability of the clinics: • Identifying and meeting student demands, needs, and expectations about teaching, mentoring, clinic access, time constraints, unit credit, etc. • Enabling students and committed CSOs to implement what they repeatedly called “the maximum” to fulfill the potential of E2J to achieve its goals and to take advantage of the talent, motivation, and commitment of students and CSOs. • Adapting to the circumstances and opportunities at each university and CSO. We note that the Year 4 Workplan calls for just such a reorientation of E2J activities. • Strategic engagement of formal justice institutions so as to promote student interest in pursuing such careers. 12 4. EVALUATION METHODS & LIMITATIONS Evaluation Methods The overall evaluation approach follows the Evaluation Practices outlined in the USAID Evaluation Policy as described in Chapters 4 and 5 of the Evaluation Policy11 and also uses what is being called the utilization based approach which means that the evaluation is intended to be of primary benefit to the major stakeholders. 12 Therefore this evaluation sought to involve a wide range of stakeholders in the process, integrate gender within the overall questioning, and address the 7 challenges described above identified by the implementing agencies in their most recent reporting to USAID. Specifically, as noted in the workplan for this Evaluation, the team focused on the evaluation criteria contained in the 3 key evaluation questions. The methodology was to create sub-questions (contained in Annex 2)13 for each of the questions and then to use 4 lines of evidence to gain sufficient data to draft findings, conclusions and recommendations based on triangulation of some or all of these lines of evidence. The evaluation also endeavored to reflect USAID’s commitment to promoting gender equality as a critical development objective. To quote its policy, “gender equality and female empowerment are fundamental to the realization of human rights and key to effective and sustainable development outcomes”. Therefore a sub-question addressing gender was added to each of the 3 key questions and will be addressed in a section of the evaluation findings on gender. The sub-questions to be addressed concerning gender were: 1. To what extent have the principles of gender equality been taken into consideration with respect to the objective of channeling well/trained educated graduates into the public service? 2. To what extent was a gender equality preliminary analysis carried out (including sex disaggregated data) to help enhance the capacity building of law schools, professors, students, CSOs and the formal justice sector to improve practice￾oriented teaching of law? 3. What gender equality activities have been implemented (or planned) to improve the capacity of CSOs and universities to support, advocate and/or monitor justice sector reform? Were the amount of resources (both financial and non-monetary) which have been invested sufficient to ensure effective integration of gender? 11 USAID Evaluation Policy, January, 2011, p.3. 12 Michael Quinn Patton, Utilization Based Evaluations, 4th Edition Sage Publishing: Los Angeles, 2008. 13 The sub-questions for the 3 questions are included in Annex II of this evaluation report. 13 Evaluation Lines of Evidence The 4 main sources of evidence from which the evaluation gathered its data included a secondary review of documents and indicators and 3 primary data sources. The 4 sources were: (1) The first line of evidence was the review of documents including performance management plans, quarterly and annual reports, workplans and BAPPENAS documents. Also reviewed was indicator data provided by E2J augmented by additional information from USAID’s Changes for Justice C4J (C4J) and other donors i.e. the Australian Department of Foreign Affairs and Trade (DFAT)’s Australia-Indonesia Partnership for Justice (AIPJ) as well as relevant GOI justice programs. (See Annex III)14 (2) The second line of evidence was key informant interviews and focus groups through which primary data was gathered at 4 of the 8 participating universities from: law faculty administrators, participating faculty at those universities, legal students taking part in the clinics, and participating CSOs. Specific interviewees who took part in key informant interviews or focus groups in 4 cities in Indonesia included 127 informants from the following stakeholder groups15: Table 2: Stakeholders met during the Evaluation of E2J Stakeholder Groups No. of Informants Senior Administrators – including a Vice-Rector, 3 Deans & 2 Vice-Deans (5 M &1W) 6 University law faculty members involved in E2J at 4 universities (14 M & 7 W) 21 Legal students participating in pilot clinics and internships (23 M & 23 W) 46 CSOs involved in pilot clinics, research, internships or other E2J activities (3 of 5 core CSO partners & 10 of 19 CSO partners) (14 M & 18 W) 32 Implementing Organization E2J staff +1 Senior Advisor (7 M & 9 W) 16 Donor – USAID/Indonesia Democratic Governance team and the justice sector development team at the Australian DFAT (plus attendees at the Debriefing) (3 M and 3 W) 6 Total Interviewees and/or focus group participants including donors (66 men and 61 women) 127 14 Annex III includes a list of documents reviewed by the Evaluation Team. 15 Annex IV includes all the stakeholders consulted during this evaluation and the meeting schedule. 14 For each of the above groups, the same base questions were asked and the answers noted and compiled by the evaluation team. Differing sub-questions specific to each type of stakeholder were used to allow the evaluators to differentiate among the perspectives of the stakeholders. The responses of the administrators, law faculty members, students and CSOs were then compiled and analyzed to identify corroborating evidence and findings. (3) During the above visits to the 4 universities, there were several opportunities for the evaluation team members to have structured observations of student-teacher interaction and students interaction with CSOs. (4) Finally, the evaluation team had an opportunity to be briefed and debriefed by USAID and the implementation agency E2J staff and advisors which provided a final source of input for the team. Except the core personnel, the attendees at the debriefing are not included in above list. Evaluation Limitations Overall the most significant limitation to this evaluation is that it is intended to focus on 3 specific questions but was not intended to be a full review of all aspects of the E2J program. The focus of the evaluation is on the clinics and their effectiveness. To do this, the focus was on a few target institutions and their clinic programs in depth rather than to work with all possible stakeholders. Therefore USAID and the evaluation team agreed that it would not be necessary for the evaluation team to visit all 8 universities but rather to focus their time in this short period on 4 university partners of E2J. The 4 universities visited were geographically well spread out including: 2 of 2 in Sumatra, 1 of 3 in Java, 0 of 1 in Bali and 1 of 1 in Sulawesi. Similarly with the focus of 2 days at each of the 4 universities during the 2 week mission, it was not possible to allocate time to interview representatives of formal justice institutions including the Supreme Court, the Attorney-General’s Office, the Judicial Commission and other formal justice institutions involved in recruitment for the public service sector. Although the team included both an international and local gender evaluator, it was only possible for the local gender evaluator to participate in the Jakarta based meetings. Nevertheless, at the other 3 universities visited outside of Jakarta, the gender questions were also posed and the responses received and integrated into the findings. 15 5. FINDINGS, CONCLUSIONS & RECOMMENDATIONS The first part of this section includes findings on the 3 main questions and on gender resulting from an analysis of the 4 lines of evidence. These sections are followed by conclusions based on the findings and linked with the recommendations. 5.1 QUESTION 1 How effectively has E2J been able to increase the capacity of law faculties (law schools) to channel well trained/educated graduates into public service? Effectiveness of steps to prepare administrators and law teachers to inform/encourage students about public service careers. Have law school administrators allocated resources? Activities have been carried out by E2J including: training of University faculty; International training of LLMs at the University of Washington; and sharing information at Public Service Days. Although these and many related activities have been done and well received by the stakeholders, there is no evidence yet that these actions have been effective at achieving the purpose of encouraging students to pursue public service careers. There really has been no opportunity since the federal justice institutions in Indonesia have not been hiring graduates during the life of E2J to date. And even when the recruitment is re-opened it will be difficult to track down the effect of the E2J program on hiring because the number of applicants from all over Indonesia is usually high16 and there is no detailed data base of those applicants, because the first stage of the selection process was conducted at the provincial level and the Supreme Court only received shortlisted candidates. The difficulty seems to be more prevalent in the prosecutor's office where the data base is less developed than at the Supreme Court. With respect to the second issue, 7 of the 8 law school administrations (and all 4 universities visited by the evaluation team) have not yet included legal clinics in their budgets although 3 of the 4 with which the evaluation team met, say that they will include the clinics in future budgets. The evaluation team understands that Padjajaran University (UNPAD) in Bandung has budgeted for clinics, but since the evaluation team did not visit that university or have the budget, this cannot be verified. 16 In 2012 for example, when Supreme Court opened positions for court civil servants (not including those for candidate Judges), it attracted 14,770 applications from all over Indonesia, according to the Administrative Body of the Supreme Court. 16 How have law faculty curricula and teaching methods been adapted towards public service careers and how widely and effectively? Faculty members report (and documentation and E2J interviews corroborate) that E2J has not yet finalized and shared all revised curricula in the form of textbooks or manuals for working with legal clinics. In all 4 locations visited by the evaluation team, the faculty members reported that they are using their own teaching materials. On the other hand, E2J informed us that the teaching materials were prepared with assistance from an E2J Expert, and had been introduced this semester for the first time. The faculty with whom the evaluation team met made no mention of this. The faculty also gave no indication that they were using their materials to provide students with analytical tools as opposed to substantive legal knowledge related to the clinics’ subject matter. We also note that it is questionable how well a large amount of material can be absorbed by students in only 3 or 4 sessions of 2 hours, along with all the other information they need to introduce and launch their first clinical experience. This again points up the need for greater unit credit and more student-faculty instructional hours. Success with justice sector institutions and CSOs to lead students to public service careers. What are the challenges and how have they been met? There are cases where effective internships with the AGO and religious courts have taken place but these are not typical. The main challenge is the lack of structured engagement among the FJIs, CSOs and students which will be addressed through many of the activities identified in the present E2J workplan. CSO placements are effective except for the systemic time/schedule conflicts between clinics and mandatory law classes which decreases student’s opportunity to fully benefit from clinics. Clinics are not compulsory and the unit credit and allocated time are insufficient. Impact of Public Service Days and other initiatives to overcome disincentives for public service careers. How is E2J reaching out and targeting the best students? The impact is mixed because only some semesters of students were made aware of the Public Service Days (including students who are only in their first semester) but have been told that it is mandatory that they attend Public Service Days. Overall the problem is lack of a coherent strategy and proper preparation to identify and engage the overall student body and to ensure that the students who are most likely to benefit attend at the time when this will have the most benefit. This was a missed opportunity to influence the latter year students who are most likely to benefit from Public Service Days. In some locations, these Public Service Days are known to be taking place by multiple stakeholders in the wider community including other donors such as Australia which surfaced in the interviews. E2J has changed its approach for future Days by: working with on-site faculty coordinators, sending out electronic invitations to clinic students and faculties, encouraging university partners to better advertise the event and developing a survey tool for feedback by attendees. 17 Finding 1: Impact on Public Sector Careers Although there is certainly evidence of actions being taken by and through E2J to inform and encourage students to consider public service careers, there is no evidence that this has had an effect on their entering the public service. This is primarily because justice sector institutions, including the prosecutors offices, are not hiring at this time. 5.2 QUESTION II How effectively has the E2J project enhanced the capacity of law schools, professors, students, CSOs and the formal justice sector to improve practice￾oriented teaching of law? This question addresses the issue of increased capacity of 5 specified groups or entities to “improve practice-oriented teaching of law.” Each of these groups must be considered separately because different measures were proposed or adopted in order to augment the capacity of each. In regard to “students”, however, we understand the question to mean whether their capacity to engage in effective legal practice was enhanced as they are not teachers but rather the objects of the practice-oriented approach to legal education implemented through E2J. This section of the Evaluation will thus address each of these groups in turn and conclude by discussing the way in which E2J attempted to link them in a collaborative effort to enhance pedagogic capacity. This section will not analyze each of these 5 groups according to the 4 sub-questions employed in regard to Questions 1 and 3 as this would produce an overly fragmented discussion. The sub-questions will instead inform the analysis of each of the 5 groups. The issue raised in sub-question 2, “How have the challenges and obstacles to the effectiveness of clinical courses identified in evaluation documents been addressed and what plans are there to deal with them in the remaining 2 years of the project?”, have already been addressed above in the section of the evaluation report specifically devoted to the 7 challenges identified in the E2J Q11 Report and E2J Year 4 Workplan. Enhanced Capacity of Law Schools This sub-question raises the issue of institutional capacity as opposed to the individual capacities of the professors/lecturers engaged in teaching the legal clinic courses. As noted above in the section on Context, each of the 4 universities which the evaluation team visited, implemented and experienced E2J in different ways. Some generalizations are nonetheless possible. The team learned that all of the law schools have in one manner or another adopted clinical education into their curriculum and have identified a group of faculty to teach these courses. In this sense, the institutional capacity of each of the schools has been measurably increased. The increase in capacity has, however, been differential in two senses. First, some of the law schools appear to have had no actual ongoing clinical programs in their curricula prior to E2J. Other schools had pre-existing clinics, for example as in the criminal law clinic at UNHAS Some CSOs also had longstanding previous clinical 18 activities with university students, as is the case at Pusaka in Medan. As we have no information about the capacity and performance of the previously existing clinics we cannot comment on how much their capacity was increased when the E2J clinic was grafted onto them. Apart from this qualification, however, on a quantitative basis there were substantial benefits in capacity. Finding 2 Law School Capacity Improvements The institutional capacity of all 4 law schools was substantially increased by E2J because all 4 implemented multiple legal clinics that brought far greater numbers of students into clinical settings than had previously been the case. Institutional capacity may be assessed on other measures than the number of clinics offered in the curriculum. Such measures include: (i) the numbers of students that can be accommodated in the clinics, (ii) the engagement of law school administrators with the clinical programs, (iii) the resources provided by the law schools to ensure the quality and the sustainability of the clinics and (iv) the placement of the clinics in the curricula. Here it is necessary to distinguish among the 4 law schools because, except in regard to 1 of these criteria, there are very substantial differences among them. As noted earlier, the 1 criterion where generalization is possible is that none of the 4 schools has placed the clinics in their 2014 or 2015 budgets. (i) E2J documentation identifies that the ideal number of students in each clinic is 5-14 but the evaluation team found that there was variation in the targets set by some law schools as to the number of students to be accommodated in the clinics. Although the reasons for this variation are not clear, in some cases there appear to have been different interpretations of what E2J communicated to them as to target numbers. In UNSRI, they set a limit of 25 students per clinic. At UNHAS, clinical lecturers understood the E2J limit to be 14, whereas at UI they limited clinics to 5-8 students. At USU they set the target at 16 and the clinic working with Pusaka had 16 the first time it was offered. Actual numbers of students participating in clinics ranges from a low of 2 to a high of 25. In the case where only 2 or 3 students are participating the question is raised as to whether such enrolments can justify the allocation of resources represented by assigning 2 faculty members to teach a clinic. The variation in participant numbers reflects differences in the applicant groups. These differences vary from university to university. At USU, for example, the number of applicants to the civil law clinic is low, perhaps reflecting the dissatisfaction which student informants noted. In UNHAS, on the other hand, where there has been an apparently very successful clinical program with the religious courts, 70-80 students have applied to the civil law clinic for each semester and 14 were accepted. The 19 UNHAS criminal law clinic had 50 applicants in its first year (they accepted 12) and in its second year had 124 applicants of whom 10 were accepted.17 If the goal of E2J is to provide clinical opportunities for the best law students this would appear to be more successful in the clinics where there are high numbers of applicants. Applicant numbers for individual clinics in the 4 universities range from 0 to 125. The location of clinics in the curriculum clearly plays a role in determining the number of applicants. UNHAShas the highest number of clinic applicants, even though the clinic itself is not compulsory because clinic participation satisfies the practice requirement18. In UNSRI, where clinics are now compulsory, the evaluation team was informed by the Vice-Dean that applications have dropped to zero in this current semester and clinics are not functioning at the moment. According to UNSRI’s new Curriculum enacted in August 2012, the clinic is now part of the practice subjects which are compulsory. However with regard to clinics, students have to choose between three subjects: KKN (Social Service Work), KKL (Field Work) or the Legal Clinic. According to the Vice￾Dean, students are now required to take a legal skills pre-requisite which has produced a “transitional” period with applications expected to sharply increase in the next academic year when the new compulsory structure is fully in place. The Vice-Dean also informed the evaluation team that there is a possibility to adopt the internship program as an approach to KKL (Field Work). With respect to clinic activity in Palembang, E2J informed the evaluation team that their recent visit identified an operational criminal law clinic with 2 students and an environmental law clinic with 3 students. The reason for the difference in information is unclear. (ii) The engagement of law school administrators also varies greatly from an apparent lack of interest on the part of the Dean at UNHAS (the only Dean with whom we were not able to speak) to strong assurances of commitment on the part of the 3 other Deans. In UNSRI, the Vice-Dean has taken a great personal interest in promoting the clinics as part of the compulsory curriculum. In regard to the 2 other institutions, the next 2 years will reveal whether the verbal commitments by the Deans are matched by budgetary and curricular developments. (iii) As already noted, none of the 4 schools have instituted a budget line-item to cover costs associated with the clinics. This raises important questions as to sustainability, of which E2J staff are well aware. Except where clinics are part of the compulsory curriculum, it is not clear that law faculty will teach clinics as part of their regular teaching load or as an “overload” course. After E2J support for this teaching ends, this also raises issues of sustainability. 17 The Makassar criminal law clinic decided to allow all the students (124) to attend what they called the “lecture” part of the clinic and 10 to participate in the field component. This again reflects the very different understandings of what purpose the classroom component is supposed to serve. 18 It is part of a group of subjects called Legal Training and Practice (PLKH) where students have to choose among subjects. It also requires students to take relevant courses before they can participate in clinics. For example, before students are allowed to take the Civil Clinic, they are required to follow the civil law and civil procedural law courses. 20 (iv) A crucial issue, already mentioned above, for the sustainability of the enhanced capacity of law schools for clinical education, centers on their placement in the curriculum. Finding 3: Integration of Clinics into the Curriculum Unless clinics are fully integrated into the curriculum as a compulsory subject (whether in the manner adopted in UNSRI (University of Sriwijaya) or in some other way) their sustainability will be in doubt. The lack of such integration in a manner that is designed to allow students to participate in real clinical practice is also an institutional issue of paramount importance. Enhanced Capacity of Professors Without a baseline for comparison it is not possible to accurately assess the extent to which the capacity of lecturers to provide practice-oriented legal education has been enhanced. Some general assessments are nonetheless possible based on evaluation team meetings with faculty, students, and CSOs stakeholders. It is clear that the capacity of the individual faculty who have participated in the UW LLM program has been significantly enhanced. But given the small number of 10 participants, their lack of seniority, and the likelihood that some, if not many, of them will pursue PhDs or other professional opportunities, it is not clear how their personal increase in knowledge and capacity will carry over to their institutions. Institutional hierarchies and the relative youth of the graduates mitigate against the LLM participants having a significant impact on the capacity of other members of their faculties most of whom are more senior. We also understand that many of those who have completed the LLM may wish to use it as a springboard for Ph.D. programs or other professional advancement but the evaluation team did not meet with the students so cannot verify this. E2J also reported that the project expended significant time and effort identifying, selecting and preparing faculty which were difficult to find primarily due to lack of English language skills among lecturers eligible for the program. Given all of the above, and not discounting the value of international exposure, there is some question as to how much the investment in a small number of LLM graduates will contribute to the target institutions and to meeting the overall goals of E2J. In regard to other faculty, the opaque nature of the selection process for clinical teaching has resulted in a situation where some faculty are assigned to clinics outside of their area of substantive expertise. This is not likely to result in a significant enhancement of capacity. The wide variation in the time commitment of faculty to their clinics also appears to reflect some lack of commitment (and to some extent lack of opportunities) to acquire the knowledge and skills needed to enhance their capacity. Finding 4: Training Approach The fact that only limited numbers of faculty were brought to training sessions in Jakarta and other cities also limited the impact that training could have on the clinical faculty as a whole. E2J is addressing this in the Year 4 Workplan by 21 organizing training workshops at each university so that more faculty can participate. This development is well-advised. As E2J progress reports demonstrate, they have carried out a significant number of training sessions on curriculum, research and pedagogy for more than 400 faculty members. But from a systemic perspective it is clear that, given the depth of the problem, much more than these trainings would be required to impart to many of the faculty the necessary knowledge and skills to make them truly effective clinical teachers and real partners for the CSOs with which their university is engaging. All clinical faculty need to be included and training for each faculty person needs to be of substantial duration (e.g. 2 weeks or more) and accompanied by observation, feedback and further training. Training teachers, who have for many years engaged in rote learning lecture￾style pedagogy, to develop and use highly interactive teaching methods aimed at increasing critical thinking and analytical skills, rather than “book knowledge”, is challenging and cannot be done easily or quickly. Unfortunately it appears that the textbooks commissioned by E2J have not served to increase the knowledge and pedagogic skills of the clinical faculty since the textbooks are not yet finished. It also appears to the evaluation team that the announced completion date of April 2014 is unrealistic given the amount of work yet to be done. The Dean at UI noted that if provision of new textbooks were combined with appropriate training they could be useful in enhancing the capacity of the faculty. In UNSRI the faculty asked us where were the E2J textbooks and teaching manuals they had been promised? What has happened at all 4 universities is that individual clinical faculty have prepared their own teaching materials. In some cases these apparently include manuals and SOPs, in others the quality of materials was questioned but not reviewed. It is not apparent to us to what extent faculty are using materials designed to enhance analytical capabilities. We should also note that in our discussions with Kemitraan/Asia Foundation staff who are in charge of preparing the textbooks, serious questions emerged as to whether these textbooks are in reality designed to teach analytical skills of the kind required for real clinical work and legal practice. In other words, based on this discussion, our impression is that while the textbooks incorporate a wider variety of teaching materials they are still based upon a pedagogic model where students are “taught” what the correct analysis of an issue or case is rather than being prepared to engage in their own analysis. In other words, the development of critical analytical skills, which is at the very core of practice-oriented analytical pedagogy seems to be missing. For example, rather than giving students actual case excerpts or other appropriate materials to analyze in exercises designed to develop their own skills, students are given summaries of cases accompanied by the correct analysis of the case. This is in fact essentially the antithesis of critical skills pedagogy so the approach in the text books needs to be reviewed. Enhanced Capacity of Students As stated above, we take the reference to students’ capacity in Question 2 to refer to the enhancement of their capacity to develop the kinds of analytical skills that will make them effective lawyers and advocates when they engage in legal practice, particularly in 22 regard to public service and justice sector reform. Preliminarily, we must note that we found a wide variation in the clinical experience of students and the opportunities with which they were provided. For example, the 2 students in the civil law clinic at USU indicated that they had not learned from being occasional passive observers for short periods in random trials. On the other hand, the students at USU who worked with Pusaka all indicated that they had profited greatly from their clinical experience. This reflects what the vast majority of students reported in the focus groups. The universal complaint of these students was not that they did not learn a great deal from the clinics but rather that they could have learned so much more. Finding 5: Learning Clinical Skills The students universally attributed this perceived lost opportunity to learn more clinical skills to the structural and systemic impediments created by the lack of integration of the clinical courses into the curriculum in a manner that would have encouraged full clinical practice. In order to enhance effectively the capacity of students to engage in legal practice in public service and other settings, a solution must be found by all the law schools to the scheduling conflicts that arise when students are participating in clinics at the same time as they have their normal heavy course load. A variety of factors are currently mitigating against students obtaining what they repeatedly called “the maximum” benefit from their clinics. These are included in the Table below: Table 3: Factors Impacting on Students Learning 1. The current structure in which students are expected to spend only 4 weeks outside of the classroom based on a requirement of only 2 hours per week of participation at their CSOs or FJIs. 2. Court schedules and the activities of CSOs (such as field investigations, interviews, counseling, etc.) cannot be arranged in advance to fit a predetermined 2 hour window when students can plan to be in the field without scheduling conflicts. Students strongly desire having a structure in which they can spend substantial blocks of time at their CSOs so as to be able to engage in actual practical activities rather than passive observation. 3. Some schools do not require, or do not enforce prerequisites means that students may arrive at their clinical settings without the adequate background knowledge required to engage in real clinical work. 4. The underlying structural impediments identified by virtually all the students we spoke with were the elective rather than compulsory place of the clinics in curricula and 2 rather than 4 or more units of credit. 5. Offering clinics to students at stages of their legal education where they have the busiest course load (e.g., 4th-6th semesters) rather than in their last semesters when they are working on their thesis and could combine their thesis topic and research with their clinical practice (as some students were very 23 effectively able to do in their 7th, 8th, and later semesters). Finding 6: Contribution of CSOs The increased capacity of students to engage in effective legal practice appears to be largely due to the contribution of CSOs to the clinical experience offered students rather than to their learning in the classroom part of the clinic. D. Enhanced Capacity of CSOs Although there was variation in the way in which CSOs engaged with the students, the evaluation team found that on the whole CSOs were committed and active partners and provided students with important opportunities to increase their practice-oriented knowledge and skills. Although there is no baseline data, it appears that participating in E2J increased the capacity of CSOs to provide a valuable practice-oriented experience for student participants. CSOs indicated that they benefited from training and advice provided through E2J activities but also that they would have liked much more of these activities. The increase in capacity of CSOs is indicated, for example, by the way in which they created new procedures and resources for working with the clinic students. Most CSOs, for example, informed us that they had adopted SOPs or manuals, mentoring, and evaluation procedures to ensure that students were participating fully. Several CSOs explained that students had to make weekly reports or journal entries that were evaluated by CSO staff. Some of the CSOs also assigned a mentor to each student. In other words, CSOs learned from their interaction with students how to be more effective providers of practice-oriented legal education. As already noted above, the core cooperative sphere of activity through which students benefited from the clinics was in their intensive interaction with the CSOs, not in the classroom. The lack of communication of law faculty with the CSOs and the lack of sustained engagement of law faculty with the students while they were working at the CSOs prevented the program from fully achieving its stated goals. The CSOs also stated that the structural impediments noted above by students also prevented them from benefiting themselves and the students “to the maximum.” E. Enhanced Capacity in the Formal Justice Sector Because E2J programs designed to increase the capacity of justice sector institutions to provide practice-oriented training and education is only at an early stage we have no data on which to base findings or conclusions. But the E2J Year 4 Workplan envisages such activities which may provide the basis for future evaluation. 5.3 QUESTION III How effectively has E2J improved the capacity of justice sector CSOs to effectively advocate for and promote justice sector reform and how is that improvement measured and reflected in concrete activities? 24 Capacity Building Capacity building activities for CSOs have been conducted through workshops together with university partners and by providing research grants. CSOs which were interviewed expressed positive responses to the benefits of research grants to strengthen their capacity to conduct evidence based research. However no baseline data is available to measure the increase in the capacity of CSOs to effectively advocate for and promote justice sector reform. Capacity varies significantly among E2J partners. Some partners have strengths in research and advocacy in the justice sector, as well as institutional stability, such as LBH Jakarta, PUSAKA and KOPEL. Some other CSOs have not been in contact with legal issues. For example, WALHI conducted their last litigation in 1997 and are not using environmental law instruments which has made it difficult for them to work effectively with the Environmental Clinic in UNSRI. The capacity of CSO partners is critical to determine the success of clinics. The CSO selection process during the initial phase of the program was conducted with 300 CSOs and finally narrowed the selection down to 19 participating CSOs and 5 core CSOs. In some areas, such as Jakarta and Medan, the selection process may have been easier than in smaller cities such as Palembang where there are not many CSOs eligible to be legal clinic partners. The availability of an appropriate partner is essential in each area to ensure the success of a clinic. CSOs did receive research grants to strengthen their capacity to conduct research in their core area. All CSOs interviewed expressed positive responses to the benefits of research grants to strengthen their capacity to conduct evidence based research. It appears that collaborative research conducted by CSOs and Faculty has benefited faculty and CSOs in strengthening the quality of research from the academic perspective and bridging communication between CSOs and faculty. In the initial phase of the program, capacity building of CSO partners has been conducted through workshops and training in which CSO representatives and faculty members have participated. The workshops and training include subjects on research methodology and how to conduct academic research. The training content is more relevant for faculty. Some CSOs noted that although the capacity building training in academic research was of benefit to CSOs, it is less directly beneficial than training in advocacy research. No activities have been conducted so far to directly strengthen the capacity of CSOs to support, advocate and/or monitor justice sector reform. But some CSOs, such as LBH Jakarta, have conducted research on labor cases by reviewing court decisions and collecting lessons learned from previous LBH Jakarta’s cases related to labor. This research helped LBH Jakarta to identify important issues in labor laws and is used to conduct advocacy in labor issues. PUSAKA, together with Magenta and LBH APIK Makassar, also conducted research implementing Law Number 11/2012 on the Juvenile Court. Both have used the research to conduct advocacy relevant to their areas of work. Finding 7: Improving Capacity on Research and Advocacy E2J has improved the capacity of CSOs in research but has not been able to support development of strong linkages among CSOs, faculties and FJIs. 25 Improvements to the curriculum, faculty and the overall pedagogical approach are necessary to produce law students who will be more effective advocates and monitors that are motivated to pursue justice sector advocacy careers. The 4 law schools visited by the evaluation team have implemented changes to the curriculum in a variety of ways. UNSRI has been the most successful in addressing challenges in its curriculum by integrating its clinics with other relevant subjects. Other law schools are still struggling with time management conflicts. However all students in the 4 law schools are motivated and enthusiastic about the clinics. Some of them expressed their interest to pursue justice sector careers including all 9 students interviewed at UNHAS. Although the 4 universities visited have inserted clinic courses into the curriculum, in reality clinics have not been seen as unique subjects which demand practical experience for students outside classes, or different from other subjects which are taught in classrooms. Students spend the same number of hours in the classroom and doing practical work with the CSOs. For example if a clinic is worth 2 credits then students are only required to spend 1.5 hours in 1 week in class or with the CSO. All CSOs expressed concern about lack of time available preventing students from getting maximum benefit from the process of knowledge transfer and capacity strengthening. In contrast with the legal clinic program, the internship program seems to have more impact on students career choices due to the increased interaction time. The internship program allows students to spend 2 to 5 days a week for a duration of 3 to 4 months with CSOs or FJIs. And because they spend more time with CSOs or FJIs, students are able to participate intensively with CSOs and FJIs: giving legal assistance to clients (internship in LKBH FHUI, LBH Jakarta and WCC Palembang); providing legal drafting assistance to develop village regulations (internship in YKL); conducting case investigations (internship in LBH Jakarta and KOPEL); or assisting and observing criminal cases administration (in Palembang Prosecution Office). Some students who join internship programs in CSOs decided to join as CSO staff. In LBH Jakarta for example, 2 of 3 interns who participated in KALABAHU19 have joined LBH Jakarta. In the Women’s Crisis Centre in Palembang, 2 students decided to join WCC. Almost all CSOs consider that an important benefit of this program is to introduce students to the work of CSOs which will assist with recruitment of lawyers by CSOs, and ultimately benefit the sustainability and rejuvenation process of CSOs. The recruitment and rejuvenation process of staff in CSOs has been hampered due to lack of student interest in joining CSOs. KOPEL has developed several internship programs in collaboration with several universities in Makassar outside of the E2J program. From a total of 10 interns, 6 have decided to join KOPEL. 19 A number of the LBH organizations have routine structured training programs for final year law students. The training programs are generally referred to as KALABAHU. 26 Cooperation and collaboration among law faculties, students, and CSOs in creating a framework for sustainable capacity development for advocacy and support for justice sector reform All CSOs indicated that the effort to develop relationships with the Law Faculties is important and useful. LBH APIK Makassar stated that research cooperation with the faculty provides an advantage in introducing research programs to law enforcement agencies. Research collaboration between faculty and CSOs has helped them to develop initial engagement and relationships. However this relationship does not contribute directly to clinics, because in the implementation of clinics, almost all CSOs stated that the relationship with the faculty in the management of clinics was poor and all CSOs are concerned about the lack of engagement by faculty lecturers. Some CSOs said that faculty only contacted CSOs at the beginning of class and then again at the end of the course to determine the student’s marks. Almost all CSOs stated that they wanted to be involved in the development of teaching plans and teaching materials which is logical since the largest portion of the CLE classes are the responsibility of the CSOs. Since many of the faculty lack practice-oriented teaching experience, increased engagement and knowledge sharing with CSOs would likely increase the practice￾oriented teaching capacity of those faculty. In some clinics where relationships between CSOs and Faculty are well developed, such as the Women and Children Clinic at USU and UI, there had been pre-existing relationships between faculty and CSOs. For example, PUSAKA Indonesia some Board members are also professors at USU, and in UI the faculty of Women & Children Clinic has been active in gender networks which include gender activists from several CSOs. Although students have been inspired (mostly by CSOs) and are very keen to learn about legal practice more than theory, limited time at the clinics prevents them from being effectively integrated into CSO activities and from learning practice-oriented skills. UNSRI has successfully run an internship program in collaboration with the Attorney General which was created based on personal relationships. UNSRI is currently facing a challenge because the officer who has been the point of contact in the prosecution office, has been replaced by another officer. Very limited cooperation with FJIs has been developed for example by inviting judges or prosecutors to be resource persons during the Public Career Days, or by conducting court watch or court monitoring, for which a MoU was not needed. The issue of the poor relationships between the Law School and FJIs is not surprising considering that historically almost no formal relationship has been well developed between FJIs and schools of law. The absence of systematic measures and strategic steps constructed by the program to follow up the MoU will result in a network not being developed. E2J could play a role as an effective bridge to establish these relationships at the national and provincial levels. Finding 8: Linking with Formal Justice Institutions Despite the Memorandum of Understanding (MoU) between E2J and Formal Justice Institutions being developed, networks and relationships among faculty, CSOs and the FJI, has been weak. 27 Response to E2J initiatives and results that indicate sustainability Faculty and administrators for the most part express positive responses to E2J initiatives, but law students and CSOs are stronger and demand the continuation of the legal clinics. Levels of commitment and ownership at the 4 universities are varied which suggests different levels of potential sustainability. None of the 4 universities has secured funding for CLE continuation even though initial discussions to secure funding have taken place in UI and UNSRI. The proportion of budget for CSOs in E2J is only 25% of the overall program budget. It is used for research grants and capacity building of CSOs. The research grants are often viewed as incentives for CSOs to continue supporting clinics. CSOs did not receive specific budgets to carry out activities related to the legal clinics but the CSOs did spend money to support the program. Despite not having a particular budget available for CSOs for clinic implementation, it is the CSOs which are the most committed and supportive of the clinics. In some cases, such as KOPEL, the CSO even provided a budget for transportation and lunch for students. KOPEL also provides budget for faculty who attend classes conducted in KOPEL’s office. LBH Palembang and YKL both provide transportation budgets for students who are involved in advocacy outside the cities or in the islands. Some faculty expressed doubts that their CSO partners would be willing to continue to cooperate with clinics because of budget issues. But the CSOs with which the evaluation team met all indicated that they continue to be willing to take part in the program and are committed to the sustainability of clinics, even though many CSOs have difficulties financing their day-to-day activities i.e. WCC, LBH Palembang and Magenta. Additional responsibility for managing students and financing them in some cases, is certainly not easy for some CSOs, although this is less the case for CSOs with more budget stability i.e. KOPEL, LBH Jakarta or PUSAKA Indonesia. Finding 9: Sustainability CSOs and law students demand the continuation of the legal clinics. CSOs are very committed and supportive of the clinics which indicates a strong potential for sustainability. Levels of commitment and ownership at the 4 universities are varied which suggests different levels of potential sustainability, and only 1 university (UNPAD), has secured funding for clinic continuation. 5.4 GENDER EQUALITY USAID’s approach to gender equality and women’s empowerment is organized around guiding principles aligned to specific aspects of gender equality including: (1) integration of gender; (2) women’s inclusion to foster equality; (3) building partners across a wide￾range of stakeholders; (4) addressing gender gaps with respect to access to technology; and, (5) looking at success and failures, best practices etc. through the capturing of results and outcomes. These served as guiding principles for this 28 evaluation team’s review of how well E2J reflected USAID’s commitment to promoting gender equality as stated in the USAID policy “gender equality and female empowerment are fundamental to the realization of human rights and key to effective and sustainable development outcomes.”20 Specifically the team addressed 3 sub-questions: To what extent have the principles of gender equality been taken into consideration with respect to the objective of channeling well trained/ educated graduates into the public service? With respect to the principles of gender equality, in the E2J supported women’s and children’s clinics, these principles are demonstrated in programming and in the gender issues with which the clinics are dealing. E2J has supported positive initiatives on gender such as through the sub-grant to Pusaka Indonesia, Medan (one of the 5 core CSOs) to carry out legal research initiatives on gender in 3 cities. As part of their larger research study on Restorative Justice for Juveniles, they supported the following research on women’s and gender issues:– by WCC Palembang - handling of cases of violence against women and children in Palembang’s formal justice system; LBH Apik Makassar - civil and criminal justice system settlement of domestic violence cases in Makassar); and, the Women and Gender Study Center at the University of Indonesia in Jakarta (carrying out socialization/campaign and advocacy on sexual violence law in the educational environment for students and civil society in Depok).21 Finding 10: Mainstreaming Gender to Develop Capacity in Graduates A widened integration of gender into all aspects of programming and clinics would increase the capacity of graduates in terms of understanding and applying gender equality principles. To what extent was a gender equality preliminary analysis carried out (including sex disaggregated data) to help enhance the capacity building of law schools, professors, students, CSOs and the formal justice sector to improve practice￾oriented teaching of law? With the exception of the 2 clinics focused on women’s and children’s issues which carried out gender assessments, most of the other clinics visited in our sample did not have evidence of having carried out gender assessment or gender analysis. At the 2 women’s and children’s clinics, students participating at these clinics (and students who attended the criminal law clinic which was located at the Women’s and Children’s Clinic at USU) said that they learned a great deal about the impact of gender. Another capacity building example is the cases handled by LBH Jakarta which focused on 20 USAID Evaluation Policy, 2. Purposes of Evaluation, January, 2011, p. 3. 21 Quarterly Report, October – December, 2013, pp. 25-27. 29 violence against women. During the evaluation team’s interviews, those experiences were described as “eye-opening for male students”. Several interviewees said that such experiences had changed their perceptions of gender in a powerful way. Creating capacity and introducing gender issues into all the clinics and into the curriculum from the outset would widen capacity building in gender equality and better prepare graduates once public sector hiring recommences. In the Performance Monitoring and Evaluation Planning Reports, 22 including the Revised Performance Monitoring and Evaluation Plan/Program Indicators, 23 it is clear that targets including sex disaggregated numbers were identified and in many cases were achieved. For example, 60 law lecturers and 38 CSO practitioners were trained in 2012 in practice-oriented legal education of which 33% were women.24 Similarly in legal research training, 56 law lecturers of which 35% were women and 42 CSO practitioners, of which 28% were female, were trained in 2012.25 Finding 11: Gender Tools to Identify Issues and Opportunities Gender assessments and gender profiles that identify gender gaps, and suggest solutions to overcome them, don’t appear to have been used throughout E2J. In our sample, issues such as differential access to justice and violence against women are only included in the women’s and children’s clinic except for the criminal law clinic in 1 location managed by the same CSO. There is no evident gender mainstreaming strategy in E2J. Sex disaggregated data however has been collected effectively in E2J. What gender equality activities have been implemented (or planned) to improve the capacity of CSOs and universities to support, advocate and/or monitor justice sector reform? Was the amount of resources (both financial and non-monetary) which have been invested sufficient to ensure effective integration of gender? Most CSOs interviewed had a superior understanding of gender. Although some demonstrated gender knowledge, some interviews with University lecturers demonstrated limited understanding of gender and how best it could be integrated into the clinical programs. These views have led to differential application of laws as reflected in the recent 2013 study by Komnas Perempuan (National Commission on Violence Against Women) which identified 342 discriminatory laws and ordinances enacted since 2009 at the district level in Indonesia. This type of issue needs to be surfaced and understood as part of legal education. Ideas to improve understanding of gender included exposure to these issues through the courts, inclusion in clinics and 22 Performance Monitoring and Evaluation Plan (PMP Report, 15 March, 2011 – 31 March, 2012. 23 Revised Performance Monitoring and Evaluation Plan/Program Indicators submitted to reflect Year 3 Workplan 24 Ibid, p.1. 25 Ibid, p.3. 30 guest speakers discussing gender and Islam which some students mentioned had been done and was helpful to build understanding and overcoming resistance. Except for the women’s and children’s clinics, the amount of resources (both financial and non-monetary), invested were not targeted on gender mainstreaming and linked with a corresponding budget. Without that, it is difficult to achieve gender results, which is why many organizations strongly favor allocating resources specifically to gender from the beginning of projects. Finding 12: Deepening Understanding of the Importance of Gender Some interviews reflected a misunderstanding about the role of gender mainstreaming by some faculty and students interviewed- i.e. in some interviews the local gender equality expert found that when she asked about gender the most common response was that it was about men and women and not about how the legal system affects men and women differentially. In addition, gender assessments should be better utilized to identify issues and opportunities. Findings from these could then be used in a gender strategy to better promote gender equality. Finally, lessons learned should help the project maximize its gender equality results, e.g., when male students were exposed to gender in the Women’s and Children’s clinic, they claimed it was eye-opening and changed their views. 5.5 SUMMARY OF FINDINGS Finding 1: Impact on Public Sector Careers Although there is certainly evidence of actions being taken by and through E2J to inform and encourage students to consider public service careers, there is no evidence that this has had an effect on their entering the public service. This is primarily because justice sector institutions, including the prosecutors offices, are not hiring at this time. Finding 2: Law School Capacity Improvements The institutional capacity of all 4 law schools was substantially increased by E2J because all 4 implemented multiple legal clinics that brought far greater numbers of students into clinical settings than had previously been the case. Finding 3: Integration of Clinics into the Curriculum Unless clinics are fully integrated into the curriculum as a compulsory subject (whether in the manner adopted at UNSRI (University of Sriwijaya) or in some other way) their sustainability will be in doubt. The lack of such integration in a manner that is designed to allow students to participate in real clinical practice is also an institutional issue of paramount importance. Finding 4: Training Approach The fact that only limited numbers of faculty were brought to training sessions in Jakarta and other cities limited the impact that training has on the clinical faculty as a whole. E2J is addressing this in the Year 4 Workplan by organizing training workshops at each university so that more faculty can participate. This development is well-advised. 31 Finding 5: Learning Clinical Skills The students universally attributed this perceived lost opportunity to learn more clinical skills to the structural and systemic impediments created by the lack of integration of the clinical courses into the curriculum in a manner that would have encouraged full clinical practice. Finding 6: Contribution of CSOs The increased capacity of students to engage in effective legal practice appears to be largely due to the contribution of CSOs to the clinical experience offered students rather than to their learning in the classroom part of the clinic. Finding 7: Improving Capacity on Research and Advocacy E2J has improved the capacity of CSOs in research but has not been able to support development of strong linkages among CSOs, faculties and FJIs. Improvements to the curriculum, faculty and the overall pedagogical approach are necessary to produce law students who will be more effective advocates and monitors that are motivated to pursue justice sector advocacy careers. Finding 8: Linking with Formal Justice Institutions Despite the Memorandum of Understanding (MoU) between E2J and Formal Justice Institutions being developed, networks and relationships among faculty, CSOs and the FJI, has been weak. Finding 9: Sustainability CSOs and law students demand the continuation of the legal clinics. CSOs are very committed and supportive of the clinics which indicates a strong potential for sustainability. Levels of commitment and ownership at the 4 universities are varied which suggests different levels of potential sustainability, and only 1 university (UNPAD), has secured funding for clinic continuation. Finding 10: Mainstreaming Gender to Develop Capacity in Graduates A widened integration of gender into all aspects of programming and clinics would increase the capacity of graduates in terms of understanding and applying gender equality principles. Finding 11: Gender Tools to Identify Issues and Opportunities Gender assessments and gender profiles that identify gender gaps, and suggest solutions to overcome them, don’t appear to have been used throughout E2J. In our sample, issues such as differential access to justice and violence against women are only included in the women’s and children’s clinic except for the criminal law clinic in 1 location managed by the same CSO. There is no evident gender mainstreaming strategy in E2J. Sex disaggregated data however has been collected effectively in E2J. Finding 12: Deepening Understanding of the Importance of Gender Some interviews reflected a misunderstanding about the role of gender mainstreaming by some faculty and students interviewed- i.e. in some interviews the local gender 32 equality expert found that when she asked about gender the most common response was that it was about men and women and not about how the legal system affects men and women differentially. In addition, gender assessments should be better utilized to identify issues and opportunities. Findings from these could then be used in a gender strategy to better promote gender equality. Finally, lessons learned should help the project maximize its gender equality results, e.g., when male students were exposed to gender in the Women’s and Children’s clinic, they claimed it was eye-opening and changed their views. 5.6 CONCLUSIONS 1. Successes for Students and CSOs: E2J has been successful in creating opportunities to channel the talent and energies of highly motivated students into clinical settings at highly receptive CSOs. Students universally value such opportunities and request deeper engagement in practice￾oriented activities. 2. Structural and Systemic Challenges: Maximum benefit from these opportunities has not been achieved due to a variety of structural and systemic challenges. These include: faculty competence and engagement; effective integration of clinics into law curricula; lack of gender mainstreaming; students not acquiring substantive legal knowledge and analytical skills before they begin their work with CSOs; and, systemic limitations of time that students can commit to practice-oriented work with CSOs. 3. Effectiveness of E2J: CSOs in general have taken the lead in structuring meaningful practice-oriented educational activities for student participants. As a result the CSOs rather than the faculty classroom settings have emerged as the locus where E2J goals are most effectively being achieved. 4. Integration of Gender Equality: Overall, although sex disaggregated data has been collected effectively, gender has not been mainstreamed throughout all E2J programming and clinics except when it is the focus of Women’s and Children’s clinics where it is well integrated and includes a comprehensive understanding of how the legal system affects men and women differently. 5. Future Direction of E2J: Based on addressing the challenges identified by E2J and the findings in this evaluation, the workplan for year 4 of E2J is on the right track and needs to focus on the planned activities and on addressing the findings identified in this report. 33 5.7 RECOMMENDATIONS Recommendation 1 – Overall Sustainability E2J should prioritize systemic integration of legal clinics into the compulsory curriculum of law faculties as a necessary condition for the sustainability of legal clinics. Gender should be mainstreamed throughout the system including within all legal clinics. Recommendation 2 – Individualized Strategies E2J should identify the particular opportunities for effective implementation at each university and adopt individualized strategies for achieving ownership, sustainability, and greater faculty engagement. If and when appropriate, universities, clinics, or CSO partners should be dropped from E2J so as to focus resources most effectively. Recommendation 3 – Internships and Careers Some CSOs and clinics have demonstrated the value of internships or internship-like engagement in leading students into public service careers. E2J should explore and develop internship and externship components of its programs and their integration into law faculty curricula as the best available vehicle to channel students into career opportunities in CSOs and FJIs. Recommendation 4 - Strengthening CSO Advocacy Capacity Support strengthening the capacity of CSOs to conduct research based advocacy. This should include capacity building to produce policy oriented research products, capacity building on advocacy strategies particularly with justice sector institutions, and engagement with media. Research products should be focused on the primary targets of the CSOs. In most cases the primary audience will be justice sector institutions and government policy-makers. Capacity building can be addressed d in a comprehensive training of ‘Advocacy Strategy’ which consists of a variety of relevant materials and supports linkages among stakeholders. In conducting this activity, E2J should draw on the strengths of partner organizations and external expertise to encourage cross learning and transfer of skills and knowledge. Approaches to capacity building need to consider skills development relevant to CSOs’ area of advocacy, so that training materials can be applied and practiced in day-to-day activities. Therefore 1 training session should be dedicated to improving the capacity of CSOs in different core areas (civil, criminal, environment, women & children). Recommendation 5 – Working with the Formal Justice Institutions Focus should be on the core objective of E2J which is providing students with knowledge, skills, opportunities and incentives to pursue careers in the public service and contribute to justice sector reform efforts. As a potential opportunity, currently the Supreme Court is seeking organizations that can operate court-based legal services posts in district courts throughout Indonesia as implementation of Supreme Court 34 Regulation No. 1/2014 (“PERMA 1/2014) on Legal Services for the Poor. 26 E2J should initiate discussions involving the Supreme Court, legal aid partners and university partners on this opportunity. LBHs also have structured training programs called KALABAHU27 that are not dissimilar to the objectives of legal clinics. To achieve this objective, E2J should encourage legal aid partners to consider ways to link their internships, KALABAHU program, legal clinics and the Supreme Court’s legal consultation posts. E2J may also consider using funding to support student participation in LBH’s KALABAHU programs. Recommendation 6 – Gender Mainstreaming Data needs to continue to be sex disaggregated. Gender mainstreaming needs to be fully integrated from the wording of initial contracts throughout project design, gender analysis and in gender responsive programming in all legal clinics, including gender sensitive textbooks, teaching and other E2J materials. Recommendation 7 – Continuing USAID Support USAID should take advantage of the aspirations, talents and energies of Indonesian law students who express dissatisfaction with business as usual coursework and demand a more practice oriented approach that will better prepare them for careers as legal professionals. USAID should continue to support clinical legal education as the best means to direct the most highly motivated and capable students into public service careers. Recommendation 8 - Continuing USAID Support for Faculty Development USAID should also enable broader opportunities for faculty to engage in international collaborations and to participate in LLM or Summer Programs in American law schools, especially those, like UW or the University of Hawaii, with an orientation towards Asian law. Unless substantial numbers of faculty can be involved effectively in such programs it is difficult to see how the demand for practice oriented, skills-based clinical education can be met. To be effective this will undoubtedly require additional opportunities for required intensive English language preparation of the kind offered by some US based institutions (e.g., the collaborative program between the East-West Center and the University of Hawaii) so as to permit more faculty to be eligible to participate. 26 In January 2014, the Chief Justice of Indonesia signed Supreme Court Regulation No. 1/2014 (“PERMA 1/2014) regarding Legal Services for the Poor. This PERMA consist of regulation which helps people, especially vulnerable groups, to obtain legal services and increase their access to seek justice in the court. 27 KALABAHU are generally organized to include an intensive series of workshops on practical legal topics, generally held over 1-2 weeks, followed with several days of organized field visits. High profile alumni are invited to present the lectures. The program, advertised for final year university students, provides structured training on various aspects of legal practice. It also offers internship opportunities to students. 35 ANNEXES 36 ANNEX I: EVALUATION STATEMENT OF WORK 37 38 39 40 41 ANNEX II: DATA COLLECTION INSTRUMENTS I. How effectively has the E2J project been able to increase the capacity of law faculties (law schools) to channel well trained/educated graduates into public service? Question 1: What steps have been taken to prepare administrators and law teachers to inform and encourage their students about public service careers, how effective have they been, and how has effectiveness been measured? How well have law school administrators cooperated with and provided resources for E2J activities directed towards the goal of increasing capacity in this area? Question 2: How have law faculty curricula and teaching methods been adapted to reflect E2J goals (“well trained graduates” oriented towards public service) and how widely and effectively have they been implemented? Question 3: How successfully has E2J implemented cooperation with Justice Sector institutions and CSOs in order to provide opportunities for students leading to public service careers? What challenges have been encountered and how have they been met? Question 4: What has been the impact on students of “Public Service Career Days” and other measures to encourage students, and overcome the disincentives towards such careers? How will E2J reach out to the majority of students who have not attended or participated in such events? How are these activities being targeted to reach the best students? Question 5: To what extent have the principles of gender equality been taken into consideration with respect to the objective of channeling well trained/educated graduates into public service? II. How effectively has the E2J project enhanced the capacity of law schools, professors, students, CSOs and the formal justice sector to improve practice￾oriented teaching of law? Question 1. How have been the various E2J programs designed to develop the capacity of law teachers to provide an enriched education experience been implemented in practice and how has their success been measured? How fully have teachers responded to this initiative, and what incentives have been provided to encourage them and to enable them to devote sufficient time to such activities? Question 2: How have the challenges and obstacles to the effectiveness of clinical courses identified in evaluation documents been addressed and what plans are there to deal with them in the remaining 2 years of the project? Question 3: How effectively and through what measures has E2J “enhanced the capacity” of CSOs and the formal justice sector to contribute to practice-oriented teaching of law? What responses have such measure met with and how successfully has E2J adapted its program to take such responses into account? 42 Question 4: What has been the student response to practice-oriented law teaching, how has E2J responded to these responses, and how have responses been measured? Question 5: To what extent was a gender equality preliminary analysis used (including sex disaggregated data) to help enhance the capacity building of law schools, professors, students, CSOs and the formal justice sector to improve practice-oriented teaching of law? III. How effectively has the E2J project been in improving the capacity of CSOs and universities to support, advocate, and/or monitor justice sector reform? Question 1. How has E2J improved the capacity of justice sector CSOs to effectively advocate for and promote justice sector reform and how is that improvement measured and reflected in concrete activities? Question 2. How have the 8 law schools implemented curricular, staff, and pedagogic changes to produce students who will be more effective advocates or monitors and who will be motivated to pursues justice sector advocacy careers? Question 3. How effective has been the cooperation and collaboration between law faculties, students, and CSOs in in creating a framework for sustainable capacity development for advocacy and support of justice sector reform? Question 4. How have law students, faculty, and CSOs responded to E2J initiatives and are there results that indicate that they will continue to develop and use capacity for justice sector advocacy and support beyond the end of E2J? Question 5. What gender equality activities have been implemented (or planned) to improve the capacity of CSOs and universities to support, advocate, and/or monitor justice sector reform? Were the resources (both financial and non-monetary) invested sufficient to ensure effective integration of gender? 43 ANNEX III : SOURCES OF INFORMATION The following list of documents were reviewed by the Evaluation Team • AIPJ (Australian Indonesia Partnership for Justice) Documents - Selected reports by other donors addressing Indonesia's rule of law sector needs including documents prepared by AIPJ and related Australian Government documents • Changes for Justice (C4J) documents (as applicable to links to public service recruitment practices and training of new entrants) • E2J Annual Reports and Annexes • E2J Annual Work Plan, Year 4 – 15 March 2014 – 14 March 2015 • E2J Cooperative Agreement with Program Description • E2J Performance Monitoring Plan (PMP) • E2J Quarterly Reports and Annexes • E2J Work Plans Years 1-3 • Government of Indonesia's Blue Print for the Supreme Court and Attorney General's Office (as applicable to links to public service recruitment practices and training of new entrants) (in Bahasa Indonesia) • Ministry of National Development Planning (BAPPENAS) October 2012 Evaluation Report of the E2J program (in Bahasa Indonesia) • Modules, tools and training materials developed by the E2J program • USAID - "USAID Evaluation Policy" report and ADS 203 • USAID Strategy for Indonesia 2014-2018- Investing in Indonesia: A Stronger Indonesia advancing national and global development • Selected University documents such as Implementasi Program E2J Fakultas hokum UNSRI – Palembang, March 14, 2014 44 APPENDIX IV: SCHEDULE OF ACTIVITIES E2J EVALUATION PROGRAM Dates Time Activities Source Persons Place of Activities JAKARTA Sunday, 03/09/2014 17.30 – 19.00 Internal Team meeting Evaluator Team meeting The Four Seasons Hotel, Jl H.R Rasuna Said, Kuningan Jakarta Monday, 03/10/2014 08.30 – 12.00 Meeting with USAID - Debriefing with Acting Mission Director of USAID - Meeting with DG Director - Meeting with Rule of Law team 1. Nancy Fisher-Gormley, Acting Mission Director 2. Miles Toder, Director, Office of Democratic Governance at USAID Jakarta 3. Miranda Jolicoeur, USAID 4. Nenad Bago, USAID USAID’s Office, Sarana Jaya Building, Jl. Budi Kemuliaan I/1 (behind the Indosat building), Jakarta Pusat Monday, 03/10/2014 14.00 – 16.30 Meeting with E2J Team (TAF & Kemitraan) 1. Laurel MacLaren, Deputy Country Director of The Asia Foundation 2. Kala Finn, E2J Chief of Party 3. Tomi Suryo Utomo, E2J 4. Laode Muhamad Syarif (E2J/Kemitraan) 5. Detania Sukardja, E2J/Kemitraan 6. Yura Pratama, E2J 7. Novrieda Hendrika, E2J 8. Nita Herita, E2J The Asia Foundation, Jl. Adityawarman No. 40, Kebayoran, Jakarta Selatan Tuesday, 03/11/2013 09.00 – 11.00 Meeting with E2J Team 1. Kala Finn, E2J Chief of Party 2. Tomi Suryo Utomo, E2J 3. Laode Muhamad Syarif, E2J/Kemitraan 4. Detania Sukardja, E2J/Kemitraan 5. Yura Pratama, E2J The Four Seasons Hotel, Executive Room, Floor 17th Jl H.R Rasuna Said, Kuningan Jakarta 45 Dates Time Activities Source Persons Place of Activities Tuesday, 03/11/2013 11.00 – 12.30 Meeting with CSO 1. Febi Yonesta, Director of LBH Jakarta, 2. Muhamad Isnur, LBH Jakarta 3. Pratiwi Febry, LBH Jakarta 4. Veronica, PKWJ UI/Magenta 5. Diyah Stiawati, PKWJ UI/Magenta 6. FR. Yohana T Wardhani, PWJ UI/Magenta The Four Seasons Hotel, Executive Room, Floor 17th, Jl H.R Rasuna Said, Kuningan Jakarta Tuesday, 03/11/2013 14.30 – 16.30 Meeting with Legal Clinic Partners, University of Indonesia 1. Topo Santoso, Dean FH UI 2. Tien Handayani, Lecture, Women & Children Clinic 3. Lidwina Inge, Lecture, Women & Children Clinic 4. Feby Mutiara, Lecture, Criminal Clinic, Vice Coordinator of LKBH UI Dean’s Room, Faculty of Law, University of Indonesia, Kampus UI Depok, 16426 Tuesday, 03/11/2013 16.30 – 18.00 Meeting with students of Legal Clinic, University of Indonesia 1. Gratianus Prikasetya, student 2. Eric Andhika Sianipar, student 3. Evi Dita Pratiwi, student 4. Rossy Dame Lasria, student Faculty Lounge, Faculty of Law, University of Indonesia, Kampus UI Depok, 16426 MEDAN Wednesday, 03/12/2014 07.50 Flight to Medan Wednesday, 03/12/2014 14.00 – 16.15 Meeting with PUSAKA, SAHDAR and students of Legal Clinic (USU) 1. Fatwa Fadilah, Director of PUSAKA Indonesia 2. Elizabeth, Pusaka Indonesia, PUSAKA Indonesia 3. Marjoko, Pusaka Indonesia, PUSAKA Indonesia Pusaka Indonesia’s office Jl. Kenanga Sari No. 20. Lk.III, Kel. Tanjung Sari, Kec. Medan Selayang 46 Dates Time Activities Source Persons Place of Activities 4. M. Mitra Lubis, PUSAKA Indonesia 5. TR Arif Faisal, SAHDAR 6. Benni Iskandar, student 7. Elly Syafitri Harahap, student 8. Arija Br. Ginting, student 9. Elvira Francisca, student 10.Dyna Sri Wahyuni, student 11.Irma Sari, student 12.Murida Khairuna, student 13.Rahmad Ramadhan, student Wednesday, 03/12/2014 16.30 – 18.00 Meeting with E2J Senior Consultant 1. Prof. Ningrum Sirait, Vice Rector of USU, Senior Consultant of E2J Program Vice Rector Room, 3rd floor, USU, Jl. Universitas No.4 Kampus USU Medan, 20155 Thursday, 03/13/2014 09.00 – 12.00 Meeting with Legal Clinic Partners, Universitas Sumatera Utara (USU), and students of Legal Clinic 1. Runtung Sitepu, Dean of Faculty of Law USU, Lecturer of Civil Clinic 2. Budiman Ginting, Vice Dean, Lecturer of Anti-Corruption Clinic 3. Maria Kaban, Lecturer, Civil Clinic 4. Rafiqoh, Lecture, Criminal Clinic 5. Agusmidah, Lecturer, Women & Children Clinic 6. Rosmalinda, Lecturer, Women & Children Clinic 7. Chairiah Ella Sari Siregar, student 8. Kusuma Ambarwati, student 9. Solatiah Nasution, student 10.Elly Syafitri Harahap, student 11.Benni Iskandar, student Faculty of Law, USU, Jl. Universitas No.4 Kampus USU Medan, 20155 47 Dates Time Activities Source Persons Place of Activities 12.Dwi Pranoto, student 13.Meirita Pakpahan, student 14.Robert, student 15.Rivera Wijaya, student 16.Arija Br. Ginting, student 17.Rafiqah Lubis, student 18.Erick MP Kaban, student 19.Dian Ekowati, student 20.Boy Laksamana, student PALEMBANG Thursday, 03/13/2014 17.40 Flight to Palembang Friday, 03/14/2014 08.30 – 11.00 Meeting with Legal Clinic Partners, Universitas Sriwijaya (UNSRI) and students of Law Clinic 1. Prof. Amzulian Rifai, Dean Faculty of Law, UNSRI 2. Fahmi Yoesmar, Vice Dean, Faculty of Law, UNSRI 3. Ruben Achmad, Lecturer, Criminal Law Clinic 4. Amrullah Arfan, Lecturer, Civil Law Clinic 5. Agus Ngadino, Lecturer, Environmental Clinic 6. Amir Syarifudin, Lecturer, Anti-Corruption Clinic 7. Artha Febriansyah, Lecturer, Criminal Law Clinic 8. Murad Zaidan, Lecturer 9. Indah Febriani, Lecturer, Anti-Corruption Clinic 10.Hamonangan Albariansyah, Lecturer, Civil Law Clinic 11.Andreas Yudistira, student Faculty of Law, Kampus Bukit, Universitas Sriwijaya (UNSRI), Palembang, Phone 0711 580063 48 Dates Time Activities Source Persons Place of Activities 12.Hotlin Marpaung, student 13.Ary Kastia Peranginangin, student 14.MGS Fachri Dwisetya, student Friday, 03/14/2014 11.00 – 12.00 Team 1 – Class Room Observation, UNSRI Class Room Observation Faculty of Law, Kampus Bukit, Universitas Sriwijaya (UNSRI), Palembang, South Sumatera, Phone 0711 580063 Friday, 03/14/2014 11.00 – 12.00 Team 2 – Meeting with Students of Law Clinic, UNSRI 1. Mgs. Fachri Dwisetya, student 2. Hotlin Marpaung, student 3. Andreas Yudistira, student 4. Sesario Julia, student 5. Ary Kasti Perangin-angin, student 6. M. Syahri Ramdhan, student 7. M. Fathur Rizki, student 8. M. Reza Pratama, student Faculty of Law, Kampus Bukit, Universitas Sriwijaya (UNSRI), Palembang, South Sumatera, Phone 0711 580063 Friday, 03/14/2014 14.30 – 16.30 Meeting with WCC, WALHI Sumsel, LBH Palembang, and students of law clinic 1. Yeni Roslaini, Director of Women Crisis Centre (WCC) 2. Desma Diana, WCC 3. Yesi Ariyani, WCC 4. Nelly Hartati, WCC 5. Aprili Firdaus, Director of LBH Palembang 6. Norman Cegame, WALHI Sumsel 7. Rifa Apriati, WALHI Sumsel 8. Mgs Fachri Dwisetya, student 9. Devi Gafriansyah, student WCC’s Office, Jl. Kapten Marzuki, Ir. Rukun Jaya No. 2450 IT I, Palembang, South Sumeatera Friday, 03/15/2014 19.15 Flight back to Jakarta 49 Dates Time Activities Source Persons Place of Activities MAKASSAR Sunday, 03/16/2014 09.40 Flight to Makassar Sunday, 03/16/2014 16.00 – 17.00 Internal Team meeting Evaluator Team meeting Arya Duta Hotel, Makassar, South Sulawesi Monday, 03/17/2014 09.00 – 11.30 Meeting with University Partners, Universitas Hassanuddin (UNHAS) 1. Ahmad, Lecture, Civil Law Clinic 2. Maskun, Lecture, Environment Law Clinic 3. Kaisar, Lecture, Criminal Law Clinic 4. Nur Azisa, Lecture, Criminal Law Clinic 5. Amir Ilyas, Lecture, Anti-Corruption Clinic Faculty of Law, Universitas Hassanuddin (UNHAS), Jl. Perintis Kemerdekaan Km.10 Makassar, 90245 Sulawesi Selatan Monday, 03/17/2014 11.30 – 12.30 Meeting with students of Law Clinic, UNHAS 1. Dwi Randi Sulistiono, student 2. Andi Rafia, student 3. St. Dwi Adiyah Pratiwi, student 4. Lestari Sainuddin, student 5. Budi Setiawan, student 6. Ahmad Fauzi, student 7. Joko Fitriyanto, student 8. Rini Ariani Said, student Faculty of Law, Universitas Hassanuddin (UNHAS), Jl. Perintis Kemerdekaan Km.10 Makassar, 90245 South Sulawesi Monday, 03/17/2014 14.00 – 16.00 Meeting with CSO Partners 1. Sjamsudin Alimsyah, Director of KOPEL 2. Madjid Bati, KOPEL 3. Anwar Razak, KOPEL 4. Elsa Damayanti, KOPEL 5. Lukman, YKL 6. Rosmiati Sain, LBH APIK Makassar 7. Sulastri, LBH APIK Makassar KOPEL’s office, Jl. Batua Raya 9 No. 3, Makassar, South Sulawesi Tuesday, 03/18/2014 09.55 Flight back to Jakarta 50 Dates Time Activities Source Persons Place of Activities JAKARTA Wednesday, 03/19/2014 09.00 – 11.00 17:00-17:45 Meeting with E2J Australia Embassy 1. Sandra Hamid, Country Director of The Asia Foundation 2. Kala Finn, E2J Chief of Party Doddy Kusadrianto c.c. Luke Arnold, Justice and Democratic Governance Assistance, Department of Foreign Affairs and Trade The Four Seasons Hotel, Executive Room, Floor 17th Jl H.R Rasuna Said, Kuningan Jakarta Thursday, 03/20/2014 10.00 – 12.00 Internal Team meeting Evaluator Team meeting The Four Seasons Hotel, Executive Room, Floor 17th Jl H.R Rasuna Said, Kuningan Jakarta Thursday, 03/20/2014 16.00 – 18.00 Meeting with E2J Senior Consultant Gregory Churchill, Senior Consultant E2J The Four Seasons Hotel, Executive Room, Floor 17th Jl H.R Rasuna Said, Kuningan Jakarta Friday, 03/21/2014 09.00 – 12.00 Presentation of Preliminary Findings 1. USAID team 2. E2J team USAID’s Office, Sarana Jaya Building, Jl. Budi Kemuliaan I/1 (behind the Indosat building), Jakarta Pusat CP: Miranda Jolicoeur Thursday 03/27/2014 10.00 - 12:00 Presentation on the Results of C4J (Changes for Justice) Participatory Gender Mapping Presented by Leya Cattleya (Attended by Lily Purba of evaluation team) USAID’s Office, Sarana Jaya Building, Jl. Budi Kemuliaan I/1 (behind the Indosat building), Jakarta Pusat 51 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov 52