Final evaluation of Component One of the Access to Justice program Report prepared by Matthew Coghlan and Selma Hayati, consultants to the Access to Justice program, funded by the United States Agency for International Development (USAID) and implemented by The Asia Foundation September 2012 Dili, Timor-Leste DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contents Acknowledgments .............................................................................................................. 1 Acronyms and abbreviations .............................................................................................. 2 Executive summary ............................................................................................................ 3 Introduction ........................................................................................................................ 4 Program and partnerships ................................................................................................. 7 1. Program .................................................................................................................. 8 1.1 Design ............................................................................................................... 8 1.2 Implementation ................................................................................................ 11 2. Partnerships .......................................................................................................... 13 2.1 Funding............................................................................................................ 13 2.2 Implementation ................................................................................................ 14 Objective evaluations ....................................................................................................... 15 Objective 1: To enhance the quality of legal aid services ............................................. 15 Activity 1.1 Mediation and legal skills training for legal aid partners ......................... 16 1.1.1 Litigation skills training .............................................................................. 16 1.1.2 Specialized short-term training .................................................................. 17 1.1.3 CFJ scholarships ...................................................................................... 17 1.1.4 Case handling guidelines .......................................................................... 18 1.1.5 New lawyers kit ......................................................................................... 18 1.1.6 Non-litigation dialogue .............................................................................. 18 1.1.7 Legal terminology glossary ....................................................................... 19 1.1.8 GBV manual .............................................................................................. 19 Activity 1.2 Case management review ...................................................................... 20 Activity 1.3 Administrative, management and financial training for legal aid partners21 Activity 1.4 Assessing the impact of legal aid services ............................................. 22 1.4.1 Monitoring and evaluation training ............................................................ 22 1.4.2 Client survey and mid-term evaluation ...................................................... 22 1.4.3 Women’s legal needs research ................................................................. 23 Activity 1.5 Legal aid coordination body ................................................................... 24 Activity 1.6 Public policy development of legal aid bill .............................................. 25 Activity 1.7 Legal aid for women ............................................................................... 25 1.7.1 Pilot study on DNA test for paternity suits ................................................. 25 1.7.2 Identify and publicize best practices ......................................................... 26 1.7.3 Improved data collection of partners’ cases .............................................. 26 1.7.4 Support for women’s legal aid organization .............................................. 26 Activity 1.8 Form networks and support groups for women lawyers ......................... 27 1.8.1 Women lawyers network meetings ........................................................... 27 1.8.2 Provide specialized trainings and support change programs .................... 27 Objective 2: To expand the reach of legal aid services ................................................ 28 Activity 2.1 Mobile legal aid services ........................................................................ 29 2.1.1 FECM ........................................................................................................ 29 2.1.2 FFSO ........................................................................................................ 29 2.1.3 LBH Liberta ............................................................................................... 30 2.1.4 Roundtable meeting with legal aid partners .............................................. 31 Activity 2.2 Contact group meetings/legal aid clinics ................................................ 31 Activity 2.3 Links with government and organizations providing legal aid ................. 31 Activity 2.4 Paralegal program .................................................................................. 32 2.4.1 FECM ........................................................................................................ 33 2.4.2 FFSO ........................................................................................................ 34 2.4.3 Training ..................................................................................................... 34 Activity 2.5 Specialized services for GBV victims (VSS) ........................................... 35 Objective 3: To promote skills-based education ........................................................... 35 Activity 3.1 Map career paths and practices ............................................................. 36 Activity 3.2 Stanford Law School (SLS) project ......................................................... 37 3.2.1 Textbooks ................................................................................................. 37 3.2.2 Foreign visits ............................................................................................. 38 3.2.3 National exchanges .................................................................................. 39 Objective 4: To improve public access to legal education ............................................ 39 Activity 4.1 Design and disseminate legal aid messages .......................................... 40 4.1.1 Communications expert and MOJ support (Activity 4.1.2) ........................ 40 4.1.3 Support to partners in conducting legal dissemination activities ............... 40 4.1.4 Promotion of legal aid at universities ........................................................ 41 Activity 4.2 Court Watch (JSMP, LRU)...................................................................... 41 Conclusions ..................................................................................................................... 44 Recommendations ........................................................................................................... 47 Annex A – Terms of reference ......................................................................................... 50 Annex B – Interviews ....................................................................................................... 53 Final Evaluation: Component One, Access to Justice Program | 2012 1 Acknowledgments The evaluators wish to acknowledge the hard work and patience of the program’s staff, especially Kerry Brogan, Adelio Tilman, Geoffrey Swenson and Timotio de Deus. Also, we thank Abetu, our driver, for his prompt arrivals and safe deliveries. Final Evaluation: Component One, Access to Justice Program | 2012 2 Acronyms and abbreviations AATL Lawyers Association of Timor-Leste ADR Alternative Dispute Resolution AusAID Australian Agency for International Development CLE Community Legal Education EU European Union FECM Fundasaun Edukasaun Comunidade Matebian FFSO Fundasaun Fatu Sinai Oecusse Foundation The Asia Foundation GBV Gender Based Violence GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit/ the German Society for International Cooperation JSMP Judicial System Monitoring Program Justice Facility AusAID Timor-Leste Justice Sector Support Facility LBH Legal Aid Institution/Organization CFJ Legal Training Center M&E Monitoring and Evaluation MFA Ministry of Foreign Affairs MOJ Ministry of Justice MOH Ministry of Health MSS Ministry of Social Solidarity OPD Office of the Public Defender OPG Office of the Prosecutor General SEPI Secretary of State of the Promotion of Equality SLS Stanford Law School UNDP United Nations Development Programme UNMIT United Nations Integrated Mission in Timor-Leste UNTL National University of Timor-Lorosa’e USAID United States Agency for International Development VSS Victims Support Service Final Evaluation: Component One, Access to Justice Program | 2012 3 Executive summary The Access to Justice program (ATJ), implemented by The Asia Foundation and funded by the United States Agency for International Development (USAID) started in Timor￾Leste in 2002, shortly after the birth of this new nation. Under Indonesian rule, the country’s justice sector was characterized by the lack of the rule of law, high levels of impunity, lack of access to justice, and reliance on traditional dispute resolution mechanisms. Within that early setting of weak governance, the new government worked to build the legal system almost from scratch, with strong contributions from international donors. To improve access to justice, the government established the Office of the Public Defender (OPD) to provide legal aid to those that could least afford to pay for legal assistance and representation. Unfortunately, the expansion of the OPD’s geographic coverage has been glacial, with its defenders delivering services from urban centers to urban clients until now. Into this void stepped the Access to Justice program, which supported legal aid organizations (LBHs) through capacity building and budget assistance to deliver legal services freely to the smaller urban centers and rural clients, including those in the most remote areas. Since then, the LBH’s legal aid lawyers have advised and acted for thousands of clients in criminal cases and civil matters. Under the latest and final extension of the program, ATJ established two pilot paralegal programs that have recruited paralegals from distant communities to collect cases to refer to legal aid lawyers, and to run mediations with local leaders to international standards. These paralegals have now forwarded and overseen hundreds of disputes. This report evaluates this final extension of the program and offers recommendations to The Asia Foundation and USAID to strengthen and guide the work of both organizations in Timor-Leste’s justice sector in future. The evaluators have reviewed the program’s design and implementation including considering its monitoring and evaluation and its sustainability, and investigated the relevance, effectiveness and impact of the many activities under its four objectives as follows: 1. To enhance the quality of legal aid services 2. To expand the reach of legal aid services 3. To promote skills-based legal education 4. To improve public access to legal information. We conclude that the program was extremely well-designed and well-implemented, although it might have benefited from more conscious and overt strategizing in some areas. Further, we find that the activities were very relevant to the program’s objectives with the exception of one or two activities that were more relevant to wider justice sector issues at this stage. Though many of the program’s objectives were achieved successfully, there might have been follow up on one activity in particular to cement that success and increase the likelihood of a positive impact on access to justice. This could be determined by further study to understand the terms of the resolution of disputes and the link between court monitoring and government policy/court practice more clearly. Final Evaluation: Component One, Access to Justice Program | 2012 4 Introduction The final evaluation of the three-year extension to the Access to Justice program (ATJ) (FY 2010 to FY 2012) was conducted between 21 August to 21 September 2012. The evaluation team was composed of two consultants, Matthew Coghlan, a governance consultant based in Jakarta, Indonesia and Selma Hayati who has been working in the NGO sector in Timor-Leste for more than 11 years. Sector and problem This evaluation is a final review of a three-year Access to Justice program, implemented by The Asia Foundation with funding from the United States Agency for International Development (USAID) from October 2009 to September 2012. This program phase formed part of a 10-year program which commenced in 2002. The program has two main components, namely Component One: Improved Legal Access and Legal Aid and Component Two: Strengthening Parliament Research and Public Awareness. This evaluation covers only Component One of the program which has four objectives: 1. To enhance the quality of legal aid services 2. To expand the reach of legal aid services 3. To promote skills-based legal education 4. To improve public access to legal information The program was developed to address problems within the community and formal judicial system which limit access to justice. The majority of Timorese people lack access to justice due to factors including poverty, lack of access to information, poor infrastructure, illiteracy, low levels of education, and issues of gender. ATJ and its partners also reflected that the development of formal judicial institutions is progressing slowly with only a small number of Public Defenders, courts and Prosecutors in only four district capitals. This makes it difficult for many people to reach legal services, particularly poor people in remote areas, and vulnerable groups such as women and children. There is a limited number of private lawyers providing free and pro bono legal services in Tmor-Leste and the majority of private lawyers are based in the capital, Dili. High prevalence of gender-based violence (GBV) in Timor-Leste is an additional concern that needs to be addressed by the program in providing free legal services through legal aid organizations, including to GBV victims. The presence of legal aid organizations offering free legal services and taking on a high number of cases demonstrates how legal aid organizations can function to bring communities closer to the formal justice system and provide access to legal information. However the program also needs to improve the delivery legal services of the legal aid organizations and institutional capacity of the organizations to support the legal services. The program encourages its partners, Fundasaun Fatu Sinai Oecusse (FFSO), Fundasaun Edukasaun Communidade Matebian (FECM), LBH Liberta, and Victims Support Service (VSS, part of JSMP, the Judicial System Monitoring Program) to Final Evaluation: Component One, Access to Justice Program | 2012 5 enhance their capacity by strengthening non-litigation and litigation knowledge, building skills and administration, case management and financial training. The program also encourages the partners’ legal aid lawyers to participate in mandatory training at the Legal Training Center (CFJ) as required by the Private Lawyers Law 2008/111 . The program has also been supporting and filling the gap by providing technical and theoretical skills training for the partners’ legal aid lawyers who have not yet passed the CFJ training. During the 3-year period of the program there were two draft laws which affected the sustainability of legal aid organizations and their services. The Private Lawyers Law requiring private lawyers to undertake formal training provided by CFJ has raised concerns within legal aid organizations and The Asia Foundation, as the training requires all legal aid organizations to participate in full time training in Dili. On the one hand, legal aid lawyers are required to be State-qualified, but on the other hand, they are taken from their organizations for training and cannot continue to provide legal services. Furthermore, the Legal Aid Bill has been challenging the legal aid partners in relation to future financial sustainability, particularly with the decreasing number of donors in Timor￾Leste focusing on the justice sector. In relation to legislation, the program has been facilitating the legal aid organizations’ participation and ability to be heard by the Government, although draft legislation does not address matters such as the limited number of Public Defenders providing legal assistance to communities in remote areas and the reluctance of Public Defenders to visit and be pro-active in taking cases from these communities. Limited access to legal information is also a concern among the legal aid organizations and ATJ. It affects the ability of vulnerable community members to understand and access their right to justice. For instance the Government faces the challenge of ensuring there is public understanding of domestic violence as a public crime, and that the Law is upheld by communities in conjunction with formal judicial actors and institutions. For that reason, ATJ has been supporting the legal aid partners to conduct community legal education programs through mobile clinics, which actively disseminate legal information; working with local government institutions, other NGOs and judicial actors. The program also addressed challenges facing the formal education of law students. Law students and lecturers lack legal textbooks specific to the Timorese context, and there is a need within universities to have a teaching methodology, which develops students’ critical thinking and understanding of real problems for communities seeking access to justice. On the other side there is an increasing number of law graduates from three Dili universities, namely UNTL, UNPAS, UNITAL, with most graduates attracted jobs within the Government, private organizations, and commercial companies. A smaller number of law graduates and private lawyers want to offer legal services to legal aid organizations. Through Component One of the program, ATJ funded four legal aid organizations, FFSO, FECM, VSS and LBH Liberta to provide free legal aid assistance for litigation and non-litigation cases in three court jurisdictions - Oecusse, Baucau and Dili. The legal aid 1 Final Evaluation: Component One, Access to Justice Program | 2012 6 program provides legal assistance and information to the people of Timor, particularly those in remote areas where there is very limited access to legal information and assistance through the formal judicial system. The Access to Justice program also funds paralegal programs through FFSO in Oecusse and FECM in Baucau court jurisdiction (which covers the districts of Viqueque and Lautem), to bring legal services closer to remote and isolated communities. Twenty￾seven paralegals are able to support communities’ access to justice by acting as mediation facilitators, community educators and legal assistants to explain legal issues and refer legal concerns to legal aid lawyers, the National Police of Timor-Leste PNTL or the local authorities. These paralegals also report and document cases and provides support to victims of crime. In addition, ATJ has demonstrated its commitment to enhancing the quality of legal aid services through various other activities, including:  Facilitating mediation and legal skills training for legal aid partners, including litigation skills training, non-litigation dialogue, specialized short-term training, and the Judicial Training Centre (Centru Formasaun Juridika CFJ) scholarships.  Producing educational materials for legal professionals including a Tetum legal terminology glossary, a new lawyers kit, two volumes of a gender-based violence manual, case handling guidelines and a case management review.  Institutional strengthening on administration, management and financial training for legal aid partners, including monitoring and evaluation (M&E) training.  Supporting legal aid organizations to have broader networks at the regional level, including study exchange with LBH Jakarta and LBH Apik in Jakarta, Indonesia as well as inviting regional legal aid lawyers to participate in the LBH Fund Raising Seminar in May 2012.  Contributing to the debate about the sustainability of legal aid sector, by providing technical support to the Government in developing a draft legal aid law. ATJ also facilitated legal aid organizations to provide input in relation to the Private Lawyers Law, which requires all private lawyers to participate in two years of mandatory training provided by the CFJ. The Law was initially planned to take effect in 2012 and would permit only lawyers who had undertaken the mandatory training to exercise their duties before the court. However, the organizations advocated that the transitional period for the implementation of the Law to be extended to 2015. The evaluators appreciated the approaches and input of legal aid organizations toward the Law, which provides an opportunity for legal aid lawyers to continue their work for an additional two years.  Supporting women legal aid lawyers and enhanced services for women clients.  Conducting legal information dissemination activities, including through legal aid organizations and paralegals.  The program has also established a legal education program in cooperation with Stanford Law School (SLS), which has produced legal textbooks on Professional Responsibilities and Contracts in three different languages- Tetum, Portuguese and English. These books provide students in Timor-Leste with the option to learn from textbooks according to their language capacity. The program has also facilitated exchange visits between lecturers and students of UNTL and SLS. Objectives/methods Final Evaluation: Component One, Access to Justice Program | 2012 7 This evaluation covers the program overall, its partnerships and the objectives of the program including the program design, implementation and monitoring and evaluation (M&E) of ATJ. The partnership evaluation focuses on coordination and collaboration between The Asia Foundation, donors, Government and other stakeholders in the justice sector. The evaluation also analyzes the relevancy of the program, examining program effectiveness and its impact with regard to the program’s objectives. The evaluation concludes by identifying priority issues which may require further assistance and examining the sustainability of the legal aid sector, financially, operationally, legally, and with regard to human resource conditions. This evaluation is based on a desk review of key documents, which include work plans, semi-annual reports submitted to USAID, the performance monitoring and evaluation plans (PMEPs), a mid-term evaluation, the 2011 client survey, legal education resources produced by The Asia Foundation, legal aid partners reports, the case database, handbooks/manuals and activity assessments. The evaluators also conducted structured and semi-structured/tailored interviews with donors, UNMIT, Government, the Office of Prosecutor General, the Office of Public Defender, the CFJ, ATJ’s legal aid partners, two private lawyers/training facilitators, AATL and university interviewees, as well as four focus groups with FECM/FFSO legal aid lawyers and paralegals. The evaluators have been guided by four of the five OECD evaluation criteria in evaluating the program, its partnerships and its objectives and activities, and reaching their findings. Not all evaluations or findings refer to all of the criteria used except in cases where it made sense to use them all. In doing so, we have used the following definitions:  Effectiveness is the extent to which the objectives/activities were achieved  Relevance is the extent to which the objectives/activities are consistent with beneficiaries’ requirements and country needs  Impacts are the positive and negative, primary and secondary long-term effects produced by the objectives/activities, directly or indirectly, intended or unintended  Sustainability is the continuation of benefits from the objectives/activities after development assistance has been completed2 Program and partnerships This section of this report evaluates the Access to Justice program from an overall perspective, while the following four sections evaluate the individual activities underneath each of the program’s four objectives in substantial detail. The evaluators decided it would be useful to examine two components of the program - the program itself, and the program’s funding and sectoral partnerships – to gain this perspective. For ATJ, we consider its design and implementation, and for the partnerships we review the program’s communication and cooperation with the United States Agency for International Development (USAID) and key justice sector stakeholders from government, the legal profession, universities, and international organizations. 2 http://www.oecd.org/dac/evaluationofdevelopmentprogrammes/2754804.pdf Final Evaluation: Component One, Access to Justice Program | 2012 8 1. Program A program is a set of interventions seeking to attain specific development objectives.3 The Access to Justice program has been running for 10 years now. In evaluating its design and implementation, we have examined the program’s objectives and activities, inputs and outputs, outcomes, and monitoring and evaluation to varying degrees. With such a long history of building capacity, funding service delivery, organizing networks and advocating for access to justice, it is unsurprising that the program’s design is highly relevant and its implementation is generally effective. There is also strong evidence that its impact is usually positive. However, in our view, the program had minor weaknesses in its sustainability strategy and monitoring and evaluation framework that we point out for the sake of completeness. 1.1 Design The program has four objectives that intervene in relation to different elements of access to justice and apply different models of justice sector development. The fundamental elements of access to justice generally consist of legal protection, legal awareness, legal aid/representation, adjudication, enforcement and oversight.4 In summary, ‘organizational capacity development’ focuses on training and equipping justice agencies and supporting management systems and processes; a ‘service-delivery approach’ improves coverage and quality of formal and informal justice services, building on existing strengths and capacities; a ‘problem-solving approach’ seeks to solve problems in justice agencies or services and is often multi-stakeholder; and a ‘thematic approach’ sees justice intervention through a wider lens of human development.5 Objective Element Model 1. To enhance the quality of legal aid services Legal aid/representation Organizational capacity development to support service-delivery approach of LBH legal aid lawyers, with thematic approach for women (VSS legal aid lawyers/women’s legal network) 2. To expand the reach of legal aid services Legal awareness and legal aid/representation Organizational capacity development to support service-delivery approach of LBH legal aid lawyers and paralegals 3. To promote skills-based education Legal awareness Problem-solving approach for TL legal education material and organization capacity building for US law school visits 3 http://www.oecd.org/dac/evaluationofdevelopmentprogrammes/2754804.pdf 4 UNDP Access to Justice Practice Note, p6 5 Marcus Cox, et al, ODE evaluation of Australian Law and Justice Assistance: a Synthesis Report, 2011, pp19-20 Final Evaluation: Component One, Access to Justice Program | 2012 9 4. To improve public access to legal education Legal awareness and oversight Service-delivery approach for JSMP LRU, with organization capacity development for LBH legal aid lawyers and paralegals There is little doubt that the first two objectives are highly relevant for strengthening access to justice in Timor-Leste. The program’s legal aid organization partners (LBHs), FECM, FFSO and LBH Liberta, have been representing clients through legal aid for many years, but they suffered from institutional limitations such as weak or no case management, lack of financial management systems, and their legal aid lawyers lacked essential substantive law knowledge and basic legal skills. Moreover, while the legal aid lawyers could represent clients close to their offices, if the Office of the Public Defender (OPD) was unable or unwilling to, or their clients chose independent counsel, they could not represent clients far from their offices because they did not visit them or they did not have local paralegals referring them. At the same time, many criminal cases and civil matters in remote areas were resolved through informal justice/traditional law. This raises two major problems: first, the state has exclusive jurisdiction to try criminal cases, so the community leaders should not be doing so; and, second, the community leaders use mechanisms and norms for settling disputes that are not always impartial and non￾discriminatory. The program provided its partners with organizational and professional training, and funded their salaries and transport. In this extension, it established two pilot paralegal programs in FECM and FFSO. The case reports, client surveys, capacity building assessment and, to a lesser extent, CLE assessment attest to the legal aid lawyers and paralegals improvement and success in providing legal representation and information. Consequently, they evidence the program’s effectiveness and impact in implementing this objective, although the client surveys do not necessarily reveal the terms of resolved cases or quality of legal services. The burning issue that the three LBHs face with the end of the program is their financial sustainability. During the program, they were supported financially from various sources. For example, ATJ and AusAID’s Justice Facility funded FECM, while ATJ, Caritas Australia, Irish Aid and Oxfam Australia funded FFSO. The Justice Facility terminated funding to FECM because it was dissatisfied with its program performance after 2 years, and ATJ suspended funding after discovering financial mismanagement in mid-2012. Similarly, ATJ suspended funding to FFSO after discovering financial mismanagement in late 2011, and the other donors followed suit. The Access to Justice program continued paying salaries. During the program, ATJ implemented planned and unplanned activities to ensure LBH financial sustainability through donor and state funding. In this regard, ATJ applied the concept of ‘sustainability’ in both its narrow and wide senses. In its narrow sense of donor funding, it assisted FECM and FFSO to seek alternative funding by placing volunteers within each organization, and it encouraged donors to continue funding all three LBHs by organizing a conference in early/mid 2012, to secure their short-term survival. The organizational and professional capacity building that the program provided to LBHs is likely to have increased their attractiveness for short-term funding. It is important to note, too, that at least one of LBHs, FFSO, has been fundraising; for Final Evaluation: Component One, Access to Justice Program | 2012 10 example, it has started a micro-business offering photocopying services and chair/speaker hire, and it has explored the Prime Minister Office’s funds for civil society. In its wide sense of state funding, ATJ advocated for a legal framework to institutionalize support through the draft Legal Aid Bill to guarantee LBH long-term operation. The evaluators find that ATJ used best endeavors to ensure LBH financial sustainability, although we note that it did not include its sustainability strategy in the program design, and it did not plan and write it during the program. The program’s activities to encourage alternative funding were relevant within Timor-Leste, yet we feel that ATJ might have done more outside the country. There is still time to do so, of course, with the volunteers still placed in FECM and FFSO. The evaluators offered suggestions of possible regional funding sources. We observe that these fundraising activities have not attracted many offers so far, but we understand that the EU is encouraging The Asia Foundation to apply for funding. That might be sufficient. Further, the evaluators find that the program’s advocacy for amendments to the draft Legal Aid Bill was highly relevant and extremely effective, but finalization and passage of the law has been delayed by the recent election of a new government, and it appears that the current version does not have the support of all sectors, in particular the OPD although we do not know what changes it would demand. If the law recognizes the legitimate place of LBHs in the country’s legal aid regime by guaranteeing their funding, it will be a major victory for their sustainability, not to mention their ability to delivery services to clients that require them without delay. We note, too, that the Foundation is intending to continue its advocacy on the law. While the program included advocacy in its design, it did not plan and write an advocacy strategy. The evaluators believe that The Asia Foundation would benefit from designing and drafting an advocacy strategy in future to record issues, objectives, targets, and especially resources and progress. Objective 3 responds to the lack of legal educational materials available in the country. On its own, it is strongly placed to increasing legal awareness levels, which is an access to justice element. Yet, there is something of a disconnect between this objective’s primary activity, the Stanford Law School project (SLS project), and the other objectives because it involves preparing legal textbooks for use in the four law schools and building the relationship between SLS and UNTL Law School (UNTL) through student and lecturer visits. The evaluators perceive that, in time, the SLS project can be linked more firmly to legal aid services delivery through, for example, LBH/law student internships (if the law schools use the textbooks for teaching in future), or CFJ’s current use of the first textbook for training private lawyers (if trainees return or decide to practice legal aid). In this respect, we are confident that, once funding is secured to continue the SLS project, it will investigate the range of channels by which it can contribute to strengthening access to justice. Indeed, the evaluators find that SLS project was very effective in preparing legal textbooks on key areas for use in the law schools, as well as the wider legal profession.6 Like FECM and FFSO, JSMP is a long-standing partner of the Access to Justice program (Objective 4). JSMP’s Legal Research Unit monitors hearings in the four district 6 Similar conclusions might be reached about this objective’s secondary activity, the career path mapping study in that it is somewhat detached from the other objectives, but it might be connected to them if it is pursued. Final Evaluation: Component One, Access to Justice Program | 2012 11 courts for fair trial and legal compliance, publishes updates on the health of the country’s justice system, and issues reports on justice themes and disseminates them through press releases/conferences, workshops/seminars, and email distribution lists. In theory, these activities are highly relevant for strengthening access to justice (and the rule of law) because they seek to improve the coverage of legal protection and standard of adjudication. The concern that the evaluators have is that, although the program monitored JSMP effectiveness in producing its documents, it did not evaluate them per se, and it did not monitor or evaluate their impact on legal practice or justice reform; and, more specifically, on its contribution to strengthening access to justice. Despite our concern, most of the interviewees acknowledged the importance and legitimacy of JSMP’s watchdog role (but one questioned the constructiveness of its approach, and another queried the rigor of its monitoring and quality of its research), and believed that it positively influences legal practice and justice reform. Lastly, JSMP is an experienced fundraiser so we expect it to sustain itself in future. Finally, the interviewees that commented on the program’s design thought that it was well-designed because it was consistent in financing LBH and JSMP partners and services, innovative because it added the paralegal and SLS project activities, and flexible because it modified activities if required in consultation with USAID and partners, such as funding more paralegals for FECM to allow it to cover a large area more easily. The evaluators find that the program was well-designed overall. 1.2 Implementation In our evaluation of the program’s design, we reviewed aspects of its implementation that were linked closely to the design’s relevance and sustainability, namely the effectiveness and impact of its activities. Accordingly, we will review other aspects of its implementation here – its leadership/management; resources, processes and systems; and approach – albeit in much less detail than our evaluation of the design. The program’s Chief of Party is a human rights expert with 12 years experience in Timor￾Leste. She led the program and has exhaustive knowledge of it. The evaluators noted her wide and deep network in the country, and the opportunities that it created for activities, particularly advocacy for the extension of the transition period of private lawyers under the Private Lawyers Act, and the facilitation of inclusive consultations for the draft Legal Aid Bill. The program’s Legal Advisor led the pilot paralegal program and Stanford Law School (SLS) project. He studied at SLS and it is likely that his familiarity with it smoothed and strengthened trilateral relations, which would have been especially crucial in managing student interests and UNTL expectations in the many necessary steps leading to publication of the textbooks. The other key program staff included a Senior Program Officer, Program Officer (paralegal), Program Officer (legal aid/information), Program Officer (legal aid), Program Assistant (paralegal), and Program Officer (database/legal aid). The evaluators thought that the caliber of program staff was uniformly high. They provided fact-based analysis of the program’s systems and processes, and they offered convincing explanations of underlying societal and legal issues such as the causes of gender-based violence. We observed that the team culture was very constructive and industrious, which suggests mature and motivating leadership. The regular loss of management staff during the extension must have disrupted Final Evaluation: Component One, Access to Justice Program | 2012 12 implementation to a degree, requiring new recruitment and induction. For the evaluation, this meant that some staff could not discuss the whole extension. The evaluators did not inspect the program’s budget to appraise funding sufficiency and program efficiency. Similarly, we did not investigate its grant-making systems and processes to assess them. However, we explored the program’s M&E systems and processes. The M&E framework consisted of program activities and USAID PMEP. General program M&E comprised annual work plans, semi-annual reports, a mid-term evaluation and this final evaluation. Specific M&E activity consisted of quarterly partner/community visits to monitor program implementation and financial management, client surveys, capacity building assessments and CLE assessments. Further, the program conducted training needs assessments of LBH legal aid lawyers and paralegals to identify and prepare training for them, it introduced a case management system to the LBHs, trained them to use it and improved the system through monitoring its use and providing further training. The program also trained LBHs to strengthen their financial management. These professional and organizational building activities contributed to LBH capacity to provide the program with the information that it required for monitoring and the basis for evaluating. Finally, the monthly roundtable meetings (RTMs) provide all staff with the opportunity to monitor the partners in the program’s office. Most management staff reported that these M&E systems and processes were relevant to specific activities and effective in collecting information. Leadership staff indicated that they wished for more monitoring resources as it is a costly and lengthy exercise. This was especially the case for observing and checking legal aid lawyer and paralegal services, where delivery locations were isolated and large, and their infrastructure and transport is deficient. Accordingly, the program was strategic in its monitoring: checking on interesting cases and difficult issues, and ensuring subsequent visits monitored missed locations. Some program staff thought that LBH legal aid and paralegal reporting was still too descriptive when it should have been more analytical by the end of the program, and top-down monitoring was probably less effective than regular mentoring in achieving the relevant objectives. There seems to have been ongoing confusion, too, between the program and grant staff regarding their roles and responsibilities, with program officers focusing on program monitoring and relying on grant officers for financial monitoring, which they did irregularly. Several of the partners confirmed that, while the program officers visited frequently, they spent their time reviewing program implementation not financial management. In this context, FECM and FFSO mismanaged finances in the final year of ATJ, leading the program to suspend funding to them except for payment of salaries. USAID expressed surprise that these irregularities took place so late in the program. While it noted the recruitment of a dedicated M&E officer was a wise addition for tightening supervision, it wondered if other measures might have been needed too. The Asia Foundation explained that the program’s grant-making system alerted it to the incidents rapidly, it investigated them promptly and addressed them proportionately. Without delving into this further, we find this explanation reliable. We find that the program had relevant and effective M&E systems and processes for monitoring LBHs, particularly their legal aid lawyer and paralegal services delivery, and it prepared innovative and thorough evaluations of their capacity building, clients and legal education. Although, as we have noted, we believe that there were gaps in monitoring Final Evaluation: Component One, Access to Justice Program | 2012 13 the use of the SLS project textbooks and evaluating the impact of JSMP’s Court Watch and related activities. Further, we find the program’s semi-annual reports to have been prepared accurately and systematically. We appreciated the use of headings for achievement/analysis to separate program facts from staff opinions. In closing, it is important to mention the remarks made by two of the four LBHs that the program’s approach towards them was participatory during initial assessments for training needs but tended to impose the courses it had designed and delivered after that. Thus, it failed to seize its own opportunities ‘to empower’ the organizations further; for example, by encouraging them to design and deliver training with assistance from the program. While the evaluators identify with the LBHs desire to be empowered, they recognize the low initial capacity of LBHs and the program’s ambition to build it in three large fields - substantive law, legal skills and organizational management – reasonably quickly without sacrificing quality until the government or legal profession through formal legal training could do so. Where partner capacity was higher, such as the UNTL and JSMP LRU, we note that ATJ’s approach to them was characterized by collaboration. Accordingly, the evaluators find that the program was well-implemented. 2. Partnerships A partnership involves “collaboration to achieve mutually agreed upon objectives”.7 They are often required because organizations cannot achieve their objectives without support, although relationships falling short of partnerships can assist too. Thus, the notion of partnership provides a criterion for analysis. The Asia Foundation’s essential partner for ATJ was its funder, USAID. Further, it communicated, coordinated and collaborated with other justice sector stakeholders in implementing its objectives. 2.1 Funding USAID and The Asia Foundation were partners, collaborating for 10 years on the Access to Justice program. USAID funded the program, and the Foundation implemented it. They agreed to achieve the four objectives in the program’s extension: to enhance the quality of legal services; to expand the reach of legal aid services; to promote skills￾based legal education; and, to improve public access to legal information. They agreed because they shared the same (or sufficiently similar) aspirations for Timor-Leste – that it be a country governed by the rule of law where every person has access to justice – and conclusions regarding the problems with and solutions for its access to justice:  The state should provide essential services including legal services to people who cannot afford it for free or at reduced fee  The Timor-Leste Government has a commitment to fulfill access to justice in the country’s Constitution, including legal aid  The Government’s efforts to do so by establishing the Office of the Public Defender (OPD) to provide legal aid is heading in the right direction but there are ongoing gaps in OPD’s services delivery 7 http://www.oecd.org/dac/evaluationofdevelopmentprogrammes/2754804.pdf Final Evaluation: Component One, Access to Justice Program | 2012 14  The establishment of LBHs was required to fill the OPD’s gaps, and they should continue doing so until the OPD can increase its capacity  The Government should fund LBHs through the passage of the Legal Aid Bill but it is up to the government to decide its legal aid regime8 In meeting with us, USAID emphasized strongly the high regard in which it holds the Foundation for its commitment and professionalism, and this program for its flexibility and innovation. Likewise, the Foundation highlighted the same qualities in USAID’s management. One senior staff said “I have nothing but good words [for USAID’s management]”, noting further its willingness to participate in program delivery like meeting students and its timeliness (even urgency) in approving implementation decisions as required. 2.2 Implementation ATJ communicated, coordinated and collaborated with a range of important Government, legal profession and international organization stakeholders in implementing the program. Some of these relationships amounted to partnerships, if we apply the definition. In relation to Objective 1, we note the following shared goals and joint work: The program collaborated with two private commercial lawyers to design and delivery capacity building on substantive law and legal skills. While the lawyers were paid a reduced fee, they shared the program’s objective because they want to contribute to the development of the country’s justice sector. In advocating for the extension of the transition period for practicing without having completed mandatory training, the program worked with one of the private lawyers who drafted a submission. He shared interests in lobbying for flexibility in the training format, such as part-time or intensive training, to take into account livelihood needs and family responsibilities. It communicated with CFJ and the United Nations Development Programme (UNDP) regarding its proposal to award scholarships to legal aid lawyers in financial need and who were willing to return to legal aid after CFJ training, and they supported this intervention; The program collaborated with CFJ to clarify LBHs misunderstanding that the Portuguese language is a prerequisite for enrollment rather than a mandatory course, and ran a workshop for the trainees using its GBV manual which was highly acclaimed by some individual participants. During the course of the program, the Foundation built regular coordination with other development partners focusing in legal aid sector. The Foundation participated actively 8 There is ongoing legal policy debate regarding this last point: the legal aid regime. The existing regime is a hybrid one, which combines state and non-state legal aid. Some argue that LBHs should be sustained because they fill the OPD’s gaps now, and offer choice later on. Others contend that the OPD should be the sole provider, although it is not clear if they mean immediately despite its limitation in coverage or in the future. Lastly, others take a wait and see approach, especially because the legal profession is in a state of flux with the requirement in the Private Lawyers Act that private lawyers must complete CFJ training to register to practice by December 2015, otherwise they cannot practice – or they do so illegally. The effect of the start of training has been to drain capable lawyers rapidly from private practice, but only refill their ranks in a trickle after two years fulltime training. With the finalization and passage of the Legal Aid Bill, MOJ should clarify its policy. Final Evaluation: Component One, Access to Justice Program | 2012 15 in the National Priority Working Group Four on the Justice Sector and organized a specific sub-group of National Priority Working Group Four on legal aid, which was chaired by the Ministry of Justice (MOJ) and co-chaired by USAID. In advocating for amendments to the draft Legal Aid Bill, the program collaborated with MOJ to organize more inclusive consultations to review the draft. UNMIT was a strong supporter of the program. It invited the Foundation to participate in donor justice sector coordination and the Foundation has not missed the opportunity to advocate for continuing support. DNA testing for paternity suits was undertaken in collaboration with two Australian testing laboratories. The women lawyers network meetings arranged events that female judges and the Prosecutor General participated in like a Christmas lunch, and they collaborated with LBH APIK in Jakarta to organize a visit to understand its fundraising, management and operations with a view to establishing a women’s LBH. The Secretary of State for the Promotion of Equality launched the GBV manual, with the Vice Minister of Justice and Prosecutor General in attendance. SEPI appears to be interested in turning this early cooperation with the program and VSS into a wider three-way collaboration, and this has the potential to result in a partnership. Regarding Objective 2, the courts, OPD and police refer cases to LBHs in situations where the OPD is unable to handle them. Such cooperation is important for access to justice and significant for recognition of LBHs. However, it is informal and does not involve agreement on objectives. Moreover, ATJ and AusAID’s Justice Facility shared goals in funding the legal aid service delivery of FECM and VSS. They shared information and planned collaboratively. The Justice Facility is currently funding AATL, the Lawyers’ Association of Timor-Leste. There has been discussion between the Foundation and Justice Facility concerning the role AATL might play at the end of the program, with the idea of the Justice Facility funding AATL to disburse to LBHs being raised. ATJ has partnered with SLS and UNTL to prepare and publish textbooks for UNTL and to organize and run visits for UNTL to SLS. The Foundation has hosted SLS staff and student visits, including for the launch of the first textbook. Their relationship is a partnership, underpinned by the key SLS staff member working for the Foundation and teaching at SLS. The Foundation signed a Memorandum of Understanding with UNTL. Their relationship is a partnership too, with shared goals in increasing materials and sharing experiences, although the weight of responsibility for carrying out the work has so far rested on the ATJ program and SLS. Still, as the relevant program staff member noted: UNTL is under no obligation to be involved in the program. Objective evaluations Objective 1: To enhance the quality of legal aid services Objective 1 is largely organizational capacity development to support a services-delivery approach in legal awareness and legal aid/representation and litigation/non litigation services. Accordingly, Activities 1.1-1.4 trained the legal aid organizations in substantive knowledge, legal skills and institutional strengthening, which include administrative, Final Evaluation: Component One, Access to Justice Program | 2012 16 management and financial management, and monitored and evaluated their service delivery of legal aid. Further, this objective included a networking and advocacy component in Activities 1.5 and 1.6 that centered on roundtable meetings in Dili and the draft Legal Aid Bill. Finally, it organized substantive law, legal skills and institutional strengthening events for women lawyers under Activities 1.7 and 1.8 to support the establishment of women’s legal aid organizations through networking and support groups for women lawyers, including meetings in Dili and exchanges to Jakarta, which led to an agreement to establish a women’s LBH, like LBH APIK in Jakarta, by spinning off VSS from JSMP in 2013. It is important to be mindful of the feared and actual impact of Private Lawyers Act. The Act requires all private lawyers to complete 15 months theoretical training including Portuguese language if they do not speak it, and 9 months practical training. Initially, it seemed that the start of the course would have a dramatic impact on practicing lawyers numbers, including legal aid lawyers, because the CFJ made available large quotas for early intakes. The LBHs were extremely nervous. They were made more anxious by their misunderstanding that Portuguese language was a prerequisite; and, so, the program organized a dialogue between CFJ and the LBHs to clarify that Portuguese class forms part of the course and is not prerequisite. However, following the testing process, it was clear that capacity of private lawyers in Timor-Leste was low and that the entry requirements for the CFJ were high, so only 14 lawyers started the first course. Indeed, the CFJ was forced to offer a bridging course to 26 applicants that failed to pass the examination. Nonetheless, the impact on the legal aid organizations was significant because its most senior legal aid lawyers have enrolled in one of the three courses to date. Consequently, the legal aid organizations were forced to recruit or promote inexperienced legal aid lawyers. The program responded by revising its training strategy to build substantive law and legal skills, and to continue building case systems and financial management. Moreover, it encouraged legal aid lawyers to attend the course and to return to practice legal aid by offering scholarships to supplement the state subsidy. They targeted legal aid lawyers with dependants that may suffer greater financial hardship. Activity 1.1 Mediation and legal skills training for legal aid partners 1.1.1 Litigation skills training The program’s partners conducted two sessions in roundtable meetings: JSMP LRU on legal analysis and FECM/FFSO on paternity cases. Further, the program delivered training on legal analysis, legal drafting of petitions, evidence, and representing clients (trial simulations). Training needs assessment of partner legal aid lawyers including JSMP has been conducted. It found that the legal aid lawyers have limited legal experience and knowledge. The program redesigned its training based on the assessment. First, it extended training to one week and relocated it to within the legal aid partner organizations; and, second, it changed the method to on-the-job training coupled with mentoring. The training topics for these trainings were: professional ethics, client interviewing and taking instructions, etc. The Legal Aid Lawyers Handbook was a reference tool for them. Pre- and post-testing revealed that their knowledge improved through the training. Furthermore, in order to institutionalize the handbook, ATJ facilitated use of the handbook in training of trainers workshops at the partners’ offices to support developing induction programs. In order to strengthen the capacity of the Final Evaluation: Component One, Access to Justice Program | 2012 17 partners on substantive laws, the program developed trainings on preventative detention, introduction to the new civil law, introduction to the new land law, and introduction to the new family law. The partners acknowledged the training had improved both their knowledge and skills. The evaluators find that the litigation skills trainings were effective, relevant and responsive to the knowledge and skills needs of the partners. It met the target of 20 trainings and identified training needs and developed training and manual/handbook to meet them. The participants showed improvement in knowledge from pre- to post-test. LBH interviewees report that the training’s impact was positive because they applied their knowledge and it increased their confidence to advocate around new laws like the Civil Code. Moreover, the capacity building increased community trust in their legal aid services and their chances of being admitted to the CFJ course. FECM legal aid lawyers explained that they taught paralegals to use the training materials, and paralegals taught local authorities and community leaders. They said that before the communities “thought plates were being broken” if they heard domestic violence, but now they know it is a crime that must be reported to paralegals and then to police. The same legal aid lawyers said that they now require further training to meet their other needs. This training is sustainable if legal aid lawyers continue - or return to - practicing at LBHs and transfer their knowledge to colleagues, and if the materials can be retained for reference. Its sustainability can be enhanced if others, such as the AATL or CFJ, use the materials also. 1.1.2 Specialized short-term training ATJ facilitated one specialized short-term training on Timorese law to facilitate admission into the CFJ course for private lawyers. The training was organized by the CFJ and was conducted by five trainee public defenders. The training focused on the criminal and criminal procedure codes. The other training was planned in collaboration with UNDP and CFJ. The interviewees noted that the specialized short-term training is useful and relevant, as it has improved their analytical skills especially on GBV. However they also expressed concern with the capacity of the trainers who were relatively inexperienced and could not answer the questions during the training. 1.1.3 CFJ scholarships The program selected 8 candidates for scholarships based on the criteria required to work as LBH legal aid lawyers into the CFJ course for private lawyers, which included needs-based, women lawyers and legal aid lawyers with dependants. A commitment to attend the training course and a commitment to work in a partner or legal aid organization after completing the course was required. However, two scholarship recipients withdrew from the program and joined the judge’s course. Additionally, the program organized a meeting of legal aid directors, scholarship recipients and the CFJ director to build and maintain relationships, and to support the recipient’s commitment to legal aid and the LBH’s willingness to receive them. The program disbursed the target number of scholarships but their real effectiveness will be tested when the recipients finish their theoretical training and determine their placement organizations for practical training, and when they complete the course and decide their next employment. With the program ending, the recipients have recently Final Evaluation: Component One, Access to Justice Program | 2012 18 expressed concern that legal aid organizations might not be funded to allow them to return to practice. However, if they do resume working in legal aid, their formal skills and experiences, and private lawyer recognition and registration is likely to improve legal aid services. 1.1.4 Case handling guidelines The program produced and distributed two case-handling guidelines - a basic case management system guideline and a new civil procedure code guideline. During the finalization of this report, ATJ had just updated the three guidelines due to changes in law since the promulgation of the Civil Code, with the updated guidelines to be distributed to partners. However this activity has not been evaluated. 1.1.5 New lawyers kit In 2010-11, the program produced a Legal Aid Lawyers Handbook for legal aid organizations to use internally. The handbook explains what legal aid is, its legal basis, the role of legal aid lawyers, how to become a lawyer, the rights and responsibilities of lawyers, working as a legal aid lawyer, and providing legal assistance. The handbook was piloted in litigation trainings during the year. Follow-up to institutionalize the manual took place in 2011-12 with training-of-trainers workshops at the partners’ offices to support developing induction programs. See litigation skills training above. In one partner interview, an LBH staff member mentioned that her organization relies on the handbook for new staff to learn and understand about legal aid organizations and how to work for a legal aid organization. The handbook is useful and relevant for her organization although the organization does not have a formal induction program for new staff/lawyers. 1.1.6 Non-litigation dialogue In the past three years, the program has worked with its partners to understand their non-litigation dispute resolution practice and to agree on standards of practice. The partners have learned to outline the different forms of alternative dispute resolution (ADR) – negotiation, mediation, conciliation and arbitration – that revealed they blend them, and they do not have or use standardized procedures and principles for mediation facilitated by Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ). The program also held training and a second dialogue on non-litigation which resulted in an agreement between the program and its partners on a non-litigation policy to set out clear practices and principles for mediation, etc. New policy for partners in order to institutionalize a separation of litigation and non-litigation processes, including distinct client engagement letters for litigation and non-litigation matters, and a mediator code of ethics and mediator process guidelines for non-litigation was established. The program’s monitoring of partners non-litigation exposed that they have trouble recognizing their client in mediation; for example, they have acted for both parties in mediation, then acted for one party in litigation if the mediation ends in disagreement. According to interviewees, the mediation training was effective in training the legal aid lawyers and paralegals to increase the impartiality and non-discrimination of procedure and resolution. Now mediators and parties try to understand the problem and solve it. In Final Evaluation: Component One, Access to Justice Program | 2012 19 doing so, they exclude outside interferences such as family demands. However, the program monitoring revealed firmly entrenched issues with mediation, especially in relation to mediating GBV cases instead of referring them to the police. The evaluators conclude that there is evidence that legal aid lawyers and paralegals understand and apply mediation principles and processes. In this respect, this activity is effective, and it might point to their ongoing application after the program ends. However, this activity remains a work-in-progress too, as participant understanding and acceptance of women’s rights is still low in a society transitioning to modernity. The evaluators also note that the Ministry of Justice is currently drafting the Alternative Dispute Resolution (ADR) draft law which also includes a provision on paralegal and non-litigation mechanisms. Legal aid organizations and paralegals have much experience in resolving disputes through non-litigation processes, knowledge and skills to share with the Ministry of Justice as part of policy advocacy. During their interview with the evaluator, the Director of Judicial and Legislation of MOJ stated that the Ministry welcomed the input of legal aid organizations and AATL. This is a good opportunity for legal aid organizations to show their knowledge, skills and experience, and to reflect on whether their organization has successfully institutionalized the knowledge and advocacy skills developed in the non-litigation training workshops. 1.1.7 Legal terminology glossary This activity aimed to provide specific legal term glossary in Tetum to respond to the need of lawyers to understand the legal principles and meaning of terms used in key legislation written in Portuguese. The program commissioned a Timorese lawyer to develop a glossary in 2011-12. The glossary’s preparation was a three-step process: selecting legal terms; researching definitions/examples; and producing a draft. The glossary included around 80 standard legal terms with explanations and examples of use. During the finalization of this report, the activity was ongoing and thus has not been evaluated. Most interviewees including the Ministry of Justice and the President of Court of Appeal (CoA) are aware of the activity and are looking forward to providing comment. The former Rector of UNTL/Director of the National Institute of Language has provided input into the draft glossary book. As the activity has not been evaluated and required linguistic editing, the inclusion of the institute in the finalization of the glossary must be appreciated. 1.1.8 GBV manual The program agreed to a VSS request to produce a manual for legal aid lawyers on how to provide legal aid to GBV victims. The manual was distributed to the Minister and Vice Minister for Justice, President of the Court of Appeal, Secretary of State for the Promotion of Equality (SEPI), CFJ and the Prosecutor General for comments. Their comments were taken into account and the manual was finalized with the Secretary of State for the Promotion of Equality’s endorsement. Further, the program conducted training using the manual with legal aid lawyers from all its partners and further training on it alongside the roundtable meetings to present the partners with practical checklists, visual maps, posters and other tools to help them implement the manual’s practices. The evaluation report of Legal Capacity Building Program noted that 89 percent of the total participants (26 participants) felt the training had increased their knowledge with regards Final Evaluation: Component One, Access to Justice Program | 2012 20 to GBV cases, 74 percent had better knowledge and skills to assist the victims of GBV and 84 percent of participants said that the training had made the manual more useful to them. Thus the training using the GBV manual achieved the objectives as targeted and indicated by PMEP. Furthermore the training required the partners to practice their skills in GBV assistance by employing case studies and simulation exercises. The evaluators also find the GBV manual and the manual training to be relevant guidelines, needed by the partners and other NGOs working on GBV issues, women issues, human rights and government. The evaluators found the GBV manual was being used by SEPI and VSS to facilitate community legal education in Baucau, and in CFJ training acknowledged by the Ministry of Justice. Effective coordination and communication between the program, VSS and SEPI has contributed to the development and use of the manual in relation to the implementation of the Law Against Domestic Violence. Additionally the evaluators conclude the findings of the community legal education (CLE) assessment (conducted under Objective 4) are relevant to ensure the effectiveness of CLE and indirectly the manual. It is also important to note that 77.9 percent of women clients on paternity case, sexual assault, domestic violence and divorce, feel either very comfortable (42 percent) or somewhat comfortable with their legal aid lawyers as concluded by the Client Survey in 2012. It is not clear whether the introduction of the GBV manual or other specific trainings done by ATJ have contributed to this. However as the manual has enhanced the capacity of partners to handle GBV cases, assist victims and facilitate further training on GBV, the evaluators believe it could support the partners’ organizations and individual staff to sustain the knowledge and skills provide legal services to GBV victims with financial support from different donors. Activity 1.2 Case management review This activity aimed to enhance partners’ understanding of the importance of managing cases properly. The program facilitated the creation of a basic management system guideline covering best practices for note-taking, intake and registration procedures, case filing and recording information, and provided a workshop on the guideline for partners. ATJ hired a consultant to visit the partners for one week to assist implementing the system, provide training on it and identify required material/equipment. The program also required monthly updates from partners on system implementation. The consultant reported that the partners recognize the need for appropriate systems and are keen to adopt the procedures. The consultant submitted a report together with a revised version of the guidelines. The program donated filing cabinets and other equipment to the partners for the system. ATJ continued to conduct on-site case management implementation. The program’s visits trained partners in: preparing an outline for office policies such as Codes of Conduct and Case Management Guidelines; file management, including documenting case work and filing case documents; client management, including managing important dates and keeping clients informed; as well as staff supervision. The partners face significant challenges in implementing their new systems properly. The program introduced mandatory reporting for case management by developing a Case Work Summary Report and requiring partners to submit it quarterly. Legal aid lawyers must provide a summary of steps taken and planned for every case. However, Final Evaluation: Component One, Access to Justice Program | 2012 21 the program allowed VSS to be exempted from this mandatory reporting because the carriage of GBV cases rests with OPG prosecution. The program delivered 4 days training for partners onsite to introduce the reporting. Since then, partners have submitted reports that have revealed weaknesses in case strategy, client engagement, case documentation, and gender and juvenile policies. Consequently, the program provided them with recommendations to improve their case management system. Internally, the program finalized guidelines for its staff to monitor the new systems. The mid-term evaluation noted that the guidelines for ATJ staff helped the program to focus and maintain the implementation of case management with the partners. The LBHs informed the evaluators that the program’s case management review and support improved their case management and created a case database, allowed their entries and records to be monitored for advice on advancing their use of the system. One partner also mentioned to the evaluators that ATJ’s approach of regular monitoring with accompaniment by a consultant encouraged them to practice case management skills within their organizations. Therefore, the activity is relevant, and it seems to be effective in fostering the willingness of partners to use the system and in responding to feedback. Nonetheless, there have been issues with the accuracy and detail of reporting, and concerns regarding legal analysis and strategy. Once more, this capacity building is ongoing in nature because of the partners’ weak familiarity with case management. With the culmination of the program, it is highly likely that their use of the system will not improve or they will stop using it. Despite this, the Justice Facility has introduced an Integrated Case Management System linking police, prosecutor, public defenders and courts using one number per case. We do not know if the program’s case management system interfaces with it, but the experience of using it might have prepared partners to use the government system more effectively when it comes online across the board. Activity 1.3 Administrative, management and financial training for legal aid partners Administrative, management and financial training forms part of the program to enhance and strengthen the internal management of partners to support legal services. The program ran financial management training for partners covering the Foundation’s grant management process, administration/human resources policies, and financial management systems, and developed a financial manual based on the training. The program also held a management retreat for partner directors and program coordinators to identify organizational problems and solutions for them; understand why good leadership/management is important; identify manager roles; provide examples of leadership styles; provide workplace conflict of interest training, etc. Participants responded enthusiastically to this training and requested more in future. In May 2012 the program facilitated six legal aid lawyers and two paralegals to participate in study exchange with LBH Jakarta and LBH APIK to learn case and client management, as part of general operational management training from the LBHs based in Jakarta, Indonesia. The partners interviewed (in particular, VSS) point to the importance of financial management training for their staff/operations as well as visits to encourage them to practice it within their organizations. Given the generally weak organizational capacity of Final Evaluation: Component One, Access to Justice Program | 2012 22 the partner organizations, such training was crucial. Without more information, it is difficult for the evaluators to draw conclusions here, other than to refer to FECM and FFSO financial mismanagement issues that marred the final year of the program, which suggest financial management was still poor. The evaluators have learned that administration and management within the partner organizations also relies on the internal leadership of each organization to encourage staff to follow and respect the internal management and financial policies. Staff turnover within the organizations as well as differing levels of skills, experience and confidence may also affect how the organization implements its management and financial policies and mechanisms (as experienced by FFSO and LBH Liberta with the absence of the Director to join the CFJ training course). The evaluators also note the importance of clear responsibilities and integrated M&E between program implementation and financial matters within the project as raised by Foundation’s staff. Activity 1.4 Assessing the impact of legal aid services 1.4.1 Monitoring and evaluation training The program has conducted a review of internal monitoring systems in order to produce a proper and accurate assessment of the impact of legal aid services. The program also sent the Senior Program Officer and Program Officer to attend monitoring and evaluation training. ATJ also conducted M&E training for partners, which was attended by LRU, VSS, LBH Liberta, FECM and FFSO. The training covered the program design and the link between M&E and program design and emphasized the importance of doing M&E. The mid-term evaluation of the impact of legal services covered litigation and mediation and took into account an assessment of client satisfaction towards legal services provided by each LBH. ATJ and the partner organizations were aware of the need for the partners to enhance their M&E mechanisms and skills in order to have quality control over their work. The evaluators found the training to be very important and relevant in evaluating and improving the performance of the partner organizations. By using M&E skills they are also capable of reflecting on the impact of the legal services they provide. FECM, VSS and LBH Liberta discovered that the mobile clinic has been useful for poor people who can save money on transport costs whilst the community gains legal knowledge through community legal education. However, the evaluators also note the concerns raised by partners that they have difficulty institutionalizing and internalizing the knowledge and skills from M&E training as part of organizational management strengthening. Even though ATJ provides feedback to the partners through their 3-month, 6-month and annual report, this did not seem to improve the partners’ analytical skills and critical thinking about the activities. For this reason, the evaluators believe it is vital for ATJ Program Officers to challenge the partners to show their analytical skills and capacity for self-reflection in their reports (see Activity 1.7.3) in order to establish strong M&E within their organizations. 1.4.2 Client survey and mid-term evaluation ATJ produced two client surveys (2011 and 2012) and whilst these client surveys worked well, the program acknowledged difficulties in focusing on the quality of legal services. The surveys were conducted independently of the legal aid organizations. The client Final Evaluation: Component One, Access to Justice Program | 2012 23 surveys represent an effective M&E component of the program, aimed to provide space for clients to express their feedback to legal aid partners and to encourage the partners to respond to client feedback. The evaluators note that both surveys were also useful in evaluating how the presence of legal aid organizations could enhance legal knowledge within communities and encourage communities to deal with the formal justice sector. The surveys also looked at whether clients had consulted local authorities before bringing their case to the LBHs. The surveys also consistently, sensitively and specifically gathered the feedback of women clients, including looking at the quality of male legal aid lawyers handling women’s cases. The evaluators find the satisfaction of clients with legal aid services remains high (89 percent) and note the importance for clients to have skilled legal aid lawyers in their areas. High satisfaction of clients with legal aid services provided by legal aid organizations could strengthen the presence of legal aid organizations in the community, the Government as well as formal justice institutions and relevant stakeholders. While the clients still predominantly rely on media (34 percent) and community leaders (23 percent) to access legal information, the 2012 survey found that more clients are listing LBHs as a primary source of information (13.7 percent). The evaluators note this finding could improve the CLE activities with better strategy, activity plans and design, target groups as well as the tools. In relation to the sustainability of legal aid organizations after the conclusion of this program, it is quiet worrying to find in the surveys that some LBH staff requested fees from clients. Even though the LBHs responded seriously to the claims, compromised professional ethics may occur again in the future, particularly if funding for LBHs is not secured. Last year an external evaluator conducted mid-term evaluation of Component One of the ATJ program. The evaluation concluded that the program has been addressing unmet community access to justice needs through legal information and greater access to the formal justice system and local justice processes. It also noted that the ATJ program and its partners have made an important contribution to community to access to justice, particularly in responding to the legal needs of women who have been effectively targeted to receive legal assistance by this program. The program has been responsive to a rapidly changing legal aid sector and the changing needs of legal aid lawyers, through capacity enhancement based on consultancy and assessment of the partners’ knowledge, skills and confidence to provide better quality service to their clients. The mid-term evaluation recommended that the program focus on and take significant steps to strengthen the institutional capacity of the legal aid partners, particularly in relation to the sustainability of the organizations. With regard to mid-term evaluation, the evaluators are aware that the consultant was familiar with both the program and the environment. This was of great use to the consultant in evaluating the program with sufficient knowledge and detailed information in order to provide in-depth analysis and recommendations. The evaluators also find the consultant had sufficient time to reflect, find critical factors and activities to be improved. The evaluation was effective in identifying issues within the program and providing constructive feedback to be considered and implemented by ATJ in its final year, for instance suggestions for institutional capacity building and calling on the program to prepare an exit strategy such as providing assistance for partners on proposal writing. 1.4.3 Women’s legal needs research Final Evaluation: Component One, Access to Justice Program | 2012 24 The idea was to develop materials focused on good practice in mediation and litigation that could be used for training. ATJ initially attempted to do this with JSMP but when it fell through finally, it was replaced by the women’s legal needs research. The research recorded 24 women who have experienced with domestic violence and analyzed the support mechanisms for victims’ legal rights, effectively gathering the victims’ perspective of these mechanisms based on their experience. The research described the different attitudes of PNTL, the Office of the Prosecutor General and the court in responding to victims’ reports through the formal justice system. It found the changes in the attitudes and perspectives of local traditional leaders since the implementation of the Law Against Domestic Violence. However, there were other challenges that the victims faced, for instance the reluctance of the PNTL to progress cases further, different responses within the PNTL institution, and the lack of priority given to domestic violence cases by prosecutors. The research also noted the challenges that continue to be faced by women victims in rural areas (socially, economically and culturally) whilst they were seeking outcomes through the formal justice system. The Asia Foundation expects the report will be useful background when AusAID funding for GBV begins in October. The evaluators found the report complements other research undertaken previously by different organizations. It is also distinct from other publications in the sense that it reflects upon and analyzes the specific experiences of domestic violence victims seeking safety and justice. The research demonstrates how ATJ (and similar programs run by different organizations) has contributed to changes in attitudes and perspectives within communities, from local traditional and formal leaders, formal justice actors/institutions, to women victims of domestic violence and analyzes how social, economic and cultural factors affect these changes. The effectiveness of the research is still in question as it was completed at the end of the program and the evaluators are unsure how the report will be of use to partners. However the evaluators consider it to be a strong example for legal aid partners who may wish to record/write/document and conduct small research as they have great experience, knowledge and information in dealing with domestic violence cases and addressing the many challenges. Activity 1.5 Legal aid coordination body The program helped facilitate the building of an informal legal aid organization network with the legal aid partners and supported the partners to coordinate among others and contribute to policy development through the proposed legal aid legal framework, the draft Access to the Courts Law, and the Private Lawyers Law. ATJ and its partners also shared experiences with Indonesian legal aid organizations, built partnerships with AATL, the government and other stakeholders, facilitated a Tetum translation for the draft Access to the Courts Law and engaged in legal drafting, public consultation, civic education, radio talk shows and lobbying with the Ministry of Justice and other relevant stakeholders. Discussions between the CFJ management, the directors of the legal aid organizations and ATJ scholarship recipients on how the trainee lawyers were progressing and their professional plans following completion of their training. The program also called for broader attention from other stakeholders such as the Acting UN Special Representative of the Secretary-General (SRSG), who had raised the lack of funding for the legal aid organizations with the former President of the Republic and Prime Minister. In relation to the draft Access to the Courts Law finally the MOJ revised the draft law and allowed for a delay in a planned consultation to ensure greater Final Evaluation: Component One, Access to Justice Program | 2012 25 participation by key stakeholders. ATJ noted that the new draft of the Access to the Courts law contains some positive points in that it acknowledges the important role of private lawyers including legal aid lawyers to supplement the public defenders, and provides the opportunity for private lawyers to access state funding for legal representation or legal aid. This means there is no state funding available for legal advice/consultation. The draft remains with the Ministry of Justice. The evaluators find the efforts of ATJ in encouraging and facilitating the partners to build partnerships with AATL and other stakeholders in the justice sector relevant in order to pursue policy changes within the Ministry of Justice which affect legal aid organizations and private lawyers. The evaluators recognize the effectiveness of linking legal aid organizations with the AATL in order to gain stronger support and to enable their voice to be heard by the Ministry of Justice and other formal judicial institutions. The success in achieving an extension of the transition period for private lawyers to practice under the Private Lawyers Law should also be appreciated. The evaluators note that ATJ contributed in facilitating legal policy advocacy and broader networking with high-level institutions, including the socialization of the LBH’s roles. The program also demonstrated concern about future funding sources for legal aid organizations due to a decrease in funding for the legal aid sector and the Legal Aid Bill. While the Access to the Courts Law remains a draft, the evaluators conclude that ATJ and its partners can continue building partnerships with AATL and other civil society organizations (CSOs) such as the national NGO Forum (FONGTIL) Rede Feto, and human rights NGOs to strengthen their approach to policy advocacy with regard to the bill. FFSO, FECM, JSMP, VSS and LBH Liberta already have advocacy skills and experience including organizing campaigns and lobbying. However, they still need strong and strategic support to be confident in running policy advocacy activities. Activity 1.6 Public policy development of legal aid bill See Activity 1.5 Legal aid coordination body. Activity 1.7 Legal aid for women The following four activities are additional activities which fulfilled the program requirement to have 30 percent women clients for the legal aid partners and to improve the quality of legal services to women clients. 1.7.1 Pilot study on DNA test for paternity suits The program learned of a high number of paternity disputes across the country and this pilot study aimed to address some of the issues surrounding paternity cases. There is no legal basis to mandate DNA testing in paternity cases and as such, any DNA test should be taken voluntarily by both parties. The pilot study received support from the Northern Territory Police Forensic Unit and the Victorian Institute of Forensic Medicine, Australia, to assist the court with DNA evidence in paternity cases. In 2011-12 the pilot study provided DNA analysis in seven paternity cases, with two positive and five negative results, and one case brought to court. The evaluators note the pilot test as relevant in responding to limitations of formal judicial actors in addressing high number of paternity cases. The evaluators consulted with the Final Evaluation: Component One, Access to Justice Program | 2012 26 Prosecutor General, AATL, the legal aid partners, the Ministry of Health (MOH), and others. Partners told the evaluators that the pilot study needed to consider social issues surrounding the clients. For instance, trust between the client and the DNA taker must to be established before undertaking a DNA test. The evaluators believe the study could have been more effective by organizing deeper assessment and broader consultation on relevant social issues and the legal framework in relation to DNA test include further possibility of the test with prosecutors and judges who directly deal with paternity cases, women NGOs, private lawyers. The evaluators doubt the sustainability of DNA testing beyond the pilot study if ATJ does not continue to lead it. The evaluators did not find interest from the Ministry of Justice, Office of Prosecutor General or the court in taking over from the pilot test. 1.7.2 Identify and publicize best practices See Activity 1.4.3 on women’s legal needs research. 1.7.3 Improved data collection of partners’ cases In the second period of 2009-10, ATJ began to work with the partners to improve data collection in order to have better legal needs analysis, included the legal needs and type of legal issues requested by women clients. During this period the most common type of case was that of domestic violence. In the following year, the program continued to support the partners in collecting client data. ATJ analyzed the monthly statistical reports from partners and requested clarification when data was not clear. This improved data accuracy and helped the program monitor legal aid services and support partners with legal analysis of cases. See also the Activity 1.2 on case management. The evaluators find the activity relevant in improving data collection skills and data collection. It is effective insofar as the evaluators find the partners have improved their data collection and ATJ has been sharing data analysis and sex disaggregation skills with the partners. The evaluators also note the program’s approach of providing regular mentoring by a consultant works well in implementing the lessons learned from the visit to LBH APIK in Jakarta. VSS data collection and analysis with support from the Justice Facility could be improved in future and sustainable in future. The question is sustainability of other partners whether they would continue it or not. 1.7.4 Support for women’s legal aid organization The program supported an initiative to fund the establishment of the country’s first women’s legal aid organization. The program facilitated monthly meetings of women lawyers, assessed legal services for women and provided a mentor to work with VSS. The program also facilitated nine women (eight of them lawyers), two male lawyers and one female ATJ Program Officer to participate in study exchange and training with LBH APIK in Jakarta focusing on management and legal aid. The activity successfully supported women lawyers who had for three years been struggling to establish a women’s legal aid organization, particularly as the former AATL President was opposed to the idea. Currently JSMP and VSS have primary responsibility for the establishment of the women’s legal aid organization, while other women lawyers will continue to be consulted. Final Evaluation: Component One, Access to Justice Program | 2012 27 The evaluators conclude the program’s initiative to support and encourage women’s lawyers in the establishment of a women’s legal aid organization is relevant as the legal aid sector needs strategic goals and a focused strategy to address the high number of GBV cases. ATJ’s support in facilitating meetings, discussions and networking among women lawyers was also effective. By establishing a specific women’s legal aid organization, there is opportunity to concentrate strongly on GBV issues and state policy related to domestic violence. With the high degree of experience of VSS, it could be sustainable to continue the current activities, for instance community legal education, legal assistance and partnership with the PNTL at the district and sub-district level, judge, public defender and Office of Prosecutor General as well as referral networks and other legal aid organizations to develop the independent women’s legal aid organization. Activity 1.8 Form networks and support groups for women lawyers PMEP described three indicators for Activity 1.8 that includes facilitating 12 women lawyers meetings by the end of program, the number of women lawyers and paralegals being 35 percent working in the legal aid organisations, and organizing six specialized trainings or exchanges on two issues. There are two areas of support included under this activity. 1.8.1 Women lawyers network meetings The program organized the first women lawyers meeting in September 2009 and in cooperation with AATL, conducted a follow up meeting of the women lawyers which identified five AATL committees in which they could actively participate, namely the Women’s Committee, Access to Justice, Draft Legislation, Publications, and Training Committees. The Women’s Committee discussed an activity plan and agreed to hold meetings on a quarterly basis. ATJ encouraged the women lawyers to run regular meetings, prepare time schedules and agendas, and to chair the meetings. Meetings touched on various topics, such as the CFJ training for private lawyers, experiences of CFJ training and the Portuguese language barrier, the establishment of a women’s legal aid organization, etc. The meetings provided an opportunity for women lawyers to build a network and exchange ideas among the invitees including the Justice Facility, UNMIT’s Administration of Justice Support Unit, United Nations Population Fund (UNFPA), SEPI, and women jurists. The evaluators conclude the program has been supportive in helping the partners, especially women lawyers, strengthen their network and implement the idea to establishing a women’s legal aid organization. See Activity 1.7.4 for more on the establishment of a women’s legal aid organization. Building a network of women lawyers is relevant as it has allowed the exploration of ideas in support of women’s legal aid and other related issues. The evaluators find various meetings such as the Christmas Lunch held for women jurists are a creative way to encourage further discussion and to gain input from those outside the legal aid organisations. Legal knowledge and skills, good management and networking that the Foundation has supported through VSS and other partners should support the sustainability of VSS and the future independent women’s and children’s legal aid organisation. 1.8.2 Provide specialized trainings and support change programs Final Evaluation: Component One, Access to Justice Program | 2012 28 In the final three-years of the program, ATJ organized two exchange visits to Indonesia involving mostly women lawyers, paralegals, male lawyers and staff from the Foundation. The first exchange visit and training was for 13 women lawyers and paralegals who travelled to Indonesia with support from USAID and Justice Facility to be trained by two legal aid lawyers from LBH-APIK, the most prominent Indonesian women’s legal aid organisation. The trainings covered two main topics, firstly service provision to women clients and secondly, on establishing and sustaining legal aid organizations. The program also facilitated another exchange visit of nine women, eight of them lawyers, as well as two male lawyers and an ATJ Program Officer who participated in training with LBH Jakarta focusing on management and legal aid. The Foundation noted the idea for establishing an independent women’s legal aid organization was proposed during the first visit to LBH Apik. It also increased participants’ knowledge and understanding of planning, management and the work of legal aid. The partners acknowledged that they had learned from LBH Apik as a model for a women’s legal aid organization with good institutional management. The evaluators conclude the two visits undertaken as part of this activity to be relevant. The Foundation approach whereby they discussed planning of the visits and objectives with women lawyers had increased the ownership, responsibility of partners and effectiveness of the partners to undertake follow up after the visits. The evaluators find the exchange and training visits to be effective as partners tried to implement lessons learned from the visits, but also because the LBHs increased their awareness of planning and organizational management. However, the evaluators note that organizational management and human resource issues remain challenges for most partners hoping to be sustainable in the long-term. Objective 2: To expand the reach of legal aid services Where Objective 1 strengthens organizational capacity development, Objective 2 facilitates the wider delivery of legal aid services and increased legal education by funding the mobile legal aid services of FECM, FFSO, and LBH Liberta (Activities 2.1 and 2.2),9 introducing a paralegal program for FECM and FFSO (Activity 2.4), and supporting the provision of specialized services for typically female GBV victims (Activity 2.5). Activity 2.3 – that is, linking government and LBHs, was not implemented directly, but it can be seen in advocacy on the draft Access to the Courts Bill LBH and CFJ dialogue regarding the Private Lawyers Act, and UNTL/LBH internships. The performance of LBHs was affected by a variety of factors, some of which were beyond the control of the program: the courts often altered their hearing schedules and public defenders were frequently absent; staff resignation and illness/maternity leave affected program delivery (management) and service quality (legal), most notably with the change in LBH Liberta and FFSO directors; weak knowledge of criminal law especially GBV and mediation purpose resulted in crimes still being mediated; and, in the final year, different financial mismanagement incidents led to FECM and FFSO stopping work. 9 The program funded LBH Covalima and LBH URA in the previous extension but it stopped doing so early in this extension due to financial mismanagement. The evaluators tried to interview their staff living in Dili so that it could evaluate their service delivery but time did not permit it. Consequently, we have omitted them. Final Evaluation: Component One, Access to Justice Program | 2012 29 Activity 2.1 Mobile legal aid services The program funds three legal aid organizations or LBHs to deliver legal aid services: Fundacao Edukasaun Comunidade Matebian (FECM), Fundacao Fatu Sinai Oecusse (FFSO) and LBH Liberta. It replaced LBH Ukun Rasik An (URA) with LBH Liberta, and ended its funding of LBH Covalima, early in 2009-10 and at the end of it respectively. These LBHs provide free litigation and mediation services. Most recently, the program employs 26 legal aid lawyers, including VSS. At this point in time, none of them are women. All of them studied in Indonesia or Timor-Leste, and in Timor-Leste they only studied at private universities. There are 11 former or current partner legal aid lawyers in the CFJ course, two of whom have finished their 15 months of theoretical training. There are also two paralegal coordinators and 26 paralegals in FECM and FFSO. Their clients retain them with referrals from their community, paralegals, police, courts and NGOs, mobile team visits to sub-districts and the villages, including their community legal education activities. The program approves the partners’ monthly work plans, which it shares with USAID. During 2009-10, the program improved its database to disaggregate information on cases by process, status, category and gender, and visited partners to install the database program and provide training. The three partners have conducted 204 mobile clinics in their court jurisdictions respectively. FECM has conducted 94 mobile clinics, FFSO has conducted 66 mobile clinics and LBH Liberta (began in April 2010) has conducted 44 mobile clinics and 37 community legal education activities in different villages and high schools. Through mobile clinics, the partners proactively collected new cases (mostly paternity cases), updated clients on pending cases, provided legal education and informed communities of their role with relation to access to justice. 2.1.1 FECM In 2009-10, FECM collected 144 criminal and 98 civil cases (242) of which it handled 152 by litigation and 90 by mediation, and represented 62 female clients. It achieved its target for new cases (192) and resolved 45 cases in this period, but was below its target for female clients (30percent). It conducted 30 mobile team visits. In 2010-11, FECM took on 321 new cases of which 220 were criminal and 101 were civil, so it exceeded its target of 192 new cases. It represented 74 female clients which meant it did not meet its target of 30percent female clients. FECM conducted 23 mobile team visits in 2010-11. In the first period of 2011-12, FECM received 133 new clients, 38 of which were women. Thus, FECM still has not reached the 30percent threshold. 88 of these cases were criminal matters and 45 of them were civil matters. It completed 21 mobile team visits in 2011-12. The evaluators find that FECM has achieved its targets for new cases but consistently failed to reach its goal of 30percent women clients. ATJ considers one of the reasons for this to be that two of FECM’s experienced women lawyers have joined the CFJ training. FECM staff believe that without new funding for its legal aid services, there will be more unresolved civil disputes, which may lead to an increase in crime. There will also be less protection against state human rights abuses in the criminal system. 2.1.2 FFSO Final Evaluation: Component One, Access to Justice Program | 2012 30 In 2009-10, FFSO collected 48 criminal cases and 70 civil cases of which it handled 32by litigation and 86 by mediation, 32 in which it represented female clients. It failed to achieve its target for new cases in that year of the program but was above its target for female clients (30 percent). The change of directors at FFSO during this period had a significant impact on its performance, and ATJ started to monitor it closely. During 2010- 11, FFSO took on 122 new cases of which 16 were criminal and 106 were civil, so it again fell short of its target of 144 new cases. Sixty-one clients were women, and it met its target of 30 percent female clients. FFSO conducted 24 mobile team visits that year. While there was an improvement in the number of referrals in the first half of this year (2011-12), all of the 66 new cases were handled and resolved by mediation, and 13 of them were criminal matters. FFSO argued that the victims/clients have requested mediations. The program advised FFSO to stop mediating criminal matters and assisted it to increase handling by litigation. Three criminal cases were then handled by litigation. With 32 women clients, FFSO again met its target of 30 percent. FFSO conducted 24 mobile team visits. ATJ required FFSO to ensure at least 30 percent of new cases were handled by litigation, and it appointed a Law Program Officer to monitor FFSO. For the first period of 2011-12, ATJ reports that it suspended funding to FFSO due to serious allegations of financial mismanagement. Nonetheless, FFSO received 22 new clients, 10 of which were women but did not reach its 60 new case target. Twenty-one of these cases were criminal matters and one of them was a civil matter. It completed 12 mobile team visits. We conclude that FFSO was only partly effective as it did not take on sufficient cases and lacks confidence to determine when to litigate matters demanding immediate advocacy or to engage in frequently unresolved mediations. We find ATJ’s explanation that the loss of FFSO’s senior director, and his replacement with a less experienced person having an impact on case collection and litigation, is plausible. We visited FECM and FFSO and noticed a clear difference in leadership capacity, although we found that their legal aid lawyers were similarly confident in our meetings. Further, FFSO experienced ongoing issues with mediating criminal cases, which we observe the program did its best to correct. One ATJ program officer said: “FFSO wasn’t really effective in mediating because they tried to solve all cases via mediation, even criminal cases”, by which he meant that FFSO did not learn when to use mediation and litigation legally and strategically. However, like FECM legal aid lawyers, FFSO legal aid lawyers understood that legal aid guarantees stability and legal awareness, and deters unlawfulness and crime. As one FFSO staff member stated poetically “the people sleep in darkness about access to justice”. 2.1.3 LBH Liberta LBH Liberta joined the program in April 2010 to replace LBH URA. The organization is based in Dili and covers the Dili court jurisdiction in Dili, Liquica, Aileu and Ermera on mediation and litigation. While LBH Liberta just joined the program in second period of 2009-10, it received 57 cases, 28 criminal cases and 29 civil cases. Thirty cases were settled through litigation and 27 through mediation. Thirty clients were women pursuing a paternity case. This is only slightly less than the organization’s target for new cases (60). LBH Liberta also conducted 20 mobile team visits. In 2010-11, LBH Liberta collected 148 cases, 103 criminal cases and 45 civil cases, and ran 22 mobile team visits in different areas of Dili court’s jurisdiction. In the first period of 2011-12, LBH Liberta took on 54 Final Evaluation: Component One, Access to Justice Program | 2012 31 cases, 43 criminal cases and 11 civil cases. In 2010-11, LBH Liberta visited Becora prison 17 times and made five visits to Gleno Prison to ensure prisoners were receiving legal representation. It also visited district PNTL stations in Liquica, Aileu and Ermera and 46 different clients to share information related to their cases. In 2011-12, ATJ reported that LBH Liberta had visited Becora prison seven times and Gleno prison on five occasions. LBH Liberta also built relationships with other organizations such as local authorities where they would run mobile clinics, PNTL, prison officials, AATL, JSMP, the Justice and Peace Commission, Fokupers and LAIFET in Dili capital. The UNMIT Human Rights and Transitional Justice Unit (HRTJS) told the evaluators that they are impressed with LBH Liberta for being pro-active in offering legal services to prisoners at PNTL detention centers. According to the Unit, this activity has never been conducted by other institutions. The evaluators conclude that for the two periods of semi-annual reports, LBH Liberta showed they had exceeded taking on 30 percent female clients but had failed to achieve the target number of cases (60). This may be due to the departure of the Director, which affected the effectiveness and internal operations of LBH Liberta. However, the idea of legal aid organizations visiting prisons is relevant, effective and could be sustainable as there are no other human rights NGOs doing this work in partnership with the PNTL, prison officers and local authorities. 2.1.4 Roundtable meeting with legal aid partners The program conducted monthly Roundtable Meetings (RTMs) with partners to upgrade their legal skills and sector knowledge, and to increase the effectiveness of their legal aid services. In the three-year program RTMs discussed activities related to legal aid knowledge and skills training, institutional administrative and management trainings, policy advocacy for two draft laws and networking, the Stanford Law School and paralegal program, support to women legal aid lawyers in the establishment of a women’s LBH, as well as surveys and the mid-term evaluation. The evaluators find that RTMs were a valuable platform for LBH learning, planning and sharing. Particularly, they allowed the program to encourage its partners to advocate jointly for the transition period in the Private Lawyers Act for private lawyers to stop practicing and start training to be extended, to promote the training scholarship and agree on its terms, and to commence discussion regarding sustainability after the program. One of the LBH interviews underscored the importance of the RTMs for them, by asserting that ATJ wanted to stop RTMs but that they themselves had insisted they continue. While we do not know if the program’s partners will continue to organize collectively and secure new funding, the impact of working together has been a positive one. Activity 2.2 Contact group meetings/legal aid clinics See Activity 4.1. Activity 2.3 Links with government and organizations providing legal aid Final Evaluation: Component One, Access to Justice Program | 2012 32 See Partnerships. Activity 2.4 Paralegal program In addition to furthering Objective 2, the paralegal program has five purposes: to expand access to legal aid services in Oecusse and Baucau, Manatuto, Viqueque and Los Palos, including in remote locations; to increase outreach by having paralegals at the suco level; to identify clients, refer clients to LBHs, and assist in contacting and updating clients; to share information with community leaders and members regarding judicial processes and the law; to assist local authorities in resolving civil cases through mediation in accordance with principles of neutrality and non-discrimination; and, to increase the number of civil matters resolved at village level through mediation. The program chose FECM and FFSO to pilot this activity for the extension. In 2009-10 the program hired a national paralegal officer and international paralegal advisor to focus on paralegals. The program completed its assessment and in cooperation with USAID developed an action plan that the program implemented in coordination and cooperation with FECM and FFSO. The action plan results in FECM resuming house visits, both LBHs using an updated case reporting form and a case identification tracking system, a new policy to ensuring FFSO tracks referrals to legal aid lawyers of PNTL, and improved monitoring through unannounced paralegal coordinator visits, a revised monitoring report format, and improved transport. In 2010-11, paralegals continued to mediate domestic violence allegations and the program continued to work with them to ensure that all serious crimes were referred to the PNTL, and cases mediated through the informal system produced just outcomes. Moreover, the program sought to improve the way in which paralegals handled female clients, including training, awareness-raising, increased monitoring, further research, and cross-cutting efforts, and coordination with VSS and domestic violence referral networks. ATJ stated that paralegal activity is a positive development for communities in order to address a shortage of legal aid lawyers within communities, to increase coverage, bridge and facilitate interaction between communities and the court. It may also prevent further crime within a community. Paralegals understand community disputes, collect cases and refer criminal cases to LBH’s legal aid lawyers or the PNTL. Paralegals facilitate mediation on civil cases. The program advised that paralegals had also been facilitating mediation in domestic violence cases because according to the paralegals, the victims did not want to bring charges and wanted to settle their dispute through mediation. In some cases even when the victim had brought the case to PNTL, the PNTL would return it to village level. ATJ has produced tools for managing paralegals such as monitoring, handbook and case filing system. The client survey in 2012 found an improvement in community awareness and understanding of the role of paralegals. The 2011 survey found that only 24 percent of clients (FECM and FFSO) understood the role of paralegals, such as facilitating mediation processes in minor cases. Seventy-six percent of clients said they had never heard of a paralegal. For those who experienced the assistance of a paralegal, the majority of them were satisfied with the service. In the 2012 client survey, 34 percent of clients said they understood what a paralegal did and 19.7 percent said they did not know. In relation to importance to have paralegals, the 2011 client survey reported that 46 percent of clients believed the presence of paralegals in the community to be very Final Evaluation: Component One, Access to Justice Program | 2012 33 important, over 12 percent said that paralegals are important and 42 percent clients did not respond the question. According to legal aid clients, the presence of paralegals would be very beneficial to show pathways to justice, socialize laws, and enhance people’s knowledge about laws and the legal process. In comparison, the 2012 client survey found that 50.8 percent of a total 62 respondents believed the presence of paralegals to be very important or somewhat important, 28.7 percent did not answer the question and 20 percent said they did not know. The evaluators found there has been progress in communities’ knowledge about the presence of paralegals in the community and their roles. It demonstrates how FECM and FFSO’s paralegals have made a valuable contribution in communities where they operate. On the basis of the interviews, the evaluators conclude that paralegals were a crucial link between remote areas and formal justice, they won the trust of community leaders and members by living in the communities and providing new knowledge of law and procedure, and they inserted structures and standards into village mediation. Unlike legal aid lawyers that depend on salaries, paralegals - who receive a much smaller salary - are likely to continue to perform their role now that they have knowledge, skill and experience, and they are recognized by their communities and in demand for contributing to dispute resolution. We believe that, if this is the case, it will continue to strengthen the rule of law and prevent conflict escalating for some time, but to sustain the paralegals’ effectiveness, we expect that they will require further training, especially on the new civil code, and more monitoring to ensure that mediation principles and processes are embedded. Moreover, without institutional linkage to the courts through LBHs, communities are likely to revert to traditional law or modified processes that might mediate crime and involve bias and discrimination. The Ministry of Justice is also drafting an ADR bill that proposes to certify paralegals to practice and receive salary to support the State judicial system, while the OPD has a different view of the role of paralegals which deals with administrative activity and includes legal socialization. 2.4.1 FECM From October 2009 to September 2010, FECM paralegals received 517 new cases. There were 249 criminal matters and 268 civil matters. Of the new cases, 153 were female clients. Of these cases, 65 were referred to litigation (that is, to legal aid lawyers) and 452 were mediated. Mediation resolved 197 of them. FECM conducted 1698 house visits and met 4909 community members. In 2010-11, FECM paralegals worked on 423 new cases. The shortfall can be explained by staff resignations and maternity leave. Of these cases, 212 were criminal matters and 211 were civil matters. They had 147 new female clients. Ninety-one cases were referred to litigation, while 332 were mediated, with 177 resolutions through mediation and five resolutions through litigation. Paralegals visited 2070 houses meeting with 4782 people. From October 2011 to March 2012, the paralegals collected 248 cases, which does not meet their goal. Once more, this might be due to staff resignations and maternity leave. There were 68 female clients. FECM fell short of its 30 percent target for women with 27 percent female clients. ATJ is working with the Director of FECM to address this issue. New cases comprised 103 criminal and 145 civil matters. Of them, 33 were referred to litigation and 11 were resolved; 215 were mediated and 96 were resolved through the mediation process. There were 1644 house visits resulting in 5187 meetings. Final Evaluation: Component One, Access to Justice Program | 2012 34 We find that whilst FECM’s collection rate was below target, it was conscientious in referring cases to litigation or mediating for resolution. Like its performance in legal aid, FECM had difficulty meeting the goal of 30 percent women clients. An ATJ staff member noted that the paralegals had poor community level relationships to start with, possibly because of local language differences. He offered that “they listen but they talk too much”. Nonetheless, the role of the paralegals in increasing legal awareness contributes to building the rule of law and preventing conflict. Another program staff member noted that FECM had requested to add four more paralegals to their team, but that its case collection rate dropped afterwards. The program investigated and determined that there was demand for paralegals but the newly appointed coordinator was weak. When the Director refused to instruct the coordinator, who was a family member, ATJ coordinated directly with the paralegals after which their collection rate improved. There were no targets set for monitoring legal awareness/information dissemination. The paralegals we spoke to were keen to keep working, and wished to do so at FECM. 2.4.2 FFSO From October 2009 to September 2010, FFSO paralegals received 325 new cases. There were 46 criminal matters and 279 civil matters. Of the new cases, 149 were female clients. Of these cases, 3 cases were referred to litigation while 322 were mediated and 209 resolved. In one year FFSO conducted 4436 visits and met 8805 community members. In 2010-11, FFSO worked on 316 new cases, 14 were criminal matters and 302 were civil matters. They had 131 new female clients. 303 civil matters were mediated, with 243 resolutions. Two cases were referred to litigation and one was resolved. In one year, paralegals visited 5690 houses meeting with 10212 people. Finally, FFSO hired a new paralegal coordinator. From October 2011 to March 2012, the FFSO paralegals only worked for 3 months due to a problem of financial management. Nonetheless, they took on 82 new cases: two criminal and 80 civil matters, and 33 female clients. Seventy-six civil case mediations were resolved. FFSO conducted 1199 house visits which resulted in 2194 meetings. The evaluators determine that FFSO’s collection rate was near target but that it was reluctant to refer cases to litigation because of its legal aid lawyers’ low confidence. The program was concerned that paralegals were mediating domestic violence cases because victims were refusing the bring charges. An ATJ staff member spoke with FFSO paralegals who argued that the community wanted mediation and if they did not comply, they would lose community trust. The organisation’s paralegal program faces similar issues to FECM regarding impact and sustainability. The establishment of pilot paralegal programs in FECM and FFSO might have mitigated the departure of legal aid lawyers who have left to undertake training at CFJ, with paralegals settling minor civil matters by mediation separately or collectively with local authorities, and referring criminal cases to PNTL. Yet, there is no study if paralegals did replace legal aid lawyers in this way. 2.4.3 Training The program conducted trainings for FECM and FFSO coordinators and paralegals at their offices and jointly in Dili on the Criminal Procedure Code, Civil Procedure Code, mediation skills, communications skills, GBV, and the Domestic Violence Act. The trainings made a major contribution to paralegal capacity and were well-received. Further, the program completed and distributed the paralegal handbook. The program Final Evaluation: Component One, Access to Justice Program | 2012 35 also delivered new trainings for FECM on mediation skills, the criminal code and the paralegal manual, and completed the previous trainings in relation to the Penal and Civil Procedure Code, mediation and communication skills for FFSO, at their offices. The evaluators believe that the paralegal training was relevant and effective because it was based on the program’s assessment to cover law, skills and management training (similar to legal aid lawyers/LBHs) and paralegal education was extremely limited. One problem, noted by one program staff member, was that paralegals had widely different education levels. He thought training could be more interactive to increase learning. The FECM paralegals told us that the training was beneficial and communities liked the paralegal model. They said that the training and tools were beneficial, but they would have liked more training and tools. Activity 2.5 Specialized services for GBV victims (VSS) The program aimed to provide legal assistance to women and children victims of gender￾based violence in Baucau and Dili court district’s jurisdiction through VSS in order to facilitate legal access to communities in remote areas. VSS engaged in legal assistance, mobile visits and maintained contact with the PNTL, local authorities, court actors, and women NGOs, as well as participating in the GBV Referral Networks. VSS conducted regular visits to PNTL Vulnerable Persons Unit (VPU) in four court jurisdictions. In each of the three years, VSS recorded the increased number of GBV cases. In 2009-2010 VSS received and provided legal assistance to 176 women and children victims of GBV (including allegations of domestic violence), of which 12 percent of clients were 17 years old and under. In 2010-11 VSS assisted 395 new clients/cases, both women and 57 children clients. VSS actively worked in policy advocacy with the Ministry of Social Solidarity (MSS), MOH, the Ministry of Education as well as the Ministry of Foreign Affairs (MFA). The evaluators find the activity is relevant and effective, as VSS consistently managed to provide legal assistance to women and children victims on GBV cases, which is a strategic and focused target. VSS created links to CLE, actively participated in referral networks and a working group on GBV, built partnerships with judicial actors in the districts and advocated policy at the national level. VSS liaised effectively with core ministries, SEPI, National Parliament, the President of the Republic, UN Agencies and other NGOs working on women issues. In relation to the sustainability and impact of the activity, see also Activity 1.7.4 on support for a women’s LBH and Activity 1.8.1 on the women lawyers network meetings. Objective 3: To promote skills-based education The third objective of the program sought to promote skills-based education through two activities. Activity 3.1 was a mapping study that aimed to understand legal career requirements, trends, options and choices. In itself, this study is valuable for illuminating these facts and paths. Activity 3.2 was a three-way project between ATJ, SLS and UNTL to prepare and publish legal textbooks. Other stakeholders were involved in planning and reviewing. For USAID, this project was one of the highlights of the program, for building Final Evaluation: Component One, Access to Justice Program | 2012 36 relationships between Timor-Leste and US legal educators, and for its innovativeness in its approach and content. Activity 3.1 Map career paths and practices The program’s career path mapping study was finalized in mid-September 2012. The study is the first of its kind in Timor-Leste. It aims to develop a broad-based map of a nascent Timorese legal profession, outlining its present form, function and key capacity strengths and weaknesses; to inform future policies and reforms to support the development of a robust legal profession that promotes the rule of law and guarantees access to justice for citizens; and to identify career opportunities for law students and training needed to develop appropriate legal skills for the job market. The study interviewed 96 lawyers from the public and private sectors. 38 of the interviewees were women while 58 were men, and 34 were working in the public sector while 62 were working in the private sector. State ministries and government agencies employed those interviewees working in the public sector. Law firms, NGOs and international organizations, and legal aid providers employed those interviewees working in the private sector. The study also conducted focus group discussions (FGDs). The FGDs were in-depth discussions regarding the preliminary results from the study. Three FGDs were held: one with only female law graduates, one with law graduates working in legal aid groups and non-governmental organizations, and one with law graduates working in Baucau and Oecusse districts. The evaluators find the study has been completed and this activity achieved. Further, the study has met its own aims of mapping the profession, and in identifying career opportunities and training needs. Of particular interest to us, having conducted our own desk research and stakeholder interviews for three weeks, were its findings: - there is no complete register of practicing lawyers in the country despite their low numbers - the high numbers of law students and their distribution in the four law schools, - the impact that their yearly graduation will have on competition for places in CFJ training - the high drop-out rate for those applicants who are successful in admission to CFJ training - the need for law schools to ensure their curriculum and teaching meet CFJ requirements - the pressure on law firms to provide legal services of international standard and timeliness - the strategy that state ministries and government agencies seem to be pursuing of internal counsel. However, the evaluators would need to know more to understand its relevance in contributing to the program’s activities on legal education specifically, and legal capacity building, services delivery and information dissemination more generally. Perhaps the study arrived slightly too late to see its full value-add to programming. Similarly, we cannot know yet if it has met its third objective of informing policy, although we believe Final Evaluation: Component One, Access to Justice Program | 2012 37 that it has the potential to do so, especially in relation to the study’s findings that were of particular interest to us. Consequently, the effect of the study is more likely to be felt in the coming years, maybe in relation to curriculum reform and teaching improvement at law schools. There is no doubt that it should be disseminated widely to sustain its information. Activity 3.2 Stanford Law School (SLS) project The Stanford Law School project is a three-way legal education project among the Foundation, SLS and UNTL. For ATJ, the SLS project has five purposes that lay the groundwork and set the parameters for achieving the program’s legal education objectives: to build a foundation for local actors and influence the development of legal education; to provide practical, accessible and dynamic educational materials; to further collaboration between legal educational institutions; to share information with community leaders and members regarding judicial processes and the law; and, to increase the amount of educational materials in Portuguese, Tetum and English. Activities during the extension focused on foundation, materials and collaboration. A legal advisor was recruited in March 2010 to be the point-person for the SLS project between ATJ and SLS. He is a graduate of the school. ATJ signed a Memorandum of Understanding with UNTL in September 2011. 3.2.1 Textbooks In 2009, the SLS project started by identifying a topic for a first textbook with key justice sector stakeholders in Timor-Leste. The President of the Court of Appeal suggested the subject of Professional Responsibility, which the project then decided to take up. SLS established a course to attract students to research and write this textbook, and other future textbooks. Throughout 2009-10, the students drafted the textbook on professional responsibility and it was translated into the three languages. During 2010-11, key stakeholders reviewed the textbook and a Portuguese-trained lawyer was hired to review and edit the book. The then SLS Dean, Professor Kramer, launched the finished textbook in Timor-Leste in November 2011. One thousand hardcopies were distributed and the text was uploaded to the SLS’s project website , which was launched later in January 2012. The program comments: “The text addresses the legislative framework for civil servants, prosecutors and magistrates. The materials are clear, concise and accessible and draw on effective learning tools, such as role playing and hypothetical.” In 2011-12, the SLS project started researching and drafting second and third textbooks on Timor-Leste contract and constitutional law, and a fourth introducing the country’s laws containing chapters on natural resource law, government contracts, family law, inheritance law, legal history, and constitutional rights. Following the process for preparing the first textbook, these three new educational materials have been or are being translated, reviewed and vetted. The evaluators find that the project was effective in preparing four textbooks and publishing one of them to date in the three languages. We have had the opportunity to review the published professional responsibility textbook and the unpublished textbooks covering contracts and constitutional law. The evaluators thought the textbooks were succinctly written and clearly presented, with judicious use of educational tools. In terms Final Evaluation: Component One, Access to Justice Program | 2012 38 of the subjects chosen for the textbooks in their order of preparation, we think that the professional responsibility and constitutional textbooks have immediate utility for legal and political development, but we might have published a textbook on environment, land and natural resources law before one on contracts because these issues are more relevant to economic and societal needs in the short term. In any event, the introduction to Timor-Leste law textbook will address the natural resources law; and, ultimately, quality textbooks will be required for most if not all areas of law. Unfortunately, we do not know enough about the capacity and habits of law students in Timor-Leste to know if they will read the textbooks and learn from them. UNTL considered that the professional responsibility textbook was more relevant to the legal profession than to its law students, and it has not incorporated it into any of its courses so far. Indeed, it is CFJ rather than UNTL that has included it in its training materials for private lawyers. Yet, since this textbook is based on Timor-Leste law, we contend that it is more relevant to law students than UNTL allows. It has been designed both for students and practitioners with limited education and legal capacity. Moreover, the forthcoming textbooks on contracts and constitutional law and, especially, the introduction to Timor-Leste law, will be useful for UNTL and the three private law schools. Therefore, we expect that the impact of the textbooks will be positive if they are inserted into courses and referred to in practice, and their sustainability is high since they are in hard/soft copy and three languages. The evaluators note that the project might have done more - and can still do so of course - to mainstream textbook use in law schools. In addition, the evaluators observe that the SLS project was respectful of the need to include stakeholders in the review process for legal and factual accuracy on the one hand, and to strengthen collaboration among the three partners - as well as the ATJ and SLS relationships with the wider justice sector community - on the other hand. We believe that it is likely to have been successful in strengthening these relationships, and the tests of its success will be the adoption of the textbooks in courses and the commencement of other joint activities such as exchange programs. Lastly, we note that the project has achieved four of its five purposes with the prospect of more effort to come, subject to them obtaining carry-on funding: that is, building a foundation for local actors and influencing the development of legal education, providing educational materials in three languages; and Stanford and UNTL law schools collaborating. What it has not done, although partner legal aid lawyers and paralegals have done so as described above, is to share information with communities. 3.2.2 Foreign visits While the SLS project visited Timor-Leste several times to discuss, research or launch the textbooks, this evaluation will review UNTL visits to Stanford since the PMEP for this activity is “exposure to excellence in legal education for Timorese through establishment of an exchange program” and its indicator is “two students and one academic participating in exchange program to Stanford Law School”. The former UNTL Rector, Professor Cortereal and current UNTL Dean, Professor Tome visited SLS in May 2011, while the current UNTL Rector, Professor Guterres and a UNTL Professor, Professor Carrascalao, visited it earlier in 2012. The agenda for these Final Evaluation: Component One, Access to Justice Program | 2012 39 two senior staff visits was similar: sit-in lectures, guest lecture, meet staff including the Dean, and assist students on the SLS project. We spoke with three of the four visitors. They reported parallel feelings and lessons: they loved the campus and they discovered the ‘Socratic method’ of legal education. The evaluators find that these visits were effective for experiential learning and relationship consolidation. They cultivated or confirmed champions for the SLS project in the UNTL. They opened the door for future UNTL visits, and even exchanges of UNTL lecturers or students. We perceive the positive impact of such interaction in improving legal analysis and education if it were to take place. However, as we noted above regarding M&E, we think that, while the visitors had opportunities and responsibilities at Stanford to learn and share, they did not have complementary ones on their return. We believe that they should have been required to impart their experiences to students to spur student interest in aspiring to visit and build their knowledge of comparative law. But, of course, this is a minor quibble in the wider scheme of consolidating ties for a varied and dynamic partnership in time. 3.2.3 National exchanges In 2010-11, the program facilitated a meeting between the LBHs and the four law schools (UNTL, UNPAZ, the University of Dili, UNDIL, and UNITAL) to discuss an internship program for senior students at the partner organizations. FECM took on three student interns and LBH Liberta took on one, all of whom were from UNDIL. In 2011-12, LBH Liberta hosted three interns from UNDIL and UNPAZ. The evaluators note that this activity has been only partially achieved since a small number of student internships to LBHs have been arranged and then, presumably, undertaken. However, we have no information regarding the tasks of the interns or timing of their internships, or feedback from the LBHs or students on their experiences during them. In principle, the idea of internships is useful for giving the students an experience of practicing legal aid law, strengthening the LBHs’ connections to the law schools, and increasing the credibility of LBHs in the minds of their administrators and lecturers. Consequently, it is relevant for LBH sustainability, directly in the case of recruitment and indirectly in the case of allies for advocacy on the Legal Aid Bill and its funding. Whether or not the internships have had these hoped for impacts is the next question, and that is likely to require careful planning and monitoring to ensure that the desired rewards are reaped. Objective 4: To improve public access to legal education Activity 4.1 started a new legal awareness project with MOJ but dovetailed into supporting LBH legal awareness raising through the combined Activities 2.1 (mobile legal aid services), 2.2 (contact group meetings/legal aid clinics) and 2.4 (the paralegal program). A communications expert reviewed these dissemination activities and prepared a report that is still in draft. The program’s support for JSMP’s Legal Research Unit to provide legal awareness and oversight/enforcement is long-standing. JSMP is the best known and most respected human rights based NGO in Timor-Leste, with a focus on political/civil rights and rule of law. The LRU monitors and reports on court hearings (Court Watch) and researches and publishes thematic legal studies. Final Evaluation: Component One, Access to Justice Program | 2012 40 Activity 4.1 Design and disseminate legal aid messages 4.1.1 Communications expert and MOJ support (Activity 4.1.2) The program recruited an Australian volunteer communication expert 2009-10. The program’s thinking was that she would provide technical support to the Ministry of Justice to design and implement effective communication strategies, support the partners to strengthen their capacity to prepare and deliver high-impact information outreach, and support the Foundation’s National Communication Officer to develop communication strategies for the program. She decided to end her placement early. The evaluators have not evaluated these activities, but we imagine that there might be various issues worth reflecting on as the volunteer terminated her posting prematurely. 4.1.3 Support to partners in conducting legal dissemination activities The legal aid lawyers of the three LBHs - FECM, FFSO and LBH Liberta - conducted community legal education (CLE) activities in addition to their responsibilities of collecting cases and litigating or mediating them. They did so in their home towns and while they were undertaking their mobile legal aid clinics at the suco and aldeia administrative levels. The LBHs employed similar and different models and strategies for legal dissemination, and called similar ones by different names. FECM held justice network meetings that invited justice sector officials to speak with communities, and contact group meetings that trained communities in new laws. It distributed brochures and ran radio talk show programs. FECM was also involved in the establishment of two legal/human rights networks in its district, namely the Baucau Protection Network and Task Force Network. FFSO conducted legal clinics/community meetings that trained communities in new laws and judicial actors, and legal education activities for high school students on paternity. Like FECM, it also ran radio talk show programs. LBH Liberta provided information dissemination/legal education, including for high school students. In 2011-12, the program hired a communications consultant to assess the impact of the partners’ CLE and to run a workshop including important tools - such as developing clear messages, communicating by role-play and drama, and targeting your audience - to assist them further. The mid-term evaluation found that the CLE activities were not being monitored sufficiently. The assessment aimed to suggest improvements to this dissemination and socialization. The consultant observed the partners conducting outreach in the community. The sessions were similar in location, invitees, schedule, and agenda. They were held at the suco meeting hall, and the chefe de suco, chefe de aldeia, traditional leaders, lia nain and community members were invited to attend. The sessions ran for one day, and they covered key laws and the constitution, with questions. The CLE sessions have three main purposes: firstly to inform communities of their legal rights under Timor-Leste’s laws and constitution; and, secondly, to urge community members to use the formal legal system when faced with criminal cases and major civil cases, and, thirdly, to inform communities about legal aid. The most prominent findings and corresponding recommendations from the assessment include: weak strategy and inappropriate methodology for communicating with audiences; poor targeting of vulnerable members – hold separate women’s meetings; blurred messages – pick one topic; legalistic written content – concentrate on the access to justice issues in the new Final Evaluation: Component One, Access to Justice Program | 2012 41 laws; partiality and discrimination – correct prejudice in sessions immediately and plan sessions to fit with local patterns. The evaluators find that the partners were effective in organizing their CLE activities. However, we refer to the consultant’s findings on their strengths and weaknesses for guidance in improving their strategies and methodologies. Further, we believe that legal awareness at the grassroots level is essential for conflict prevention/resolution, access to justice where crimes have occurred or disputes have arisen, and extension of the rule of law by teaching communities what is right and wrong under national law. While the project of teaching people to be lawful and reply to illegality legalistically is a long-term one, the CLEs represent a first step to doing so beyond urban centers. The evaluators reviewed the draft assessment and confirm its excellent substance in analysis, recommendations and tools. However, we found it difficult to read because its style was not industry standard and structure was frequently disjointed. Still, we believe it will prove to be a very relevant and an effective piece of work, not only for the partners but also law schools (and CFJ) too, in tailoring their courses to meet variable student capacity, and further down-the-track when they introduce clinical legal education and even contemplate opening their own legal aid clinics. Therefore, the impact and sustainability of the assessment is likely to be positive and high, providing it is shared widely and followed-up like the handbooks and textbooks. 4.1.4 Promotion of legal aid at universities During 2011-12, ATJ continued to support the partners to have sessions at national universities to present LBHs and their legal aid lawyers’ work in order to encourage law students to consider working in the legal aid sector. JSMP began to develop law student internships. The sharing sessions were conducted at UNPAZ, UNDIL and UNTL. The session in UNPAZ was broadcast live on the university radio station. ATJ also approached and discussed this activity with AATL to work together in the socialization of the legal profession in general. The evaluators find that this activity is a good idea in principle and it can generate support for interning with LBHs and supporting their place in the legal aid regime through the Legal Aid Bill. We did not evaluate it because our interview with UNTL focused on the SLS project and we did not have the opportunity to meet with UNPAZ. Activity 4.2 Court Watch (JSMP, LRU) In addition to furthering Objective 4, the funding of JSMP’s Legal Research Unit (LRU) has four purposes: to monitor the Districts Courts and Court of Appeal for compliance with fair trial standards, the rule of law, and international law; to monitor legislative developments, provide input in draft legislation, and encourage greater civil society input in legislative processes; to disseminate information and analysis regarding the legal system and activities in the justice sector to local and international stakeholders; and to enhance the capacity of national staff members particularly with respect to court monitoring, legal research, analysis and writing, project management, and advocacy. The LRU’s activities to achieve these purposes are: court monitoring in Dili to provide coverage of high profile or critical cases; commenting on structural issues pertinent to Final Evaluation: Component One, Access to Justice Program | 2012 42 the legal system; visiting District Courts and the Court of Appeal monthly to gather case specifics, and observing institutional and legal compliance issues; monitoring parliament to follow debate on legislative issues that are relevant to human rights and the justice sector; making submissions or contributing draft provisions to contribute to legal reform; publishing press releases in bulletin form following district visits, and at appropriate times for Dili proceedings; publishing justice updates at least once per month depending on emerging issues, and one thematic report. JSMP consistently monitored cases at the four district courts across the country. However, it had difficulty accessing hearings at and documents for the Court of Appeal because it operated in closed session. The court was more open by 2011-12. JSMP’s monitoring revealed a prevalence of criminal trials that it believes is because civil matters are more complicated and required more resources, and a paucity of hearings at the Oecusse and Suai district courts because of lack of notice of schedules, absence of judges (Oecusse), and poor transport (ferry to Oecusse and road to Suai). Staff shortages impacted heavily on JSMP’s capacity to monitor cases, but monitoring in Suai and Baucau courts was able to be intensified recently with the recruitment of two staff who are now based in these two districts. On the basis of the information provided in the semi-annual reports, JSMP issued 198 press releases, justice updates, case summaries and reports, and held four press conferences until the final six months of the program. Given the end-of-program target in the PMEP is 300, it does not look like meeting it. JSMP conducted legal analysis of, made legal submissions on, or raised concerns about controversial or topical issues of the day, such as the release of Maternus Bere, draft Juvenile Justice Act, draft Civil Code (Books Three and Four), continuing weak accountability for serious crimes, the Minister of Justice attending her husband’s trial, presidential power to grant pardons, as well as the transparency of the Court of Appeal. According to JSMP, MOJ praised its submission on a draft law on pardons and requested that it draft a law (Committee A of the Parliament is currently doing this). LRU worked closely with JSMP’s Outreach Unit to distribute materials and conduct information sessions at the community level. The communications consultant did not assess these sessions, only the LBHs’ legal dissemination work. LRU recruited two new staff to take on more outreach activities than in the past. At one point, the program was considering how it could assist LRU with training and mentoring in CLE, but there is no record of its follow up. JSMP worked to enhance monitoring of the impact of its work, but the details have not been described in the reports except for evaluating its email distribution list to determine the usefulness of its materials. Early in the program, the program discussed a broader evaluation of LRU activities including its outreach, but this does not appear to have taken place. ATJ provided its analysis of JSMP’s activities in its semi-annual reports. “LRU has continued to play a critical role in monitoring and commenting on the justice system and being a vocal member of civil society.” “The program continues to produce a high volume of work, to have the most accurate information about developments in the judicial sector and to enjoy a reputation of credibility and legitimacy.” Final Evaluation: Component One, Access to Justice Program | 2012 43 “It is clear that JSMP’s reputation as an organization with legal expertise is well established and the organization is respected within the Ministry of the Justice, despite at times being critical of the Ministry.” “Their lobbying and network activities – connecting with international and national NGOs working on justice issues, their focus on maintaining productive working relationships with the Courts and government - demonstrate how they have sought to use the results of their monitoring and research to affect policy change.” We conclude that LRU is effective in preparing and disseminating documents, and interviewees have noted its relevance for and influence on fair trial and the rule of law in Timor-Leste. We have no doubt that JSMP’s monitoring in particular is a crucial service because it provides extensive oversight of the country’s justice system. However, we find it hard to identify clear examples where LRU’s efforts can be linked causally to positive or negative impact, except in the case of the submission on a draft law on pardons, although we concede that we have not had time to review JSMP’s online documents. A program staff member agreed with this finding, saying there is no assessment to prove it has affected change. At the same time, he credited JSMP with ensuring fair trial by monitoring cases and following up with the Court of Appeal, and lobbying for an extension to the Private Lawyers Act’s transition period. Moreover, he said he did not know if JSMP’s outreach is effective because there is not pre-/post-test of knowledge improvement, and many communities do not receive it. In its meeting with us, JSMP remarked that it was the originator of many of the initiatives that the government has adopted to strengthen access to justice in the country. Currently, it is arguing that the government needs to commence supervising the OPD, and LBHs need to be funded since they are an independent actor in the justice sector. It is a pity that the program did not pursue its earlier proposal for a wider assessment. Notably, JSMP explained to us that the program’s monitoring focused on its finances. Given JSMP is a high profile and well-respected NGO, nationally and regionally, we expect that it can secure replacement funding relatively easily and quickly. We will be interested to see what changes it and its new funders make to the monitoring, research and, more recently, outreach roles that ATJ has supported for a significant period of time now. In closing, JSMP complimented The Asia Foundation for their “mutually beneficial and respectful relationship”. And the same program staff member reciprocated, saying the two organizations have a “good relationship with regular communication”. Final Evaluation: Component One, Access to Justice Program | 2012 44 Conclusions Problem The gaps the program seeks to fill still exist. The OPD is still not capable of representing all the legal service needs of the poor, especially in remote areas. The program permitted the partners to reach out to them through legal aid lawyers and paralegals, and linked them to the court system through both. The impact of this has not been measured fully yet, but should not be underestimated in terms of resolving disputes and reducing the cause of more conflict and crime. Moreover, the draft Access to the Courts Bill has not passed. There is no government funding. The Access to Justice program was the primary support of legal aid/representation outside the OPD’s urban-based coverage. Despite concerns in some quarters of the legal profession about their service quality, the legal aid lawyer and paralegal clients appreciated their services. Moreover, there is no resistance to, only promotion of CFJ training because the legal profession and justice community including LBHs recognizes the weaknesses in capacity and wishes to increase service standards. In our view a balance is required between the vision of a successful OPD and the reality of LBHs filling the gaps. Additionally, the government and courts do little to increase legal awareness. Once more, while there might be issues regarding the content of education and skills in educating, the stakeholders we spoke to pointed to the development of the rule of law in communities that have deep-seated traditional laws/values. Indeed, the communities should be recognized for their willingness to change. Again, the effect of expansion of legalism and lawfulness has not been studied recently. It would be interesting to analyze both the thin and thick conceptions of the rule of law. Program The evaluators were impressed with the consistency of programming objectives and approaches (legal aid and organization building/services delivery), which is necessary in light of the lengthy process of institution strengthening especially in the justice sector, and the addition of new objectives and approaches that extended the contribution to access to justice (legal education and problem solving, and paralegals and services delivery). The program achieved the activities it set from the outset or introduced later in the program. Furthermore, it needs to be recalled the program ran for 10 years and the partners have progressed significantly in that time from a very low base. ATJ was the only enduring supporter of LBHs. Other donors dropped out in time. New donors only support specific themes. Therefore, the program played a crucial role over time in increasing legal awareness and aid, which links the government role in legal norms/decisions. Final Evaluation: Component One, Access to Justice Program | 2012 45 Monitoring and evaluation was strong for legal aid lawyer and paralegal litigation and non-litigation training, but not so strong for the other management training types. While the financial mismanagement cases were different, some interviewees thought there were weaknesses in M&E for financial management. The evaluators appreciate their analysis but suggest that the partners might not have fully institutionalized the systems that the program introduced, especially with staff turnover, to ensure that they were fully implemented. The partners have to assume responsibility too. Additionally, the program points out that it was quick to detect and respond, and in both cases its response led to the suspension of remaining programming. Partnerships The program’s partnerships with government and donors were often strong. It appears that it worked particularly well with MOJ on the Legal Aid Bill, CFJ on scholarships/training, UNTL on textbooks/visits, Justice Facility on programming, and UNMIT on policy. Activities The evaluators consider that the initial assessments linked with ongoing monitoring was particularly effective in identifying training needs and categorizing them in terms of substantive law, legal skills and organizational management. We emphasize too, the willingness of the program to design and deliver the new required trainings, or to repeat previous ones for new staff. Additionally, we respect the program’s efforts in introducing a case management system and connecting it to case reporting and data entry, improving both the delivery of services and monitoring of their delivery. The evaluators conclude that the activity design and the objectives to provide wider services delivery of legal aid and legal awareness through mobile clinics, community legal education, paralegals and specialized services for GBV cases, as well as prison visits by LBH Liberta are strategically effective to meet the community needs to have better access to justice and access to legal information. The achievement of each partner to collect cases from the community through mobile clinics, updating clients’ cases and their efforts to provide legal information should be appreciated, although they could be improved. The evaluators find the supports for LBHs to deal with GBV cases and other relevant civil cases by providing legal education trainings, publishing the GBV manual, monthly 30-40 percent of women clients, better case management and data analysis are effectively designed and well-implemented. Progress on the establishment of a women’s LBH is the most significant achievement that should be sustainable and recommendable for further extension. The evaluators acknowledge that, in the case of legal education, much of the emphasis in this extension was placed on relationship building because it was a new component and one that should continue despite the end of the program. However, it would have been useful to ensure that such a high quality textbook was incorporated into UNTL’s curriculum. CFJ has included it in its training. Of course, this can still be done in the coming months. Likewise, we recognize that the impact of JSMP’s monitoring, research and advocacy is well-known in the justice community, but it seems important to us that it Final Evaluation: Component One, Access to Justice Program | 2012 46 be properly evaluated. This might have required the development of new advocacy evaluation tools. Final Evaluation: Component One, Access to Justice Program | 2012 47 Recommendations Program These design and implementation recommendations have future programs in mind:  Planned and written sustainability strategy, especially for lengthy extended programs, ensuring partner participation. Consider narrow and wide conceptions of sustainability, and inside and outside funding sources.  Planned and written advocacy strategy in conjunction with allies/supporters. There are benefits in being more explicit and thorough in preparing a plan: involves mapping of stakeholders and power to strengthen alliances, involve supporters and counter opponents more effectively; permits identifying gaps in arguments, capacities and relationships; requires setting targets and timelines, which then assists with monitoring; encourages more strategic thinking within a framework including resource allocation; provides a hard copy to evaluate.  Monitor throughout program, with adequate resources including dedicated staff so there are no gaps; for example, program staff monitor objectives and activities and grant staff monitor funding and reporting, and the two coordinate to ensure full coverage of these primary elements of the program.  Consider having advocacy accompaniment model for partners to support advocacy components within the program and work with partners specifically on technical advocacy skills. Introduce and develop advocacy tools and include partners as potential accompanier.  Developing partners’ organization profile and financial profile to assess the administrative, management and financial profile of the partners’ organizations and update it regularly, every two years for example. The profiles include the financial and management policy and system, decision-making on finance and general issues as well as the structure of the organizations. Activities These activity recommendations extend the Foundation’s support in this program:  Consider supporting/facilitating LBH short/medium term donor fundraising. Suggested activities include: - training and accompaniment for the LBHs in fundraising skills with an emphasis on proposal writing and introducing various type of grants from different donors. - networking with and introducing LBHs to traditional justice sector donors not funding or present in Timor-Leste presently; ABA, Ford Foundation, OSJI, etc. Final Evaluation: Component One, Access to Justice Program | 2012 48 - networking with and introducing LBHs to possible new justice sector donors, such as international law firms and increasingly Singapore national law firms, corporate foundations/private philanthropists increasingly interested in it, possibly even Australian Attorney General and US Justice Department if they have funding - raising awareness of funding gap through regional and global access to justice networks, like ASEANCats and SEALAW in South East Asia, UNDP in the Asia-Pacific, and ESCR-Net, Haki and Namati in the US - identifying and informing LBHs of justice sector/human rights small grants, most obviously AusAID and EU - possibly, organizing a public appeal in the US through the Foundation’s headquarters if the “legal aid crisis” takes place  Introduce various monitoring and advocacy tools for instance theory of change, most significant change or voice and accountability tools for partners.  Increasing link advocacy program in different areas/level (national, regional) and includes the partners.  Consider encouraging LBHs to facilitate communities to self-fund paralegals and assist developing mechanisms if communities decide to pursue this idea, and monitoring the MOJ’s ADR draft law which proposes state subsidy for paralegals and certification of paralegals.  Continue supporting drafting, passing and implementation of the Legal Aid Bill to guarantee government funding of a hybrid legal aid system with LBHs, including relationship building with AATL and the OPD to secure support, facilitate inclusive consultations, and prepare joint written submissions.  Continue supporting/facilitating AATL and CFJ training design and delivery, including linking with SLS project and LBH services. Share CLE assessment with LBHs, AATL and CFJ, including organizing a Dili-based workshop to discuss drafting community-based legal education principles and approaches.  More broadly, continue supporting the concept of a hybrid legal aid system on the basis that it is needed to ensure access to justice in the first instance until the OPD offers greater coverage and better standards, and to offer choice to clients in the second instance especially for politicized disputes such as land. In this regard, continue client surveys to build body of evidence supporting necessity of LBHs, and to provide incentive for LBHs to maintain their new systems of case management and new principles of mediating civil cases impartially and without discrimination, and referring all criminal matters particularly where related to women and where traditional justice still claims a role.  Continue supporting/funding CFJ scholarships together with other donors, with needs test for receipt and requirement of working for LBHs for 1 year.  Continue facilitating the SLS project, linking it to LBH service delivery and legal dissemination where possible to increase the justice sector synergies. Consider preparing further textbooks especially on law and development issues that have Final Evaluation: Component One, Access to Justice Program | 2012 49 become prevalent and vexing in the region with greater commercialization and corruption, such as planning and investment, environment and land, freedom of association and speech, and impunity.  Support/facilitate university student internships to LBHs and UNTL student exchange to SLS with requirement of completing relevant courses for TL based on career mapping assessment such as specific business law topics for students interested in private sector practice (such as contracts, environment, land, and civil procedure) and particular national/international law subjects for students interested in government or NGO careers (constitution, administration, criminal especially gender based violence, and human rights).  Continue to support VSS and women lawyers in the establishment of a women’s LBH and encourage support from JSMP, AATL, human rights NGOs, women NGOs, Rede Feto and Fongtil/NGO Forum and donors.  Continue supporting JSMP Court Watch case monitoring and report advocacy by encouraging collaboration with LBHs through legal education and dissemination, creating opportunities for influencing government via speaking and networking engagements, and disseminating material to regional and global partners and allies like on new online legal empowerment platforms.  Consider facilitating exchanges and fellowships to regional, Australian and US legal associations and institutions, such as PSHK/LEIP and ILRC in Jakarta, LSS Academy of Law in Singapore, HRLC and ALC in Melbourne.  Discuss with donors the possibility of developing various donorships: INGO-and local CSOs co-partnership to submit proposal and manage grant together, CSOs alliance joint program to submit and manage grant collectively, independent local grant-making institutions to manage donors’ grant for local CSOs. Final Evaluation: Component One, Access to Justice Program | 2012 50 Annex A – Terms of reference Statement of Work Consultant to conduct final evaluation of Component One of the Access to Justice Program The Asia Foundation Timor-Leste is seeking an external consultant to conduct a final evaluation for Component One of the USAID funded Access to Justice Program. The evaluation will review progress in relation to a three year extension granted to the Foundation which started in October 2009. The evaluation is to commence in August and be finalized by mid-September 2012. Background The Access to Justice (ATJ) Program of the Asia Foundation (the Foundation) – funded by USAID – began in 2002. The period covered by this evaluation is October 2009 to September 2012. The program has the following four objectives: 1) To enhance the quality of legal aid services; 2) To expand the reach of legal aid services; 3) To promote skills-based legal education; and 4) To improve public access to legal information. Through the program, the Foundation currently funds four legal aid organizations. Through these sub-grants, the Foundation supports the provision of free legal aid services for both litigation and non-litigation. The legal aid program places an emphasis on accessibility of legal services for resolving disputes and seeking redress, in particular in remote areas where information about the judicial system is limited, as is access to the legal aid services provided by the government. Two of the sub-grants also provide funding for paralegal programs through which 27 paralegals are placed at the sub-district and village level in five districts – aimed at bringing legal services closer to remote communities. In addition to the provision of sub-grants to legal aid organizations, the Foundation undertakes the following activities to support community access to justice: - Training support to enhance the technical capacity of the legal aid programs, through practical litigation and non-litigation skills trainings and case management support. - Support to networking among legal aid lawyers, including regional linkages. - Contributing to the debate about sustainability of legal aid sector, including through technical support to the government in developing a draft legal aid law. - Support to women legal aid lawyers and enhanced services for women clients. - Legal information dissemination activities, including through legal aid organizations Finally, in cooperation with Stanford Law School, the program has established a legal education program which aims to allow for exposure of those teaching law in Timor￾Leste with methodologies in other countries, and the development of materials in local Final Evaluation: Component One, Access to Justice Program | 2012 51 languages on RDTL law. As part of this program, ATJ has produced text books on RDTL law, the first of which was launched in November 2011. The program also allows for exchanges between UNTL and SLS at both the student and teaching level. Activity The Foundation is engaging two consultants to work together to conduct an evaluation of Component One of the Access to Justice Program. The evaluation is required to assess the performance of the program against program objectives. The evaluation will assess the program’s overall effectiveness, its appropriateness re current context and impacts. Methodology The evaluation is to be based on analysis of empirical data related to client services, document reviews and key informant interviews, including direct interviews with former clients. The consultant is expected to collect information from ATJ project team members, partners and stakeholders of the project regarding impacts of the project activities towards the achievement of project goal and strategic objectives. Key to part of the evaluation will be reviewing data on the legal aid organizations’ client service delivery. The Foundation collects monthly statistics regarding clients and has now undertaken two client surveys each of 120 legal aid organization clients. It is also expected that as part of the evaluation, the consultant will visit at least two of the district based partners and travel to more remote regions, such as Suai. In relation to the legal education component of the program, the consultant will review the materials produced and assess the relevance of the materials through key informant interviews. Duties of the consultant The consultant will be required to work as a part of a team of two consultants. : - Collaboratively develop a work plan for the evaluation and a schedule; - Review project proposal, bi-annual reports, annual work plans, the mid-term evaluation report, partners’ reports, field visit reports, results of the assessment of the Paralegal Program, project documents, materials (including training materials) produced by the Foundation; Review the results of the 2011 and 2012 client surveys of client services provided by the legal aid organizations; - Review and analyze as appropriate the Foundation’s database of information from legal aid organizations supported by the program. - Conduct interviews with key informants regarding the program to determine program effectiveness. Those interviewed should include at least the following- Foundation staff, USAID, Ministry of Justice officials, court actors, legal aid organizations, AATL, women’s and human rights contacts, those working in legal education, former clients of legal aid organizations, and other development partners working in the justice sector. - Collaboratively develop an evaluation report outline and determine jointly lead writers for particular sections; - Take overall responsibility for the writing of the evaluation report. The report should examine program effectiveness and impact in relation to the program’s goals. To assist with determining further interventions, the report should evaluate the current sustainability of the legal aid sector. In particular the report should evaluate the different components of sustainability of the legal aid sector, that is, financial, human resource, operational and legal. The report should identify priority issues requiring further Final Evaluation: Component One, Access to Justice Program | 2012 52 assistance, comparing the community legal aid sector in Timor-Leste with similar country contexts. - Discuss preliminary findings and recommendations of the evaluation with the Foundation; - Finalize the evaluation report and submit to the Foundation. Deliverables The consultant is expected to complete the following deliverables: 1) A plan for the evaluation. 2) A report evaluating component one of the Access to Justice Program, 2010 to 2012; 3) A presentation of preliminary findings and recommendations to the Foundation; 4) Submission to the Foundation of a final report (not more than 40 pages) of the evaluation. Supervision The consultant will be supervised by Silas Everett, Country Representative, TAF Timor￾Leste. Duration of the appointment The final evaluation is expected to be conducted in August with a report finalized by mid￾September 2012. Remuneration The consultant will be remunerated by The Asia Foundation according to successful candidate’s previous salary history and industry standards. Minimum Qualifications Experience Previous experience in the legal sector in Timor-Leste and a strong understanding of the current state of the judicial system preferable; Experience with the NGO sector, in particular community legal aid organizations, in Timor-Leste preferable; Demonstrated experience in legal/rule of law project implementation, in a developing context; Demonstrated experience of evaluations of relevant programs; Demonstrated ability to work independently. Other Excellent analytical and writing skills; Bahasa Indonesia or Tetum essential; Portuguese desirable; Computer proficiency in word processing (Word), Excel spreadsheets. Final Evaluation: Component One, Access to Justice Program | 2012 53 Annex B – Interviews No. Interviewee Entity Date 1 Adelio Tilman Senior Program Officer 24 August 2012 2 Geoffrey Swenson Law Program Manager 24 August 2012/4 Sep 2012 3 Kerry Brogan Chief of Party 24 Aug/4 Sep/6 Sep/20 Sep 2012 4 Sancia Bano Director, FFSO 28 August 2012 5 Raymundo de Fatima Afi, Armaldo Colo, and Jaime Baz Legal aid lawyers and paralegal coordinator, FFSO 28 August 2012 6 Sebastiao Cob, Fransisco Sahe, Artistane, Gaspar Bubifi, and Luisa Paralegals, FFSO 28 August 2012 7 Juliao de Deus Fatima Program Officer 28 August/7 Sep 2012 8 Julio Antonio Finance Manager, FECM 31 August 2012 9 Lino Lopes Director, FECM 31 August 2012 10 Alexo da Silva Belo and Esmeralda Dasdores Martins Paralegals, FECM 31 August 2012 11 America Luis Freito Belo, Marcellus Severres Guterres, Elvino Sarmento Freitas, Francisco Joachim da Costa, Horta Romas and Haguide de Fatima Legal aid lawyers, FECM 1 September 2012 12 Marcelina Amaral Acting Coordinator of VSS 3 September 2012 13 Maria Filomena Babo Martins Chief of Department of Training, SEPI 3 September 2012 14 Aderito dos Reis Director of LBH Liberta 3 September 2012 15 Gaspar da Silva Program Assistant 3 September 2012 16 Claudio de Jesus Ximenes President of Court of Appeal 4 September 2012 17 Sergio Hornai Director of Public Defender Office 4 September 2012 18 Marcelina Tilman Director of CFJ 4 September 2012 19 Fernando do Carmo Program Officer 4 September 2012 20 Ana Guterres, Germano Boavida, Brett USAID 5 September 2012 21 Benjamin Corte Real UNTL/National Institute of 5 September 2012 Final Evaluation: Component One, Access to Justice Program | 2012 54 Linguistic 22 Roberto Pacheco Legal Research Unit, JSMP 5 September 2012 23 Nicolau dos Reis da Costa Program Officer 5 September 2012 24 Luis Sampaio de Oliveira Director of JSMP 5 September 2012 25 Ana Pessoa Prosecutor General 6 September 2012 26 Zelia Trindade Vice Prosecutor General 6 September 2012 27 Yasinta Lujina Rede Feto 7 September 2012 28 Carsten Weber UNMIT 7 September 2012 29 Maria Agnes Bere UNMIT 7 September 2012 30 Sophie Cason Justice Facility 7 September 2012 31 Annika Kovar UNDP 7 September 2012 32 Jose Guterres Private lawyer 7 September 2012 33 Sahe da Silva Private lawyer 7 September 2012 34 Aquelina da Costa Savio Program Officer 7 September 2012 35 Manuel Tilman Secretary General of AATL 10 September 2012 36 Nelinho Vital National Director of Judicial and Legislation, Ministry of Justice 13 September 2012 37 Carolyn Tanner UNMIT HRTJS, Monitoring Unit 13 September 2012 38 Silas Everett Director, Asia Foundation 20 September 2012