Mid-Term Evaluation of the Ba Distrito Program iv MID-TERM EVALUATION THE BA DISTRITO PROGRAM August 1, 2016 This publication was produced at the request of the United States Agency for International Development. Social Impact, Inc. prepared this report independently. AID-486-I-14-00001/AID-472-TO-16-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development. PHOTO CREDIT: SELMA HAYATI / USAID Mid-Term Evaluation of the Ba Distrito Program v Figure 1: District and Border Map of Timor-Leste Mid-Term Evaluation of the Ba Distrito Program vi TABLE OF CONTENTS Executive Summary ........................................................................................................................................................... 1 Evaluation Purpose and Intended Use ........................................................................................................... 1 Evaluation Questions and Methods ................................................................................................................ 1 Key Findings: Local Governance Strengthening ........................................................................................... 2 Key Conclusions: Local Governance Strengthening ................................................................................... 3 Key Findings: Decentralization and Input of Local Institutions ................................................................ 4 Key Conclusions: Decentralization and Input of Local Institution .......................................................... 5 Key Findings: Legal Aid Organization Sustainability .................................................................................... 6 Key Conclusions: Legal Aid Organization Sustainability ............................................................................ 7 Key Findings: District Court Functionality ................................................................................................... 8 Key Conclusions: District Court Functionality ........................................................................................... 9 Selected Recommendations ........................................................................................................................... 10 Evaluation Purpose and Questions .............................................................................................................................. 13 Project Background ......................................................................................................................................................... 14 Evaluation Methods and Limitations ............................................................................................................................ 15 Data Collection Methods................................................................................................................................ 15 Evaluation Limitations ...................................................................................................................................... 18 Findings, Conclusions, and Recommendations ......................................................................................................... 20 Key Findings: Local Governance Strengthening ......................................................................................... 21 Key Conclusions: Local Governance Strengthening ................................................................................. 24 Key Findings: Decentralization and Input of Local Institutions .............................................................. 26 Key Conclusions: Decentralization and Input of Local Institutions ...................................................... 28 Key Findings: Legal Aid Organization Sustainability .................................................................................. 29 Key Conclusions: District Court Functionality ......................................................................................... 35 Key Findings: Household Survey Data ......................................................................................................... 37 Key Conclusions: Household Survey Data ................................................................................................. 40 Recommendations: Decentralization and Input of Local Institutions................................................... 42 Recommendations: Legal Aid Organization Sustainability ...................................................................... 43 Recommendations: District Court Functionality ...................................................................................... 43 Recommendations: Utilization of Household Survey Data .................................................................... 44 Annexes ............................................................................................................................................................................. 45 Annex I: Evaluation Statement of Work ..................................................................................................... 45 Annex II: Interview Protocol and Templates ............................................................................................. 50 Annex III: Bibliography ..................................................................................................................................... 56 Annex IV: Ba Distrito Evaluation Interview List ........................................................................................ 58 Annex V: Evaluation Question Matrix ......................................................................................................... 61 Annex VI: Ba Distrito 2016 Household Survey questionnaire .............................................................. 63 Annex VII: Household Survey Data .............................................................................................................. 71 Annex VIII: Disclosure of Any Conflicts of Interest ................................................................................. 71 TABLE AND FIGURES Table 1: Final Evaluation Questions.............................................................................................................................13 Table 2: Findings, Conclusions, and Recommendations .........................................................................................20 Table 3: Baseline and Midline Measures for Oecusse, Baucau and Covalima (combined) .............................37 Mid-Term Evaluation of the Ba Distrito Program vii ACRONYMS AAP Annual Action Planning COR Contracting Officer’s Representative DNAAS National Directorate for Support and Suco Administration DQA Data Quality Assurance ET Evaluation Team GoTL Government of Timor-Leste KAB Knowledge, Attributes, and Behavior LADV Law Against Domestic Violence LTC Legal Training Center MOJ Ministry of Justice MOU Memorandum of Understanding MSA Ministry of State Administration NGO Non-governmental Organization PMEP Performance Monitoring and Evaluation Plan PNDS National Program for Village Development SGBV Sexual and Gender Based Violence SI Social Impact SOW Scope of Work SSD Social Science Dimensions STTA Short-term Technical Assistance TAF The Asia Foundation TEC Total Estimated Cost TL Timor-Leste TO Task Order USAID United States Agency for International Development 1 EXECUTIVE SUMMARY In September 2013, the United States Agency for International Development (USAID) awarded a four￾year program called Ba Distrito (“To the Districts”) to Counterpart International. The goal of the activity is to increase institutional and human capacity at local levels to deliver basic services effectively and in a manner that is responsive to citizen needs and expectations. The activity is being implemented in 100 sucos in four municipalities: Baucau, Covalima, Ermera, and Liquiça as well as in the Special Administrative Region of Oecusse-Ambeno. Initially the activity had four components, with a Total Estimated Cost (TEC) of $6 million. In April 2015, the Cooperative Agreement (AID-486-A-13-00007) was modified to include an additional component to support anticipated suco elections and the TEC was raised to $7 million. The main beneficiaries of the activity are members of suco councils, legal aid organizations, court actors (i.e. judges and judicial officers), and targeted rural communities, especially marginalized populations, in the five project areas. The project activities and components are structured to contribute to the wider vision of improved decentralized governance and inclusive access to justice in Timor-Leste. EVALUATION PURPOSE AND INTENDED USE The purpose of this Task Order (TO) was to conduct a mid-term evaluation of the USAID Ba Distrito Program implemented by Counterpart International. The TO included conducting a midline of the household survey in the original three project municipalities and a baseline in two additional municipalities added to the project in 2015, in addition to collecting qualitative data from the original three project municipalities. For this evaluation, the evaluation team (ET) looked at four component areas of the Ba Distrito program: (1) local governance strengthening, (2) decentralization and input of local institutions, (3) legal aid organization sustainability, and (4) district court functionality. The purpose of the mid-term evaluation is primarily to evaluate progress of the program against the baseline survey conducted in 2014; to conduct the same baseline survey for the two municipalities that were added to the activity in November 2014; and to identify challenges faced and present recommendations for addressing them to USAID and the Ba Distrito team. To assist in clarification and utility of the report, each project component area was analyzed for the following: 1) Challenges and Lessons Learned: defined as contextual impediments impacting project goals, objectives, activity planning, and lessons learned from addressing these challenges; 2) Project Implementation: defined as direct results and outcomes plausibly correlated to component and project activities; (3) Unintended Outcomes: defined as unintended impacts, outcomes, or spillover effects as a result of project implementation; and (4) Gender Sensitivity: defined as project practices, tools, and methods demonstrating gender awareness in design, implementation, and management. The USAID Mission in Timor-Leste and the project implementing partner are the primary audience for and intended users of this evaluation. The evaluation may also be shared with other donors and other USAID offices to inform the development of future efforts in other countries. This evaluation will inform the Mission about the activity’s performance and achievements, and will be used by the Mission to make programmatic adjustments to the project, if needed. EVALUATION QUESTIONS AND METHODS The ET was contracted to answer the following four questions: 1. What are the main challenges faced by the program and lessons-learned to-date? 2. What changes, if any, are proposed regarding implementation? 3. What, if any, unintended consequences have occurred as a result of the program? 2 4. How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? The ET used a mixed methods approach, including qualitative analysis and statistical analysis of household survey data both at the mid-term and the baseline. The evaluation consisted of three phases: 1) a comprehensive desk study of all relevant program documents; 2) approximately four weeks of field data collection in Timor-Leste, involving quantitative and qualitative data collection; and 3) analysis of all data collected, report writing, and final presentations. KEY FINDINGS: LOCAL GOVERNANCE STRENGTHENING Challenges and Lessons Learned  Suco Councils are not formally a part of government, but they are critical for current decentralization planning and implementation in Timor-Leste. There were challenges in coordinating some training services between Ba Distrito and the Ministry of State Administration (MSA) and particularly the National Directorate for Support and Suco Administration (DNAAS), the Government of Timor-Leste (GoTL) entities responsible for suco council administration and support.  Decentralization planning is political in Timor and suco councils play a critical role in extending the reach of the state in rural areas. According to suco council members and representative sources interviewed in Baucau, Covalima, and Oecusse, most suco council members (74%) would prefer to remain outside of government to ensure that traditional practices and roles of the suco are protected. This was also supported in interviews with the majority of Municipal Administrators (85%).  One of the major challenges is that the coordination and communication between suco councils, municipal ministry line staff is variable across the targeted municipalities and sucos. Project interventions such as municipal forums, suco exchanges and suco expos have helped to create more space for suco council and municipal and national GoTL staff to engage. MSA staff interviewed were supportive of these public forums, but most view sucos as community associations and stated that there are limited resources or incentives for line ministry staff to respond to citizen requests based on the current planning process within the GoTL. Program Implementation  Some advocacy activities implemented by Ba Distrito, particularly public consultations on draft laws confuse GoTL partners, and interviewed sources expressed a desire for Ba Distrito project staff to improve coordination and planning in advance at the municipal and national level. Specifically senior level municipal staff want to be more engaged when recommendations are being formed for draft or pending legislation specific to local power and decentralization. This concern was more acute in Baucau, but similar recommendations were offered from GoTL staff in Covalima and in interviews with the MSA staff in Dili.  There were specific requests in interviews for more detailed coordination with DNAAS staff on proposed training for suco council members, particularly suco chiefs and local leaders. Although there have been consultations on the manuals and training modules, staff changes within the GoTL may require more technical support and coordination with Ministry staff on a regular basis.  Some training, such as the “Access to Justice and Women’s Rights” module provided by Ba Distrito for suco council members was too long according to some interviews. Based on interviews with suco council members who participated in the training, the content was good, but could have been delivered in two segments.  Some members were not able to attend the entire training due to logistical issues that they felt should have been better addressed by Ba Distrito staff and partners responsible for the training 3 logistics and planning. Some suco council members were not able to attend the entire training as the length of the training, cost for transportation and the distance they needed to travel made attendance difficult. Unintended Outcomes  An unintended outcome was that Ba Distrito staff interviewed felt coordination was strong between the project and the MSA, but staff within the Ministry felt that this could be improved. Recent GoTL staff changes as well as small budgets for Directorate level staff to observe field based trainings may also have contributed to their perception of weak coordination.  According to interviews with GoTL staff working with the MSA in Baucau and in Dili, coordination needs to be more formalized. Ministry staff requested more discussion between Ba Distrito and DNAAS staff in integrating specific content from the Ba Distrito training modules into future training efforts and planning of the National Directorate.  Ministry staff specified in interviews that they have specific training protocols, manuals, and plans that need to be better integrated into Ba Distrito efforts annually. National Directorates are responsible for Annual Action Planning (AAP). Chiefs of Departments are only required to submit quarterly activities and these are integrated into the AAP for the Directorate and then combined for the Ministry’s AAP. Department Chiefs interviewed wanted more input from Ba Distrito on training activity improvements so that they could more effectively incorporate Ba Distrito training ideas into the Directorate planning process. Gender Sensitivity  Ba Distrito staff, policies, and practices have sought to integrate gender sensitive approaches into implementation and management. By specifically targeting women, suco council representatives, and youth there was a consistent and practical flow of information and feedback from these beneficiary groups that was used to better manage and adapt project activities.  In interviews with Ba Distrito staff there was evidence that project management and policies relevant to gender sensitivity were in place and well sensitized across most component areas. KEY CONCLUSIONS: LOCAL GOVERNANCE STRENGTHENING Suco Councils are not formally a part of government and Ba Distrito staff and partners recognize this, but suco council members are increasingly targeted for activities not only by USAID, but also by multiple partners at the local and national level. Suco councils are critical for extending the reach of the government and for providing critical information to local citizens, many living in remote and distant areas from municipal centers. Ba Distrito’s target of suco council members was appropriate, but one of the challenges the project faced was in ensuring that the work that they do with members was well coordinated with the MSA. Ba Distrito and DNAAS both have small staffs. More formal mechanisms for project coordination would have eliminated some of the inherent tensions that existed between the project and some of its activities. There were specific requests for more detailed coordination with DNAAS staff on proposed training for suco council members, particularly suco chiefs and local leaders. More lead time for participation of DNAAS officials in field level events was also requested to improve participation and ownership of Ba Distrito efforts and GoTL programs. The MSA specified that they wanted more coordination on specific training protocols, manuals, and plans that Ba Distrito needed to integrate into current Ministry training plans. State officials within DNAAS were clear that they valued the content of the training, but wanted more direct engagement in the development, design and delivery. Specifically they wanted to be able to find ways to integrate some of the material into their own annual planning, but their budget and resources are limited so they would need to map this out carefully with Ba Distrito staff. The ET recognized that the GoTL staff has changed over the life of the project and there has been coordination on these issues in the 4 past, but current needs within MSA are shifting. Pending legislation and changes in suco structure are driving internal needs within DNAAS to prepare for these changes. Ba Distrito as a project was particularly strong in gender sensitivity and integration. There was clear evidence that staff was well trained and had integrated thoughtful and effective targeting practices into their activities to elicit more active engagement of women, youth, people with disability, and the elderly. Of particular note was the underlying and consistent utilization of language, methods, and monitoring of gender impact on project activities. By specifically targeting women suco council representatives and youth there was a consistent and practical flow of information and feedback from these beneficiary groups that was used to better manage and adapt project activities and this was evident in staff practices and in documentation review. KEY FINDINGS: DECENTRALIZATION AND INPUT OF LOCAL INSTITUTIONS Challenges and Lessons Learned  There are not yet decentralized local government structures in Timor-Leste, so local government presence varies at the municipal level based on ministerial resources (i.e. budgets and staff) available. This condition proved to be challenging for Ba Distrito staff, particularly municipal coordinators who had to introduce the program to multiple staff as GoTL staff have changed several times at the municipal level as well as national level during the course of the project.  One of the challenges of the program is the dynamic nature of decentralization legislation in Timor-Leste (TL) and how this has impacted the timing and planning of specific activities. There are three pieces of relevant legislation, (1) the draft Local Power and Decentralization Administration Law, (2) draft Suco Law No. 32_III. January 2016 and (3) the recently approved Decree Law No. 3/2016 that defines Municipalities administration, Municipal authorities and the inter-ministerial technical group for administrative decentralization. There is also a law that was passed, Decree-Law no. 4/2014 on Administrative Pre-deconcentration that has directly impacted the project. This context has challenged staff of Ba Distrito to be in pace with proposed changes in suco council structure as well as governance and local planning practices at the municipal level. Primarily these contextual factors have impacted the project in the areas of training development and advocacy work. Program Implementation  The implementation of Decree-Law no. 4/2014 on Administrative Pre-deconcentration has resulted in a shifting landscape of GoTL staff fielded into new positions and roles. This constant shifting landscape impacts all Ba Distrito components.  Pending and recently approved legislation requires a considerable amount of analysis and careful planning with GoTL staff for government inclusion in public forums and advocacy efforts which is challenging for a small project such as Ba Distrito with a limited budget and a widely dispersed implementation area. In addition there are often varying GoTL timelines for public consultation and limited notice for international partners seeking more public information and inclusion of citizens in draft laws and legislation pending.  Delays in elections and legislation will continue to challenge the pace and impact of the project. According to interviews with Ba Distrito staff advocacy activities had to be carefully planned and gathering recommendations from suco council members was challenging as legislative delays impacted local will and understanding of critical legislative issues impacting local forms of governance. Unintended Outcomes 5  According to interviews with Ba Distrito staff and partners, this context has challenged staff to build strong formal coordination mechanisms with GoTL stakeholders at the municipal and national level. One of the unintended outcomes of these factors is that GoTL partners viewed some advocacy and stakeholder consultation events negatively as communication did not reach the necessary levels in time for national staff to feel informed and to identify areas for more effective national cooperation.  In some trainings events participants come from other sucos or areas that were not targeted by the Ba Distrito project. For example, this happened with a suco expo event and in also in some legal aid training events. This is a sign that project interventions implemented are appreciated outside of the project target areas. This is a positive unintended outcome that deserves to be noted. Gender Sensitivity  This is a component area where gender integration was evident from strategy, to implementation, to activity level monitoring. In interviews with Ba District staff and in SGDs held with suco council members there was evidence that advocacy activities and civic engagement stakeholder forums were organized consistently to include the representation of women. Ba Distrito staff in this component area had specific strategies and approaches that were used consistently and there was ample evidence that this had elevated and provided space for public speaking opportunities and the insights of women to be incorporated into recommendations presented to the GoTL. KEY CONCLUSIONS: DECENTRALIZATION AND INPUT OF LOCAL INSTITUTIONS One of the challenges relevant to the process of decentralization planning in Timor is the lack of information that extends at the national level within Ministries, but also within civil society. Ba Distrito NGO partners interviewed were also unsure about the status of the legislation and proposed changes to deconcentration and decentralization and its impact at the local level. Ultimately, despite over a decade of planning, decentralization is still a moving target in Timor and its political nature, controversial realignment of geographic areas, and potential for large-scale deconcentration and government reorganization is not well defined or understood. This impacts the relationship between civil society and the government and also has impacted some of the advocacy efforts of Ba Distrito and the way they are perceived by GoTL staff as well as select national stakeholders. Delays in elections and legislation will continue to challenge the pace and impact of the project, but creating more effective and current networks will improve access to information for local communities. One particularly dynamic area of learning is Oecusse, which by its very history of isolation has made it one of the first decentralized areas. There is much to be learned about decentralization, deconcentration, integrated planning, and suco level engagement in this particular region of Timor-Leste. This component and the staff associated have a unique opportunity to work on understanding and integrating these lessons into project planning and implementation. The pace and challenge of decentralization implementation in Timor-Leste would be difficult for any project to contend with. Advocacy and community consultations are laudable and necessary components of the Ba Distrito program design, but with these efforts come risks as well as rewards. The volatile, political, and unpredictable nature of legislation and electoral planning make these efforts sensitive to GoTL staff and partners. It is critical that the project expand their analysis and actively integrate this analysis into program planning in the final stages of project implementation. Small efforts to enhance communication, share analysis, and create practical pathways for national ownership and collaboration will strengthen the project and allow for national adoption and ownership of promising Ba Distrito approaches and models. 6 This is a component area with laudable gender integration and the staff articulation and clarity as to how these methods were utilized was impressive. Of particular note was the clear and precise way that Ba Distrito staff organized community consultations and ensured participation and voice from minority groups. This was not just a written project approach, but a clear and established staff practice. This gender integration practice was designed at the early stages of the project, but was fully implemented in this component, particularly in community consultations, building local recommendations, and aligning advocacy efforts with suco council engagement and ownership. KEY FINDINGS: LEGAL AID ORGANIZATION SUSTAINABILITY Challenges and Lessons Learned  Legal Aid organizations at the municipal level suffer from low capacity and minimal resources. This weak capacity has impacted the level of activities the partner organizations can realistically accomplish under the Ba Distrito grants program. This was evident in interviews with all legal aid partners.  Although some of the partners (i.e. FFSO in Oecusse) have been partners of past access to justice efforts funded by USAID, significant institutional issues remain, including weak financial capacity, low staffing to activity ratio and in some cases minimal capacity to facilitate cases in the local language.  Technical capacity building for legal aid partners is an area of emphasis in the Year Three Work Plan after an organization assessment was conducted by the project However, the needs are significant and partners may have benefitted from these activities being planned at earlier stages of the project cycle or more emphasis placed on capacity building as a primary component of legal aid organizational sustainability.  Based on fieldwork, the legal aid partner in Oecusse only worked with perpetrators or accused parties of domestic violence cases (95% men). Based on interviews with field staff of the implementing partner, there was minimal coordination linking perpetrators to anger management or case management services with other service providers in Oecusse.  There were partners in Oecusse who were part of a monthly working group, but it was evident from interviews with the lawyer and senior staff that the coordination could be improved to ensure more support for these mitigation needs. Program Implementation  In terms of implementation, the key areas where legal aid organizations needed more support were proposal and report writing, financial planning, and legal training. These areas of deficiency were noticeable in Baucau, Oecusse, and Covalima.  Language is also a significant challenge to effective implementation as local language is essential in some settings, particularly with poor, vulnerable, and under-educated clients.  Legal Aid partners interviewed stated that the personal legal aid training at the suco level was effective and several partners supported radio programs on legal aid information, which included radio dramas with gender sensitive messaging. However, during interviews at the suco level most suco council members interviewed did not have radios.  At the municipal level in interviews the radio messages had been heard and well broadcast, but reaching the most vulnerable and marginalized through this medium may need some strategic planning based on the norms in Timor and the limited access at the suco and aldeia level to radios.  Legal aid partners interviewed felt that the mobile clinics and local legal education sessions in Baucau, Covalima and Oecusse were well received.  During the interviews with suco council members this was confirmed, but participation was limited, as not all suco members interviewed had attended the trainings. Sometimes based on availability other suco members would attend. Based on interviews, the most popular trainings 7 pertained to access to justice and women’s rights, particularly those with information on land and property rights. Unintended Outcomes  One of the unintended outcomes of this particular component area was the weak environment that existed as a baseline for the project in choosing effective legal aid organizations to partner with in municipalities with few partners working in this technical area.  There is an evident need based on fieldwork for legal aid information to be made more accessible to marginalized populations, but some of the partners were too under resourced to effectively meet the targets established based on the budgets proposed for these activities.  Building upon past USAID efforts by utilizing past partners was efficient, but there was a need to look at some of the dependency and sustainability issues of a few of the NGOs chosen. This is particularly relevant to particular legal aid organizations, but all partners interviewed noted funding and sustainability issues.  The project has a very limited budget for these activities and the project scope and scale is challenging. Creating access to justice for remote and marginalized populations is not as easy as creating social and media messaging. The enabling environment itself is weak and civil society partners are not strong financially. There are limited protections and regulations for the work and services that they provide. Gender Sensitivity  This component did seek to target and increase access to justice for women and marginalized groups. Training efforts targeted marginalized groups and women. Training was provided on issues such as access to legal aid for domestic violence issues, women’s rights relevant to land and property, and training for women suco council members on conflict resolution and mediation.  Radio programs also sought to create gender sensitive messaging, and most programs tried to incorporate programming specific to the needs of women. KEY CONCLUSIONS: LEGAL AID ORGANIZATION SUSTAINABILITY One of the biggest challenges for this component was the weak capacity of the legal aid service provider partners. It was not just the capacity, but the organizational structure, financial management, and staffing resources. Although the project had conducted an organizational capacity assessment for legal aid partners earlier in the project cycle, these needs may have needed to be prioritized earlier in the project cycle. The budget allocated from Ba Distrito provided to these partners was also very low and the staff ratio to activity output was challenging at best. This is an issue for USAID to consider as the budget for the access to justice components is limited for the scope and scale of the project. In several instances the local language was also a challenge in disseminating information and ensuring that the targeted population was reached. The legal aid environment in Dili is small, and at the municipal level, the partner selection is even smaller so the pool of potential partners for a project with the ambitions and objectives of Ba Distrito is a challenge not easily overcome. Some of the particular gaps noted by Ba Distrito legal aid partners were in the following technical areas: report writing, strategy planning, and legal training. These areas of deficiency were noticeable in Baucau, Oecusse, and Covalima. In interviews with program managers, technical staff, and directors, specific examples were given to show how these deficiencies impacted project performance. In analyzing the small grant funding levels for these partners it was clear the budget issue was also a significant hurdle. One of the unintended outcomes of this component area was in some ways beyond the control of the project as the pool of potential partners was small and the baseline was significantly low capacity and 8 resources of these groups. What is needed is a technically driven mentorship program for legal aid providers and a targeted capacity building approach with clear institutional strengthening measures tracked. The project has conducted a detailed baseline assessment of the organizational capacity of legal aid partners, but budget and staffing issues have relegated many of these interventions until the final year. The environment for legal aid provision in Timor-Leste is weak, but there is a significant need to extend these services into remote environments where services and information are nonexistent. Former, albeit much larger, justice programs funded by USAID had used a local paralegal approach and this may have been a better practice for long term sustainability where weak justice access is likely to be the norm for an extended period of time. This allows more local access and also increases resources at the suco level. The budget levels, however, of Ba Distrito are far lower than the previous USAID access to justice efforts, a reality that should be a consideration for future planning. This component sought to address the most marginalized and vulnerable groups, but in some cases the work of the legal aid partner was not monitored and managed as effectively as it could have been. The legal aid partner in Oecusse only worked with perpetrators or accused parties of domestic violence cases (95% men). Based on evidence there was minimal coordination with this particular legal aid partner and other services that may have been appropriate to create gender awareness or link potential perpetrators to anger management or case management services. This is an area that should be addressed and better coordinated with other service providers to ensure a more holistic and measured approach to address gender needs in this particular component area of the project. Although there are monthly coordination meetings based on interviews conducted all legal aid partners working in this technical area do not always attend these and follow up services are not well coordinated. KEY FINDINGS: DISTRICT COURT FUNCTIONALITY Challenges and Lessons Learned  The Justice Sector in Timor-Leste is weak, and political problems impact the performance and partnerships of the project. One of the challenges the project faced was the inability to sign a memorandum of understanding (MOU) with the Ministry of Justice. This was not the fault of the project, but was a current political and practical reality that the project faced.  The Justice sector in Timor is also critically aligned with the Portuguese system and this stems from Portuguese being the primary language of law in Timor, but also from the support over the years of Portuguese legal providers and the role and influence of CPLP. This preference for Portuguese rooted training also impacted the project.  To mitigate these challenges, the project researched and analyzed gaps in the legal aid environment in Timor. The project also published technical assistance and research reports highlighting the deficiencies the justice system faced. These studies focused particularly on analyzing the access to justice issues that minority groups, particularly women, the elderly, and victims of domestic and sexual and gender based violence (SGBV) faced in accessing justice. These studies were widely referred to by partners, and in interviews with district court staff. Program Implementation  Based on initial problems partnering with the Ministry of Justice the project established an MOU with the Court of Appeals and established two small-scale pilot programs for the district courts of Baucau and the Special Administrative Region of Oecusse. Based on interviews this was a reasonable pathway for mitigation, but the results were mixed and specific to the particular pilot court interventions.  There are weak procedural issues that impacted the functioning of the court in Baucau and Oecusse. According to NGO partners working with the courts, variations in court procedures impact citizens who seek access to these courts. In a review of data and based on interviews with 9 Appellate judges and Court Secretaries the procedural issues in both courts are extensive. These are young courts with immature systems, and minimal time for training for judges and for court staff. Primarily this impacted data and case management.  There are sentencing irregularities in Oecusse that are the result of insufficient staffing and human resources. There is only one judge in Oecusse who can serve on over 200 cases a year. The caseload alone is unsustainable, so irregularities are not surprising based on interviews with district court staff in Oecusse as well as the Legal Training Center responsible for judge training. There was clear evidence that there is just not time or resources for judges to be trained and to cover caseload management needs.  Court cases on domestic violence are the highest in Oecusse and the legal aid partner of Ba Distrito, believes this is a result of increased awareness of suco chiefs in Oecusse to not mediate domestic violence issues. Training, mobile clinics, and strong relationships between legal aid providers, women’s rights organizations, and the suco chief all contribute to these factors.  Based on field interviews with suco council representatives the increase in cases is a natural reaction to more training for traditional leaders who with increased awareness refer more cases to the formal justice system. However, while this could not be clearly attributed to Ba Distrito interventions. Ba Distrito’s three-day training course “Access to Justice and Women’s Rights” was delivered to all suco councils in Oecusse together with JSMP and aimed to increase the awareness of suco chiefs as well as Lia-Na’in (traditional leaders) about the law against domestic violence and the requirement to refer these cases to the criminal justice system. Unintended Outcomes  Ba Distrito’s support for statistical reporting is simple, but still requires significant training in data entry and tool management. It also requires local oversight on a more regular basis. Statistical reporting services will help improve the functionality of the district courts, only to the extent that the analysis is used by staff for improved case management and tracking. More regularized reporting can assist case management, and also allow more effective trend analysis and monitoring. One of the unintended outcomes is that the tool is appreciated by staff and stakeholders interviewed, but there are limitations to its utility and adoption based on the outstanding training and implementation issues. KEY CONCLUSIONS: DISTRICT COURT FUNCTIONALITY The Justice Sector in Timor-Leste is weak, and political problems persist within the Ministry of Justice and in independent institutions such as the district courts that make access to justice an exceptionally complex challenge that is likely to be embedded in the Timor context for many years. One of the challenges the project faced was establishing ownership of the activities, strategies and trainings planned with a small budget. There is a strong preference for national or Portuguese trainers as the system is predicated on establishing national capacity and Portuguese is an essential and deeply rooted relationship within the justice sector. There is a clear need within the project to address these challenges head on for the remainder of the project performance period. Requests were made for Portuguese Administrative and Legal Trainers both at the national and the municipal level. This project component has smart, competent, and well-informed staff, but mitigating these challenges with a relatively small budget and minimal staff is a significant hurdle to overcome. The research and analysis of gaps in the legal aid environment in Timor conducted by the project holds a wealth of information for future project planning, but it would be better to separate these into two separate projects, one focused on support for decentralization and the other looking at increasing access to justice for marginalized populations. The context requires separate resources, strategies, and staff approaches. This impacted Ba Distrito and though they did an exceptional job managing these technical disconnects, this may be something USAID may want to consider in the future. National budgets are small and there 10 is less of a need for huge access to justice programs, but more targeted and effective methods to address critical gaps and deficiencies in increasing access for the poor and marginalized. Partnering with the Ministry of Justice the project proved to be a challenge and implementation mitigation strategies were established to address this issue. An MOU was signed with the Dili Court of Appeals and two pilot programs established with the district courts in Baucau and the Special Administrative Region of Oecusse. Based on interviews, this was a reasonable pathway for mitigation, but the results were mixed and specific to the particular pilot court interventions. Weak procedural issues impacted the functioning of the court in Baucau and Oecusse. There were significant variations in court procedures and this impacted citizens seeking access to justice in these courts. In the courts there is irregularity in the sentencing, fine structure and remediation offered to victims and perpetrators of domestic violence. There has been an emphasis on increased awareness in communities of GBV by Ba Distrito, but it appears that the courts themselves also need enhanced training. Consistent court monitoring and advocacy to improve judicial practices to protect victim rights are considered critical by most sources interviewed. Judges have been trained in the Law Against Domestic Violence (LADV) as well as other issues relevant to human rights infractions, but there is not enough awareness at the district level and perhaps experience in establishing guidelines for sentencing that protects victim’s rights. Irregularities in sentencing impact whole communities and also influence victims who may not seek formal remediation for crimes based on past sentencing of the courts and their knowledge of this. There are procedural issues that impact the functioning of the court in Baucau and Oecusse. Variations in court procedures challenge court reporting, monitoring and case documentation and management. Ba Distrito’s support for statistical reporting is simple, but still requires significant training in data entry and tool management. There are still challenges in recording using the Excel system developed. This is improving, but there was minimal experience with Advanced Excel programming and if not well monitored this may impact the consistency and utility of this tool. If not monitored and managed carefully this tool will not be effectively adopted within the two pilot courts despite the political will and interest in particular partner stakeholders. SELECTED RECOMMENDATIONS Local Governance Strengthening Recommendation: Provide more training for suco councils in local governance planning and in prioritizing and sharing information on the impact of decentralization within their communities.  Strengthen project efforts working on suco level planning and prioritization of recommendations. When the draft Suco Law passes and the draft Local Power and Administrative Decentralization Law is implemented, there will be distinct changes in local government coordination with suco councils and there will be a need for this skill transfer. This will be particularly relevant in the areas of participatory planning, project monitoring, and management. Recommendation: Strengthen the relationship with DNAAS to ensure national ownership and increased GoTL cooperation.  Ba Distrito Component Managers and Municipal Coordinators need to coordinate more effectively with the MSA staff particularly, the National Directorate for Support and Suco Administration, the Department of Capacity for Community Leader, and the Department of Technical Cooperation for Sucos. These efforts will improve this relationship and increase the likelihood of adoption of training module approaches and methods currently used in the project. Decentralization and Input of Local Institutions 11 Recommendation: Support decentralization through innovation.  Ba Distrito should look at innovations that may assist in more productive decentralization implementation and planning and integrate this into future efforts that support decentralization. The following actions are suggested: o Review the decentralization pathways that are being proposed within the GoTL such as within the Vice Ministers Office in the MSA and how these ideas can be leveraged and supported. o Consider the Suco Expo as a good example of innovation. More efforts like this are needed, although the scale can vary. This effort was well received by GoTL counterparts, but sustainability and ownership of future events need to be shared within the GoTL, particularly the MSA and ensure there is an annual budget discussion as to how to sustain such forums.  USAID should consider analyzing the impact of these pending draft laws on local governance and the role that these laws will play on citizen engagement in local planning. o Municipal forums, suco exchanges and suco expos have been appreciated by GoTL staff, particularly in MSA, but the ability of line ministry staff to respond to citizen needs are limited. o In multiple GoTL interviews, more services were requested to improve local planning and accountability processes within MSA at the municipal level. Legal Aid Organization Sustainability Recommendation: Increase technical capacity building efforts for legal aid partners and increase organizational monitoring of project agreement activities.  Increase technical assistance in report writing, case analysis, proposal development, and financial management and planning for legal aid partners in Oecusse, Covalima, and Baucau.  In the future, USAID should consider combining these two component areas into a separate project that focuses on access to justice for marginalized groups. o This has been done in the past and the small budget allocated for these component activities and the scope and scale of the targeted implementation area made this challenging for Ba Distrito. o Areas of focus should be on supporting more formalized resolutions and legislation for customary law and increasing mobile clinics and legal aid information provided by the GoTL. Other technical area USAID should consider focusing on are citizen advocacy for human rights and access to information. o More effective support for legislation that supports legal aid organizations is also an area that should be continued as these organizations are critical for marginalized and impoverished populations seeking recourse in the judicial system. District Court Functionality Recommendation: Monitor the utilization of the statistical reporting tool and staff skills in Excel management.  Ba Distrito should provide more oversight both in Oecusse and Baucau for statistical reporting improvements and tool adoption.  Provide additional training for court staff in ensuring that information is entered consistently. A Data Quality Assurance (DQA) process done by Ba Distrito staff or technical advisors should be conducted in Oecusse and Baucau. Recommendation: Improve court performance through targeted training and support for, advocacy, research and improvements in establishing sentencing guidelines. 12  Ba Distrito should provide a research and advocacy grant to develop sentencing guidelines for cases of domestic violence to help judges determine the appropriate penalty. These guidelines could address specific inconsistencies in the Penal Code and the Law Against Domestic Violence (LADV).  Ba Distrito should provide a research grant to identify alternative types of sentencing and court oversight for suspended sentencing used in lieu of prison time.  Future USAID efforts should focus on sentencing guidelines and training for judges on specific issues relevant to marginalized groups such as land and property rights, SGBV, and citizenship and human rights issues. Recommendations: Utilization of Household Survey and Data Recommendation: Use baseline and midline data to assess areas for increased or adjusted intervention by the project.  The Ba Distrito M&E team should consider updating end line targets for high-level indicators.  Use variances from baseline to midline to inform staff of specific changes in knowledge and behavior and to assist in more accurate and informed activity level planning in the specific objective areas with downward trends. 13 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE AND INTENDED USE The purpose of this TO was to conduct a mid-term evaluation of the USAID Ba Distrito Program implemented by Counterpart International. The task order also included conducting a midline of the household survey in the original 3 project municipalities and a baseline for two additional municipalities added to the project in 2015. The evaluation team (ET) looked at four component areas of the Ba Distrito program: (1) local governance strengthening, (2) decentralization and input of local institutions, (3) legal aid organization sustainability, and (4) district court functionality. A fifth component was added to the program for electoral support, but this is in the early stages of implementation and was not included in the scope of this evaluation. The purpose of the mid-term evaluation is primarily to evaluate progress of the activity against the baseline survey conducted in 2014; to assist in conducting the same baseline survey for the two municipalities that were added to the activity in November 2014; and to identify challenges faced and present recommendations for addressing them to USAID and the Ba Distrito team. The primary audience for this evaluation is the USAID Mission in Timor-Leste. The evaluation may also be shared with other donors and other USAID offices to inform the development of future efforts in other countries. This evaluation will inform the Mission about the activity’s performance and achievements, and will be used by the Mission to make programmatic adjustments to the project, if needed. EVALUATION QUESTIONS Following an initial desk review of relevant and available documentation provided by USAID, as well as a call with the client to clarify the evaluation purpose and intended users, the ET discussed how to reorganize, combine, and refocus the questions included in the evaluation Scope of Work (SOW), with a particular emphasis on producing a list of sub-questions that the ET felt confident they could answer definitively and completely given the resources, time, and design of the evaluation. Based on fieldwork and discussions with USAID and key stakeholders of Ba Distrito, question four as stated in the original SOW was reformulated to more effectively address some of the gender related aspects of the project. Originally, this question sought to assess project attribution to increased levels of awareness regarding Gender Based Violence (GBV). Based on discussions, desk research and field work, it was established that a more useful evaluation question for USAID and the Ba Distrito team would be the following: How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? This question revision was agreed to by the USAID Contracting Officer Representative (COR) as well as Ba Distrito stakeholders and has now replaced question four in the SOW for this task order (TO). The list of the final evaluation questions follows below (see Table 1); and a more detailed breakdown of the evaluation guide and data sources for answering these questions can be found in Annex V: Evaluation Questions Matrix. Table 1: Final Evaluation Questions Challenges and Lessons Learned Question 1 What are the main challenges faced by the program and lessons-learned to-date? Sub Question 1.1 How was the original program design appropriate/not-appropriate for addressing the program’s goals and objectives? Sub Question 1.2 How were program activities and implementation influenced by the baseline results and data collected? Sub Question 1.3 If so, which activities? Please provide specific examples and geographic focus areas. Program Implementation 14 Question 2 What changes, if any, are proposed regarding implementation? Sub Question 2.1 What changes, if any, seem necessary at this point in the implementation of the Ba Distrito program? Sub Question 2.2 If so, which activities? Please provide specific examples. Sub Question 2.3 Are these changes specific to a particular municipality area? If so, why? Please provide specific examples. Unintended Consequences Question 3 What, if any, unintended consequences have occurred as a result of the program? Sub Question 3.1 Have staff or beneficiaries noted any particular “unintended” consequences as a result of Ba Distrito activities? Sub Question 3.2 How have these “unintended consequences” impacted particular component areas? Which ones? Please provide examples. Sub Question 3.3 How have these “unintended consequences” impacted particular target groups? Which groups? Please provide examples. Sub Question 3.4 Are there specific component areas that seem to have had greater impact than others? Which areas? Please provide specific examples. Gender Sensitivity Question 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? Sub Question 4.1 What specific Ba Distrito activities have demonstrated this? Sub Question 4.2 How has this been monitored at the project level? Sub Question 4.3 What Ba Distrito activities have been most effective in impacting this? PROJECT BACKGROUND In September 2013, USAID awarded a four-year program called Ba Distrito (“To the Districts”) to Counterpart International. The goal of the activity is to increase institutional and human capacity at local levels to deliver basic services effectively and in a manner that is responsive to citizen needs and expectations. The activity is being implemented in 100 sucos in four municipalities: Baucau, Covalima, Ermera, and Liquica as well as in the Special Administrative Region of Oecusse-Ambeno (see Figure 1 for a map of Timor-Leste). Initially, the activity had four components, with a Total Estimated Cost (TEC) of $6 million. In April 2015, the Cooperative Agreement (AID-486-A-13-00007) was modified to include an additional component to support anticipated suco elections and the TEC was raised to $7 million. The main beneficiaries of the activity are members of suco councils, legal aid organizations, and court actors (i.e. judges and judicial officers). Ba Distrito’s goal is to increase institutional and human capacity at local levels to deliver basic services effectively and in a manner that is responsive to citizen needs and expectations. The project activities are structured along four components that contribute to the wider vision of improved decentralized governance and inclusive access to justice. The program’s strategy of putting people first and focusing on all citizens, local governance structures, and representative Civil Society Organizations (CSOs) is designed to ensure the implementation of democratic reforms that are relevant and will be owned by the citizens they serve. Ba Distrito’s focus on strengthening human and institutional capacity in local governance and rule of law seeks to ensure that the gains Timor-Leste has made in peace, security, economic growth, and human development in the years since its independence will be bolstered and further enhanced. The program employs a whole-of-government approach (working to strengthen vertical and horizontal linkages 15 of multiple public agencies) applied to local government performance and access to justice, with integrated project components and objectives. There is ongoing tension between decentralization needs and inclusion in Timor-Leste that needs to be addressed within the program-planning framework as well as in Ba Distrito steam management. Doing this formally and in program planning will enhance lessons learned from the Ba Distrito program. Decentralization is dynamic, but is enshrined in the Constitution of Timor-Leste. The specific history and relatively nascent nature of centralized governance in Timor makes decentralization planning still a part of post conflict consolidation. There is a high need for inclusionary practices that are specifically responsive to the pathway of decentralization and deconcentration planning in Timor-Leste. The project has met these challenges by seeking to create forums for advocacy and inclusion in the planning for decentralization and in justice improvements and practice. In 2014, Counterpart International hired a contractor, Social Science Dimensions (SSD), to carry out a baseline survey for the activity. The survey was undertaken in 2014 in 22 sucos throughout two targeted municipalities (Baucau and Covalima) and the Special Administrative Region of Oecusse-Ambeno. The survey instrument comprised 25 questions divided into the four themes of (1) “Citizens’ understanding of their role in political processes,” (2) “Citizens’ knowledge and awareness of decentralization in Timor￾Leste (including suco legislation),” (3) “Citizens’ knowledge of the roles and responsibilities of sucos, and their expectation of and satisfaction with suco service provision,” and (4) “Citizens’ access to justice.” A fifth, crosscutting theme concerning “Citizens’ perception, understanding and agreement with concepts of gender equality and the participation of women, youth and minorities in decision making around development priorities” was integrated into the survey design process. The survey results were submitted to USAID in September 2014. Data collected from the survey served as a basis for developing and adapting Ba Distrito interventions to ensure responsiveness to citizen needs and expectations, as well as provide baseline data to assist with the monitoring and evaluation (M&E) of the Ba Distrito Program over the term of its implementation. EVALUATION METHODS AND LIMITATIONS The evaluation team used a mixed methods approach, including qualitative analysis and statistical analysis of household survey data collected from SSD both at the midline and the baseline. The evaluation consisted of three phases: 1) a comprehensive desk study of all relevant documents; 2) approximately four weeks of field data collection in Timor-Leste, involving quantitative and qualitative data collection with SSD and Social Impact (SI); and 3) analysis of all data collected, report writing, and final presentations. DATA COLLECTION METHODS Quantitative Fieldwork Data Collection Methods The ET used a household survey developed in 2014 (and slightly adjusted in 2016) for quantitative data collection. This survey was conducted in a total of five municipalities (Baucau, Covalima, Ermera, Liquica, and Oecusse), as opposed to the three districts1 (Baucau, Covalima, and Oecusse) included in the original 2014 survey. Note however, that whereas the survey methodology (described below) was common across all five municipalities, the 2016 household survey represented a Baseline Household Survey for the municipalities of Ermera and Liquica, but a Mid-term Household Survey for the municipalities of Baucau, Covalima, and Oecusse. 1 In 2014, the term ‘district’ was still in use in Timor-Leste, hence this term is used for past references. 16 The objective of the survey was to interview, in total, 1,540 randomly selected respondents from 35 sucos. At the suco level, this meant 11 individuals from each of the target groups (Female 16-30; Female 31+; Male 16-30; Male 31+), giving a total of 44 individuals per suco and a total of 1,540 respondents over the entire 35 sucos. The survey was administered in the targeted municipal areas from March 30 to June 10. Twelve national enumerators were trained to conduct the survey, supervised by two national district field supervisors and two expatriate specialists (one on-site survey director and one remote computer engineer/database specialist). The midline survey included 29 key questions in addition to a number of demographic questions such as age, education, and occupation. Household Survey Data Utility The primary data from the household surveys that was used for this report to assist USAID and the project team to measure change were survey questions related to indicators identified in the Ba Distrito monitoring and evaluation framework. The Ba Distrito Performance Monitoring and Evaluation Plan (PMEP) utilized specific household survey questions to measure change over the life of the project. These results indicators (referred to in this report as objectives) and their measures were drawn directly from the baseline measurements conducted in September 2014, and the updated household survey conducted was used to measure midline results for the project. Only one baseline question was identified by the project for midline measurement, although the SI team calculated all midline measures as reference for the Ba Distrito and USAID team. These PMEP measures were used as points of analysis and integration for the findings, conclusions, and recommendations section of this report based on the qualitative data collection. Additional baseline information for the two additional municipalities and disaggregated data by household survey question are also presented in Annex VII. Data Analysis At the conclusion of the survey fieldwork by SSD, cleaned data sets including respondent identification and survey data was provided to SI, along with a Data Quality Report outlining challenges that occurred and mitigating actions taken. Based on analysis of the data collection approach and data analysis methods used during the baseline and the midline with SSD, SI determined that a nearly identical data analysis approach could be implemented. Exceptions are noted were relevant and explained in more detail in Annex VII. Annex VII provides a statistical report, including additional details on methodology and responses by survey question. Data cleaning and analysis was performed in Stata Version 12.0 using Stata’s built in survey analysis commands. Data from baseline and midline from all available municipalities was cleaned and standardized for analysis in a single database. Survey weights were calculated using the approach established at baseline (see Ba Distrito Baseline Survey Report for details).2 Weighting allows for population-level estimates across the 35 sampled sucos. At midline, the sampling strategy to sample households with replacement within a suco led to multiple respondents from a single household (or 200 meter area if the target respondent was unavailable) being interviewed. To account for the additional homogeneity (similarity) of responses from respondents living in the same household (or 200 meter area), the analysis treated household as the primary cluster and suco as the stratifying variable. This issue did not arise at baseline since the sampling design did not lead to multiple respondents being interviewed per household. In order to analyze the data in a consistent way, the SI team recalculated baseline values assuming each household respondent represents a unique household and using suco as a stratifying variable. This approach does not change the point estimates reported in the result, but would impact standard errors and confidence intervals (not shown). 2 SI determined that the baseline values needed to be recalculated based on identified discrepancies in the weights. SI corrected these discrepancies for the baseline dataset, which resulted in very small variances, and should not be statistically relevant for project measurement purposes. 17 Using the survey weights and accounting for stratification at the suco and clustering at the household, the PMEP indicators were calculated by reducing categories and using cross-tabulations (example –response choices of Effective and Very Effective were combined). Response choices of “don’t know” or “no answer” were included in the denominator, while missing data was excluded. This process was repeated first for the three municipalities participating in baseline and midline and then for the two new municipalities, as detailed in Annex VII. Additional cross-tabulations were conducted for the demographic variables and each survey question to understand the range of responses to each question. For the three municipalities participating in both baseline and midline, the survey questions were disaggregated first by district and then survey instrument (baseline or midline). A chi-squared test was used to test for differences between categorical variables. For questions with multiple components, the cross-tabulations were broken down by sub-question. For the two new municipalities participating in the survey, the results were tabulated separately using the same methodology, but without using the survey instrument as a variable. Annex VII includes the detailed responses for each survey question. At midline, a multivariate regression model was built for each question to explore how the responses varied as a function of district, gender, age, and occupation. Additionally, the time period of the survey instrument (baseline or midline) was included as a covariate to understand how responses varied by time period. At midline, standard multivariate regression models using Stata svy commands were used to explore changes in outcomes while controlling for potential confounding covariates and incorporating complex survey design elements (weight, clustering, and stratification). At baseline, mixed effects models were used. While the ET considered both approaches at midline, standard regression models were chosen for analysis for midline data. Annex VII includes further explanation regarding the use of standard multivariate regression models. Similar to baseline, the type of regression model used was determined by the distribution of the outcome variable. Logistic regression was used for questions with binary responses (e.g., yes/no) and linear regression was used for questions with multiple sub-parts (1a, 1b, etc). However, the midline approach for questions with varying level of response (e.g. strongly agree, agree, disagree) used ordinal logistic regression as opposed to converting the response choices to a numerical rank and using linear regression model. Ordinal logistic regression does not assume the distance between response categories is equal (Long & Freeze, 2012). Annex VII includes the tables reporting results of the multivariate models. Qualitative Fieldwork Data Collection Methods Primary methods of qualitative data collection were (1) document review and analysis of USAID-provided documents as well as secondary data sources such as relevant assessments, Ba Distrito program documents, and other donor studies relevant to the Ba Distrito program component areas (as detailed above); (2) 49 key informant interviews (KIIs) to explore key issues in-depth with individual stakeholders using open-ended questions; (3) two small group discussions (SGDs) when and where personal opinions were critical to research; and (4) six focus groups discussions (FGDs) with targeted suco council members to discuss open-ended questions intended to generate data on broad or consensus opinions relating to Ba Distrito component area impact at the local (suco or district) level. A comprehensive initial list of respondents can be found in Annex V: Key Contacts and interview protocols and templates in Annex II. The ET of three (the Team Leader and two Democracy and Governance specialists) traveled to two municipalities (Baucau and Covalima) and the Special Administrative Region of Oecusse with the support of an interpreter and a logistician. Data Collection Instruments 18 As there are several component areas of Ba Distrito, each component had several types of stakeholders and each tool was tailored to ensure that data collection methods were aligned with the anticipated role of the key informant. An Evaluation Interview Guide was used to guide all field based data collection and included the approved household survey as well as all data collection tools for the qualitative and quantitative fieldwork. This guide ensured consistency across the ET. Gender Integration Gender integration was a critical part of the design, implementation, and analysis of this evaluation. Both sexes are included in the sampling design and data collected was disaggregated by sex where possible. Gender sensitive data collection methods and tools were applied with the confidentiality and informed consent statement included in the data collection tool introduction. Tools were designed to address gender data gaps anticipated that are specific to the Timor-Leste context such as gender issues relevant to access to justice, GBV, and local governance and decision-making. The process of data collection also took into account the time and location for interviews and surveys, ensuring their accessibility by both women and men. Secondary gender specific data was analyzed and triangulated in KIIs, SGDs and in FGDs to ensure gender issues were addressed and recommendations were grounded from evidence-based findings. Specifically relevant to the Timor-Leste context is careful consideration of key respondents with knowledge of gender mainstreaming needs at the suco and municipal level in Timor-Leste were assessed. The ET also identified in each district the appropriate gender focal points for the Secretary of State for the Support and Socio-Economical Promotion of Women as well as appropriate women’s organizations at the municipal and local level the ET felt may have relevant insight of the evaluation questions. Data Analysis In the data analysis phase, the ET analyzed and assessed qualitative data, and also integrated the household data (as detailed above) to triangulate qualitative data collected, and provide additional information for USAID and the Ba Distrito team on progress and challenges of the project. The ET used parallel analysis to examine the evidence from its desk review, KIIs, SGDs, FGDs and the household survey. In this “methods triangulation” analytical approach, the ET analyzed data related to an evaluation question, and relevant indicators, obtained using different methods in parallel, and then across the data collection methods as applicable. Through this analysis, the ET determined if Ba Distrito Program activities contributed to particular outcomes or trends identified, as inferred through “plausible contribution” 3 analysis. This final report presents qualitative and quantitative findings from the evaluation fieldwork to address identified evaluation questions. Qualitative data collected from KIIs, SGDs, and FGDs was integrated with quantitative findings to elicit information on changes based on sampled populations. These findings were used to form the conclusions and final recommendations, and were triangulated with information from the desk study research. EVALUATION LIMITATIONS Complex post-conflict environments can be challenging, so the ET designed the evaluation approach and management plan to mitigate problems to the extent possible. Examples of mitigation strategies included utilization of multiple data points for each question, confirmation of data through triangulation and document review, customized data collection protocols, and staggered data collection timeframes to provide additional time to locate beneficiaries and other stakeholders who were no longer affiliated with the Ba Distrito Program. 3 Contribution analysis is an approach for assessing causal questions and inferring causality in program evaluations. 19 Anticipating the Timor-Leste context, the team identified several important challenges that the evaluation approach needed to address:  Identifying contributions to Ba Distrito goals and component objectives derived from relatively low-intensity, short-term, and issue-specific interventions. Quantitative data collection is from a household survey designed as pre, mid, and post without a comparison group and involving indirect beneficiaries meaning that attribution or impact cannot be identified statistically. The qualitative data collection relied on perceptions of direct beneficiaries regarding intervention contribution to high-level goals as well as analysis of plausibility, given the lack of a comparison group. The ET mitigated this challenge by following the analysis plan detailed above, using household survey results to triangulate data from the qualitative fieldwork.  Anticipating and analyzing potential response bias of respondents. The ET worked with USAID and other stakeholders to identify potential respondents with varying programmatic experiences, both positive and negative, and relied on some respondents to provide links to other knowledgeable actors. While the ET provided information to all respondents regarding the purpose of the evaluation, highlighting their role as external evaluators and the utility of honest responses, respondents were not selected randomly for the qualitative data collection and therefore their views may not be representative of the broader program or Timor-Leste population.  Identifying potential outcome information from activities partially completed. The ET relied on triangulation of data and information from experts to assess the reasonableness of impact assumptions. It was challenging to attribute outcome as this was a mid-term evaluation and outcome-level indicators were not tracked. The baseline data and the household survey results helped to strengthen analysis in some critical areas of measurement for the project.  Identifying particular political or governance constraints unique to the Timor-Leste context that may have resulted from changes in government structure and organization, or legislative delays. The ET worked to establish rapport with key informants to enable honest and open responses, but the sensitive nature of the work and environment may have inhibited fully candid answers. This was particularly relevant to legislation pending regarding decentralization and local power laws, as some were in varying states of approval in parliament and were not public documents yet promulgated.  Changes in knowledge, attitudes, and behaviors (KAB) explicated by the ET may result in lower confidence in the stated change based on the small size of sampling conducted. The ET conducted several SGDs with Government of Timor-Leste (GoTL) staff and sought examples of KAB factors from key informants (KIs) interviewed and triangulated this data with FGDs held with suco council members. However, this was a small sample size and there was limited secondary data to support findings.  Lower representativeness of disabled persons within sample sucos at the midline. The number of people living with a disability was lower than anticipated at the midline. The ET had less success meeting with suco chiefs to identify disabled citizens at the midline, which could have led to this challenge. 20 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS There were two primary goals of the Ba Distrito project each goal organized around two component areas. One goal is focused on support for decentralization while the other supports access to justice for marginalized groups and improvements in district court functionality. The primary beneficiary groups were specific GoTL justice sector members, suco council members, and staff of partner legal aid organizations. National NGO partners through competitive small grants awards implemented some Ba Distrito activities. The project also had Memorandums of Understanding (MOUs) and agreements with several GoTL institutional partners. Table 3 details the Ba Distrito program and provides information about Ba Distrito goals and component objectives. Where relevant, the national non-governmental organizations (NGO) and institutional partner stakeholders are tasked with the implementation of select program activities under specific component areas were also listed. The two goals of Ba Distrito were distinct and in many ways they could have been the goals for two separate development program interventions. For future project planning USAID may want to consider making these goal areas into two separate projects, one focused on support for decentralization and the other looking at increasing access to justice for marginalized populations. The context requires separate resources, strategies, and staff approaches. This impacted Ba Distrito and while they did an exceptional job managing these technical disconnects, this may be something USAID may want to consider in the future. This is notable as there were some inherent tensions between resources and program planning that was evident from fieldwork, program documentation, and analysis of interventions. Where this is relevant these distinctions have been made in the findings and conclusions section of the report. This section synthesizes the findings, conclusions, and recommendations (FCRs) for each component area of the Ba Distrito Project. To assist in clarification and utility of the report each project component area was analyzed for the following: 1) Challenges and Lessons Learned: defined as, contextual impediments impacting project goals, objectives and activity planning and implementation and lessons learned from addressing these challenges; 2) Project Implementation: defined as, direct results and outcomes plausibly attributed to component and project activities; (3) Unintended Outcomes: defined as, unintended impacts, outcomes, or spillover effects as a result of project implementation; and (4) Gender Sensitivity: defined as, project practices, tools, and methods demonstrating gender awareness in design, implementation, and management. These areas directly correlate to the core evaluation questions and explicate specific FCRs aligned to these questions as specified in the task order SOW (see chart below). Table 2: Findings, Conclusions, and Recommendations Challenges and Lessons Learned Question 1 What are the main challenges faced by the program and lessons-learned to-date? FCRs are designed to explicate contextual impediments impacting project goals, objectives and activity planning and implementation as well as lessons learned from addressing these challenges. Program Implementation Question 2 What changes, if any, are proposed regarding implementation? FCRs are designed to explicate direct results and outcomes plausibly attributed to component and project activities. Unintended Consequences Question 3 What, if any, unintended consequences have occurred as a result of the program? FCRs are designed to explicate unintended impacts, outcomes, or spillover effects as a result of project implementation. Gender Sensitivity 21 Question 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? FCRs are designed to explicate project practices, tools, and methods demonstrating gender awareness in design, implementation and management. A final section, Household Survey Data, addresses relevant issues specific to the household/ baseline survey and qualitative data collection. This section is meant to explain in narrative the analysis of the household survey and to also allow greater utilization of this information for future program planning. These evidence based findings, conclusions and recommendations are also presented in Annex VII where detailed quantitative findings and the analytical tools used can be assessed. Table 3: Ba Distrito Objective and Activity Areas Goal 1: Improved Decentralized Governance Component A: Local Governance Strengthening Objective: Enhanced capacity of sucos to strengthen citizen participation and representation in local governance. Implementing and Institutional Partners: Suco Council Member (i.e. Women reps, youth reps and suco chiefs), MSA, and Belun. Component B: Decentralization and Input of Local Institutions Objective: Improved communication and linkages of sucos with municipal/regional administrations, local GOTL line ministries and other providers of basic public services at the sub-national level. Implementing and Institutional Partners: MSA, Parliament, and Suco Council Members (i.e. Women reps, youth reps and suco chiefs). Component C: Legal Aid Organization Sustainability Objective: Strengthened local justice sector institutions that increase access to formal and informal justice for marginalized citizens and the poor. Implementing and Institutional Partners: Fundacão Fatuk Sinae Oecusse (FFSO), JSMP, Baucau Justice and Peace Commission, and Liberta Component D: District Court Functionality Objective: Strengthened local justice sector institutions that increase access to formal and informal justice for marginalized citizens and the poor. Implementing and Institutional Partners: National Court of Appeals, PACT with Oecusse and Baucau district courts and the Legal Training Centre. KEY FINDINGS: LOCAL GOVERNANCE STRENGTHENING Goal 1: Improved Decentralized Governance Component A: Local Governance Strengthening Objective: Enhanced capacity of sucos to strengthen citizen participation and representation in local governance. Implementing and Institutional Partners: Suco Council Member (i.e. Women reps, youth reps and suco chiefs), MSA, and Belun. Component Overview: The goal of Component A is to strengthen local governance by enhancing the capacity of suco councils to strengthen citizen participation and representation in local governance. This component also conducts activities to improve communication and linkages of suco councils with municipal administration, local Government of Timor-Leste line ministries and other providers of basic public services at the sub-national level. To achieve this, the project provides a grant to an implementing partner Belun and engages in activities that include (1) evaluating the capacity development improvements of all 100 suco councils; (2) finalizing a series of 10 training modules for suco councils; (3) delivering formal trainings to each suco council followed by one-on-one technical support; (4) awarding constituent engagement grants to promote engagement between suco councils and their communities; (5) facilitating communication between sucos by holding suco exchange visits one suco expos; and (6) improving collaboration between sucos and the municipal level by coordinating suco/municipal forums. 22 Challenges and Lessons Learned Finding 1: Suco councils are not formally a part of government, but they are critical for current decentralization planning and implementation in Timor-Leste. Based on desk review and fieldwork the project has specifically targeted suco council members. This is the primary beneficiary of the project and also the primary target of specific state responsibilities within the MSA. According to staff interviews and interviews with government officials working in the MSA this target was appropriate, but there were challenges in coordinating some training services between Ba Distrito and the National Directorate for Support and Suco Administration (DNAAS). Finding 2: Decentralization planning is political in Timor-Leste and suco councils play a critical role in extending the reach of the State in rural areas. According to suco council members and representatives sources interviewed in Baucau, Covalima, and Oecusse, most suco council members (74%) would prefer to remain outside of government to ensure traditional practices and roles of the suco are protected. This was also supported in interviews with the majority of Municipal Administrators and senior GoTL staff (85%). Most government staff viewed suco councils as similar to “neighborhood associations” and their roles and responsibilities were described as promoting community stability and not to engage in advocacy, particularly relating to political issues. In interviews in Baucau this was particularly emphasized, but was also supported in similar discussions in Oecusse. Finding 3: Recent proposed changes in local electoral processes will impact the way suco councils are formed (i.e. suco elections). These “proposed” changes have already caused tension between the State and local leadership (primarily suco chiefs) according to interviews with Ba Distrito governance staff, GoTL municipal staff and FGDs with suco council members in Baucau. Based on fieldwork this impacts suco chiefs and aldeia chiefs most acutely. Based on interviews, further delays in suco elections4 are anticipated and based on FGDs with suco council members this has caused confusion as well as tension at the local level as to how long their tenures will continue. Finding 4: One of the major challenges is that the coordination and communication between suco councils and municipal ministry line staff is variable across the targeted municipalities and sucos targeted by Ba Distrito. Project interventions such as municipal forums, suco exchanges and suco expos have helped to create more space for suco council and municipal and national GoTL staff to engage, but GoTL staff interviewed see sucos as informal community associations and there are currently limited resources or incentives for line ministry staff to respond to citizen requests. Program Implementation Finding 5: Suco council members are critical partners for several projects and activities at the local and national level. Specific suco council members (i.e. suco chiefs, women representatives and youth representatives) are the core focus of Ba Distrito training and implementation activities in this component area. According to desk research and interviews with Ba Distrito staff, INGO and national NGO partners there is virtually no formal coordination of suco training efforts either by the GoTL, INGOs or by donors. This was noted in interviews as impacting implementation as weak coordination at the local and municipal level hampers activity level planning and can cause stress within suco council structures. Finding 6: In interviews in Oecusse, suco council members had trouble remembering all the trainings they had been asked to participate in. This was also consistent in FGDs held in Baucau and Covalima. There was evidence that local NGOs utilized as trainers for Ba Distrito under the small grants program do not adequately sensitize community participants to the objectives and intent of the Ba Distrito project. In 4 Suco elections were originally scheduled for October 2015 but there has been no defined timeline for suco elections until key draft legislation has been promulgated. 23 several interviews with suco council members they had never heard of Ba Distrito, but instead were familiar with the NGO implementing partner (i.e. Belun or FFSO). Finding 7: Currently some advocacy activities, particularly public consultations on some of the draft laws implemented by Ba Distrito, confuse some GoTL partners and sources interviewed want to be better informed at the municipal and national level when these events are planned. Some GoTL staff interviewed said that although they were given letters of invitation, they were not involved in discussions on planning these events. Specifically, senior level municipal staff want to be more engaged when recommendations are being formed for draft or pending legislation specific to local power and decentralization. This concern was more acute in Baucau but similar recommendations were requested from GoTL staff in Covalima and in interviews with the MSA staff in Dili. Finding 8: Suco municipal forums are planned to allow line ministries to communicate and hear concerns from suco council members regarding service coordination. Based on interviews conducted with GoTL staff, these events are appreciated, but they are not always able to respond to citizen requests based on annual budgets and limited planning resources. Based on interviews from senior staff in MSA, once decentralization becomes implemented at the local level, planning and accountability forums will be easier for the GoTL to manage, as resources will be aligned with the new legislation. Finding 9: Based on project reports from 2014, the project planned and implemented public consultations jointly with the former administration of MSA on the first draft suco law in 2014. This was done jointly by MSA and the project in three municipalities so in the past there was strong collaboration in support of MSAs goals. The administration current administration, however, has changed as has some of the emphasis on decentralization and its impact on local forms of governance at the municipal as well as suco level. Finding 10: The Access to Justice and Women’s Rights module training provided by Ba Distrito for suco council members according to some interviews was too long in length and some suco council members were not able to attend the entire training as the cost for transportation and the distance they needed to travel made attendance difficult. Based on interviews with suco council members who participated in the training the content was good, but some members felt it should have been delivered in two segments. Some members were not able to attend the entire training due to logistical issues that they felt should have been better addressed by Ba Distrito staff and partners responsible for the training logistics and planning. Finding 11: Ministry staff specified in interviews that they have specific training protocols, manuals, and plans that need to be better integrated into Ba Distrito efforts annually. National Directorates are responsible for Annual Action Planning (AAP). Chiefs of Departments are only required to submit quarterly activities and these are integrated into the AAP for the Directorate and then combined for the Ministry’s AAP. Department Chiefs interviewed wanted more input from Ba Distrito on training activity improvements so that they could more effectively incorporate Ba Distrito training ideas into the Directorate planning and budgeting process. Ministry staff requested more discussion between Ba Distrito and DNAAS staff in integrating specific content from the Ba Distrito training modules into future training efforts and planning of the National Directorate. Unintended Outcomes Finding 12: In some trainings, participants come from other sucos or areas that are not targeted by the Ba Distrito project. This has happened with the suco expo event and in some legal aid training events. This is a sign that information and interventions that the project is implementing are appreciated outside of the project target areas. This is a positive unintended outcome that deserves to be noted. 24 Finding 13: An unintended outcome was that Ba Distrito staff interviewed felt coordination was strong between the project and the MSA, but staff within the Ministry felt that this could be improved, particularly at the Directorate and Department level within DNAAS. Recent GoTL staff changes as well as small budgets for Directorate level staff to observe field based trainings may also have contributed to these perceptions. According to interviews with GoTL staff working with the MSA in Baucau and in Dili coordination needs to be more formalized and regular meetings established to exchange ideas between the project and DNAAS staff. Gender Sensitivity Finding 14: It was evident in interviews, document review and in FGDs held with select suco council members that Ba Distrito staff, policies, and practices have sought to integrate gender sensitive approaches into the implementation and management of this component area. Of particular note in interviews and program documentation review was the underlying and consistent utilization of language, methods, and monitoring of gender impact on project activities. Of particular note was the targets set for activities in this component area both in the PMEP and in the approach and methodology used for the baseline and household survey. By specifically targeting women suco council representatives and youth within the project there was a consistent and practical flow of information and feedback from these beneficiary groups that was used to better manage and adapt project activities. Finding 15: In interviews with Ba Distrito staff, there was evidence that project management and policies relevant to gender sensitivity were in place and well sensitized across all component areas. This component area, according to interviews conducted with Ba Distrito staff acted as the “core” and most activities were coordinated through this component area manager. This ensured that coordination across the components targeted the identified beneficiary groups and also allowed for monitoring of gender representation across component activities. This process according to staff and Ba Distrito NGO partners was cited as being particularly effective. KEY CONCLUSIONS: LOCAL GOVERNANCE STRENGTHENING Challenges and Lessons Learned Suco Councils are not formally a part of government and Ba Distrito staff and partners recognize this, but suco council members are increasingly targeted for activities not only by USAID, but also by multiple partners at the local and national level. Suco councils are critical for extending the reach of the government, and for providing critical information for local citizens, many living in remote and distant areas from municipal centers. Ba Distrito’s target of suco council members was appropriate, but one of the challenges the project faced was in ensuring that the work that they do with members was well coordinated with the MSA. Ba Distrito has a small staff, as does DNAAS; more formal mechanisms for project coordination would have eliminated some of the inherent tensions that existed between the project and some its activities. Formal and regular coordination would have also made the project objectives clear to DNAAS. There are regular calendars that are maintained by the senior staff for Municipal Administrators and Ba Distrito needs to ensure that their events are a part of this schedule and that they schedule regular formal meetings with cabinet staff at the Municipal level. Although the project in the past has had strong collaboration with MSA, new staff and systems require more targeted support and integration with DNAAS; in particular, were missed opportunities to advocate and collaborate on training services between Ba Distrito and DNAAS that allow the Directorate and Departments to integrate some Ba Distrito training into their annual activities and budgets. The budget of DNAAS is small and more targeted coordination on training modules will extend the reach, utility and eventual adoption of these modules to the Timor-Leste context, particularly within this Ministry. One of the challenges the projects faced is the highly political nature of the current state of decentralization planning in Timor-Leste. Most suco council members have limited information about decentralization and 25 what proposed changes will mean for community level planning, and even GoTL officials are compartmentalized in their tasks and do not always understand how decentralization will impact suco council roles and responsibilities and what support they will need in the future to improve their relationship and coordination at the local level in municipal level planning. Most GoTL officials see suco councils as mechanisms that assist the government in coordination and see a clear dividing line between suco council responsibilities and traditional forms of leadership. Recent proposed changes in local electoral processes as well as local power legislation will impact the way suco councils are formed and perform and this has caused some tension between suco council chiefs, traditional leaders and GoTL officials. This is more acute in municipal areas and sucos where there is a long history of local suco representatives being elected from families with extensive histories of traditional leadership. In Oecusse and Baucau this was acute, but there are other suco and municipal areas where this will continue to cause tension between GoTL suco council electoral procedures and local practices, norms and traditions. The targeted municipalities are quite unique and the project would benefit from identifying the distinctions and challenges across these municipal areas. There is a need to create a project management mechanism where research and advocacy grants can be used to highlight some of these differences and how these may impact future decentralization planning and citizen and government engagement. These proposed changes have already caused tension between the State and local leadership according to interviews with Ba Distrito governance staff, GoTL municipal staff and FGDs with suco council members in Baucau. Based on fieldwork this impacts suco chiefs and aldeia chief most acutely. Based on interviews delays in suco elections are anticipated and based on FGDs with Suco Council members this has caused confusion as well as tension at the local level as to how long their tenures will continue. Program Implementation Suco council members are critical partners for several projects and activities at the local and national level in this component area. One of the strengths of the Ba Distrito design was the emphasis on internal coordination across the component team managers and with the municipal coordinators. The weakness in the project was that formal coordination was not as strong as it could have been. The Ba Distrito staff was small and two municipal coordinators (one from Counterpart and one from Belun) was insufficient for a project of this size and scope. It was clear in interviews with MSA staff and senior GoTL officials at the Municipal level that this had impacted implementation, particularly in the areas of coordination, training and advocacy development. In 2014, the project planned and implemented public consultations jointly with the former administration of MSA on the first draft suco law in 2014. In the early stages of the project, the coordination with MSA was stronger, but current administration has changed as has some of the emphasis on decentralization and its impact on local forms of governance at the municipal as well as suco level. Ba Distrito planned suco municipal forums to allow line ministries to communicate and hear concerns from suco council members regarding service coordination but line Ministry staff are not always able to respond to citizen requests based on annual budgets and limited planning resources. The political will for public accountability forums is likely to improve as legislation passes and decentralization becomes implemented at the local level and national budgets and resources aligned with the new legislation. DNAAS staff, particularly the National Director and Chiefs of Departments want more input from Ba Distrito on training activity improvements so that they can more effectively incorporate Ba Distrito training ideas into the annual Directorate planning and budgeting process. Ministry staff requested more discussion between Ba Distrito and DNAAS staff in integrating specific content from the Ba Distrito training modules into future training efforts and planning of the National Directorate. This is a good sign, 26 but will require knowledge of MSA planning processes and more analysis of the project of Directorate and Department funding realities. Another issue of note is that the small grants program and the partnership with national NGOs was effective in being able to scale the activities in this component area, but the impact was mixed. Often the participants did not know the Ba Distrito project, but only the NGO that provided the training. There was at times a clear disconnect between the project intent and objectives and the knowledge and awareness of suco council members of these elements of Ba Distrito and their role and agency in the program. In Oecusse, Baucau and Covalima participants often had trouble remembering all the training they had undertaken. There were often so many trainings not only by Ba Distrito, but also by other providers that the suco council members were not sure which trainings were affiliated with the program. Perhaps 100 sucos were too many to target, or the implementing partner responsible for the training needed to better track and plan suco council participation to maximize material retention. Unintended Outcomes Current advocacy activities, particularly some of the public consultations on draft legislation relating to suco administration and decentralization were confusing for some of the GoTL staff interviewed. Although they recognized the value in recommendations provided by suco council members, they wanted to be more engaged in the process of soliciting these recommendations and in the local level consultations held. This was acute in Baucau, Covalima, and in Dili with the MSA Staff. There were specific requests for more detailed coordination with DNAAS staff on proposed training for suco council members, particularly suco chiefs and local leaders. More lead time for participation of DNAAS officials in field level events was also requested to improve participation and ownership of Ba Distrito efforts and GoTL programs within this Ministry. The MSA specified that they wanted more coordination on specific training protocols, manuals, and plans that Ba Distrito needed to better integrate into Ministry training efforts. State officials were clear that they valued the content of the training, but wanted more direct engagement in the development, design, and delivery. The unintended outcome of this was that several GoTL staff felt that coordination and communication was weak and that this impacted their ability to engage with some Ba Distrito activities minimizing the potential for adoption and national ownership. Gender Sensitivity Ba Distrito as a project was particularly strong in gender sensitivity and integration. In this particular component, there was clear evidence that staff were well trained and had integrated thoughtful and effective targeting practices into their activities to illicit more active engagement of women, youth and the elderly. Of particular note was the underlying and consistent utilization of language, methods, and monitoring of gender impact on project activities, particularly in this component area. By specifically targeting women suco council representatives and youth there was a consistent and practical flow of information and feedback from these beneficiary groups that was used to better manage and adapt project activities and this was evident in staff practices and in documentation review. KEY FINDINGS: DECENTRALIZATION AND INPUT OF LOCAL INSTITUTIONS Goal 1: Improved Decentralized Governance Component B: Decentralization and Input of Local Institutions Objective: Improved communication and linkages of sucos with municipal/regional administrations, local GOTL line ministries and other providers of basic public services at the sub-national level. Implementing and Institutional Partners: MSA, Parliament, and Suco Council Members (i.e. Women reps, youth reps and suco chiefs). 27 Component Overview: The goal of Component B is to strengthen local governance through contributing to improved communication and linkages between suco councils and municipal administrations, local GoTL line ministries, and other providers of basic services at the sub-national level. In particular, Component B works to address the weak tradition of citizen advocacy and participation in the legislative process and aims to mitigate this challenge by supporting a variety of academic, professional, and other non￾governmental entities to understand and comment on the decentralization strategy and corresponding laws and implementation. This is done through (1) soliciting and presenting research on citizens’ opinions on decentralization policy and practice; (2) providing legal advice and legislative drafting support if required; (3) designing and awarding decentralization research and advocacy grants; and (4) supporting information dissemination about approved laws. Challenges and Lessons Learned Finding 1: There are not yet decentralized local government structures in Timor-Leste, so local government presence varies at the municipal level based on ministerial resources (i.e. budgets and staff) available. This is the result of not having a decentralization law and local governance structure they can provide for local planning and prioritization across sectors and in close collaboration with citizen structures such as suco councils. This is a product of the legacy of decentralization planning in Timor￾Leste post conflict. This condition proved to be challenging for Ba Distrito staff, particularly municipal coordinators, who struggled to keep local ministerial staff informed about the program and to identify the correct line ministry staff and stakeholders to engage in municipal consultations and events. The small number of Ba Distrito field staff made scaling up with local NGOs and conducting research a strategic and necessary decision, but staff resources were low for a project of this size and volume of activity. Finding 2: One of the challenges of the program is the dynamic nature of decentralization legislation in Timor-Leste (TL) and how this has impacted the timing and planning of specific activities. There are three pieces of relevant legislation, (1) the draft Local Power and Decentralization Administration Law, (2) draft Suco Law No. 32_III. January 2016 and (3) the recently approved Decree Law No.3/2016 that defines Municipalities administration, Municipal authorities and the inter-ministerial technical group for administrative decentralization. There is also a law that was passed, Decree-Law no. 4/2014 on Administrative Pre-deconcentration that has directly impacted the project. This context has challenged staff of Ba Distrito to be in pace with proposed changes in suco council structure as well as governance and local planning practices at the municipal level. Primarily these contextual factors have impacted the project in the areas of training development and advocacy work. Program Implementation Finding 3: The implementation of Decree-Law no. 4/2014 on Administrative Pre-deconcentration has resulted in a shifting landscape of GoTL staff fielded into new positions and roles. This constant shifting landscape impacts all Ba Distrito components. According to GoTL interviews and FGDs held with suco council members, there was limited and sometimes minimal understanding of the major objectives of the project. This was in large part due to the constant reorganization of municipal level staff and national staff and leadership at MSA. At the suco level, this was the result of the small staff footprint of Ba Distrito and the use of NGO partners as trainers who in some cases did not adequately introduce the objectives of the project. Finding 4: Delays in suco elections and legislation will continue to challenge the pace and impact of the project. According to interviews with Ba Distrito staff working in this component area advocacy activities had to be carefully planned and soliciting recommendations from suco council members was challenging. In addition to this, GoTL representation was not always easy to coordinate and in some cases was not done formally or with enough lead-time for adequate staff engagement. Based on staff interviews, GoTL SGDs, and a review of consultation events planned and implemented there was evidence that the project 28 team could have monitored these conditions more closely to minimize tensions that emerged around some component activities. Unintended Outcomes Finding 5: According to interviews with Ba Distrito staff and partners the constant change in GoTL staffing challenged the staff to build strong formal coordination mechanisms with GoTL stakeholders at the municipal and national level. Primarily these contextual factors impacted the project in the areas of GoTL coordination and engagement in training and advocacy work. One of the unintended outcomes of these factors is that GoTL partners viewed some advocacy and stakeholder consultation events negatively as communication did not reach the necessary levels in time for national staff to feel informed or to identify areas for more effective national coordination and engagement. Gender Sensitivity Finding 6: In interviews with Ba District staff and in SGDs held with suco council members there was evidence that advocacy activities and civic engagement stakeholder forums were organized consistently to not only include the representation of women, but to also ensure that their voices, recommendations, and challenges in integrating their ideas into mixed forums was monitored for impact. Ba Distrito staff in this component area had specific strategies and approaches that were used consistently and there was ample evidence that this had provided space for public speaking opportunities for women and their recommendations to be incorporated into GoTL recommendations at the national level. Evidence included a review of recommendations from women suco council members as well as interviews held with suco council reps that had attended municipal level consultations on specific legislation planned. Finding 7: The difference in this component was the thoughtful strategic and mindful management of the results of community consultations and the care in which staff took to ensure the messages were written in the voice of the community members engaged. This gender integration practice was designed at the early stages of the project, but was fully implemented in this component, particularly in community consultations, building local recommendations, and aligning advocacy efforts with suco council engagement and ownership. KEY CONCLUSIONS: DECENTRALIZATION AND INPUT OF LOCAL INSTITUTIONS Challenges and Lessons Learned One of the largest challenges faced by this component and the project is the dynamic and unpredictable nature of decentralization in Timor-Leste. Most government ministries despite over a year of planning for deconcentration have yet to adequately staff and resource their line ministries at the municipal level. Ba Distrito has a small number of staff and Ba Distrito and NGO partners interviewed were often confused about the proposed changes to deconcentration and decentralization and its impact at the municipal level on staffing and resources. Basically, despite over a decade of planning, decentralization is still a moving target in Timor-Leste and its political nature, controversial realignment of geographic areas, and potential for large-scale deconcentration and government reorganization is not well defined or understood. This impacts the relationship between civil society and the government and also has impacted some of the advocacy efforts of Ba Distrito and the way they are perceived by some national stakeholders. Ba Distrito has a small staff and has not yet developed concise message for civil society partners and for GoTL stakeholders that allows advocacy activities to be seen as constructive methods to increase civic engagement and strengthen decentralization information at the local level. The project has done this through a series of small research grants, and the staff has worked diligently to establish advocacy efforts that are strategic and well aligned with the context, but there is more to be done. It is even more critical 29 now with suco elections delayed and national elections pending for Ba Distrito staff to be aware of these delays and how they will impact local communities and their cooperation and coordination with GoTL staff. Program Implementation The Ba Distrito project struggles with an operational environment where GoTL staff are constantly changing and where legislation is delayed and often poorly understood by local populations. The constant reorganization of GoTL staff, stakeholders and ministerial representation and resources at the municipal level makes coordination and communication a full time job. It also makes rolling and current analysis of these delays, transformations, and realignments critical for project planning, and implementation monitoring. This component can be an anchor in accomplishing these tasks, but analysis and information about delays and current GoTL challenges are almost as important as advocacy activities themselves. There is a need for more analysis and for more information sharing between citizens and the GoTL facilitated by Ba Distrito. Delays in elections and legislation will continue to challenge the pace and impact of the project, but creating more effective forums for citizen and GoTL exchanges will improve the impact of this component area. One particularly dynamic area of learning is Oecusse, which by its very history of isolation has made it one of the first decentralized areas. There is much to be learned about decentralization, deconcentration, integrated planning, and suco level engagement in this particular region of Timor-Leste. This component and the staff associated have a unique opportunity to work on understanding and integrating these lessons into project planning and implementation. Unintended Outcomes Strengthening formal coordination mechanisms with GoTL stakeholders at the municipal and national level requires a significant staff investment with a small staff located primarily in Dili. One of the unintended outcomes of this small staff configuration is that GoTL partners viewed some advocacy and stakeholder consultation events negatively as communication did not reach the necessary levels in time for national staff to engage effectively. Gender Sensitivity This is a component area with laudable gender integration and the staff articulation and clarity as to how these methods were utilized were effective and well monitored. Of particular note was the clear and precise way that Ba Distrito staff organized community consultations and ensured participation and voice for minority groups. This was not just a written project approach, this was a clear and established staff practice. This was not only evident in interviews in this component area, but also was evident in a review of recommendations proposed by some of the community consultations held and in FGDs with suco council members who participated in advocacy events and consultations as well as suco exchanges. KEY FINDINGS: LEGAL AID ORGANIZATION SUSTAINABILITY Goal 2: Increased Access to Justice Component C: Legal Aid Organization Sustainability Objective: Strengthened local justice sector institutions that increase access to formal and informal justice for marginalized citizens and the poor. Implementing and Institutional Partners: Fundacão Fatuk Sinae Oecusse (FFSO), JSMP, Baucau Justice and Peace Commission, and Liberta Component Overview: The goal of this component is to bring justice closer to rural residents by increasing access to formal and informal justice for marginalized citizens and the poor. This is done by (1) supporting the government to develop responsive, integrated and sustainable legal aid assistance; (2) 30 improving the capacity and outreach of legal aid and alternative dispute resolution providers; (3) assessing and reinforcing alternative dispute resolution mechanisms; (4) providing targeted assistance on improving the legislative framework for legal aid; and (5) providing support to government and CSOs to create accessible and effective legal information. Challenges and Lessons Learned Finding 1: Legal Aid organizations at the municipal level suffer from low capacity and minimal resources and this has impacted Ba Distrito program performance. Based on interviews, program documentation and cross municipal comparison, some of the partners chosen for legal aid support and provision at the municipal and suco level suffer from donor dependency, weak institutional strategies, and extremely low levels of staffing. This has impacted the level of activities the organizations can realistically accomplish. Finding 2: Although some of the partners (i.e. Oecusse) have been partners of past access to justice efforts funded by USAID based on field interviews with staff, a review of project agreement documents, and interviews with past INGO partners who had worked for years with this NGO there are institutional issues that are significant such as weak financial capacity, low staffing to activity ration, and, in some cases, minimal capacity to facilitate cases in the local language. Finding 3: All legal aid organizations interviewed and analyzed by the field team were struggling financially with sustainability and staff retention issues. Ba Distrito’s small grants and limited technical assistance was helpful, but it was not sufficient based on field analysis to address the deficits specified by some of the partner organizations. Weak donor funding and coordination in this sector make addressing these deficiencies challenging for the project, but based on the project objectives new strategies should be engaged. Ba Distrito has sought to strengthen the regulatory environment for legal aid organizations and donor coordination and advocacy may be another area the project to engage to strengthen the sustainability and viability of the legal aid environment. Program Implementation Finding 4: Legal Aid partners interviewed stated that the personal legal aid training at the suco level was effective and several partners supported radio programs on legal aid information, which included radio dramas with gender sensitive messaging. However, during interviews at the suco level most people interviewed did not have radios and had not heard the messages. At the municipal level in interviews the radio messages had been heard and well broadcasted, but reaching the most vulnerable and marginalized through this medium may need some strategic planning based on the norms in Timor and the limited access at the suco and aldeia level to radios. Finding 5: Legal aid partners interviewed felt that the mobile clinics and local legal education sessions in Baucau, Covalima and Oecusse were well received. During the interviews with suco council members this was confirmed, but participation was limited, as not all suco members interviewed had attended the trainings. Sometimes based on availability other suco members would attend. The most popular training based on interviews was the training on access to justice and women’s rights. Finding 6: In terms of implementation, the key areas where legal aid organizations need more support based on interviews with senior program managers and court officials working with Ba Distrito legal aid partners are report writing, strategy planning, proposal development, and legal training. These areas of deficiency were noticeable in partners in Baucau, Oecusse, and Covalima. Finding 7: Language is also a significant challenge to effective implementation as local language is essential in some settings, particularly with poor, vulnerable, and under-educated clients. In some cases, such as Oecusse only the Director could provide training at the local level and this impacted the efficacy of local 31 efforts, as it seemed unlikely that the Director could cover all the training and coordination events at the local level as specified in the work plan. Unintended Outcomes Finding 8: One of the unintended outcomes of this particular component area was the low number of legal aid organizations to partner with. In each municipal area, there are between one and three legal aid organizations and many of them have specific areas of intervention and in some cases donor funding that made them unable to partner with Ba Distrito. There was evidence that in some cases, such as Oecusse, the partner chosen for the small grant may not have been the first choice, but rather was the partner available. As a result of this selection, activities suffered and this was clear based on field analysis as well as in gaps identified by partner organization staff. Finding 9: Based on interviews with Ba Distrito staff, Justice Sector officials, suco council members and legal aid organization staff these services are critical. There is an evident need based on fieldwork for legal aid information to be made more accessible to marginalized populations, but some of the partners were too under resourced to effectively meet the targets established based on the budgets proposed for these activities. One unintended outcome is a lack of oversight and careful monitoring of these partner weaknesses to strategically realign or reduce the activity level to meet the resource realities. Finding 10: Building upon past USAID efforts by utilizing past partners was efficient, but there is a need to look at some of the dependency and sustainability issues of a few of the NGO partners chosen. Based on evidence and fieldwork conducted, some operate with minimal staff and struggle to maintain their presence and programming. This is particularly relevant to particular legal aid organizations, but all partners interviewed noted funding and sustainability issues. Gender Sensitivity Finding 11: This component did seek to target and increase access to justice for women and marginalized groups. Based on fieldwork the legal aid partner in Oecusse only worked with perpetrators or accused parties of domestic violence cases (95% men). This was highlighted by the partner organization, but was not well reported in program documentation. Based on interviews with field staff of the implementing partner, there was minimal coordination linking perpetrators to anger management or case management services with other service providers in Oecusse. Although there are monthly coordination meetings based on interviews conducted, all legal aid partners do not always attend these and that planning can be improved across service areas. Finding 12: This component did seek to target and increase access to justice for women and marginalized groups. Training was provided on issues such as access to legal aid for domestic violence issues, women’s rights relevant to land and property, and training for women suco council members on conflict resolution and mediation. Radio programs also sought to create gender sensitive messaging and most programs tried to incorporate programming specific to the needs of women. KEY CONCLUSIONS: LEGAL AID ORGANIZATION SUSTAINABILITY Challenges and Lessons Learned One of the biggest challenges for this component of Ba Distrito was the weak capacity of the legal aid service provider partners and their dependency or project funding. It was not just the weak technical capacity, but the organizational structure, financial management, and staffing resources. The budget provided to these partners by Ba Distrito was very low in most cases and the staff ratio to activity output seemed insufficient in some cases based on the level of activity agreed upon. In several instances the local language was also a challenge in disseminating information and ensuring that the targeted population was reached. The legal aid environment in Dili is small, and at the municipal level, the partner selection is even 32 smaller and the pool of potential stakeholders for a project with the ambitions and objectives of Ba Distrito is a real challenge and one not easily overcome. There was a vetting process for the selection of these partners and in most cases this worked well, but in the case of Oecusse the selection was not based on the best fit, but rather who was available and able to accept funding at the time. All legal aid organizations interviewed and analyzed by the field team were weak and funding from Ba Distrito was not sufficient to bridge the gaps evident. This was not an ideal process for partner selection. Some of the partners had been affiliated with past USAID justice efforts such as the partner in Oecusse, but their financial performance, staffing and management structures seemed less than ideal, particularly when analyzing the amount of cases they were meant to handle monthly and the activities that they had agreed to complete for Ba Distrito. Program Implementation One of the strengths of this component was the research conducted showing the need for these types of interventions and the critical role such organizations can fill for creating more accessible justice for the poor and underserved. The project did extensive initial analysis, and in some cases the gaps apparent are due to the weak justice environment, and would be a challenge for any donor seeking to strengthen these services. Gaps noted in fieldwork of Ba Distrito legal aid partners were in the following technical areas: report writing, strategy planning, proposal development and legal training. These areas of deficiency were noticeable in Baucau, Oecusse, and Covalima and also noted in interviews with Liberta conducted in Dili. In interviews with program managers, technical staff, and directors, several specific examples were given to show how these deficiencies impacted project performance. In analyzing the small grant funding levels for these partners it was clear the budget issue is a significant hurdle, but also the staffing capacity of Ba Distrito to technically address these gaps with their service delivery partners and stakeholders presents a challenge. Language is a persistent challenge to effective implementation as legal aid professionals responsible for case management did not always speak the local language and this affected partner workload distribution as well as client access to case management support. In Oecusse, only the Director could provide training at the local level and this impacted the efficacy of local efforts as evidenced in suco interviews held in the field. There was a need to mitigate this by providing translation support or limiting the scope of activities required to be conducted by the Director. The sheer distance from one suco to another also made it questionable whether these activities could be effectively carried out in the specified time frame. It was also clear from the SGDs held at the suco level that these trainings were not always attended as widely as one would have hoped in looking at the targets both for the project and for the partner compliance requirements. Unintended Outcomes One of the unintended outcomes of this particular component area was in many ways beyond the control of the project although may have been able to be addressed at the design stage of the Ba Distrito program. What was needed was a technically driven mentorship program for legal aid providers and carefully designed capacity building approach with clear institutional strengthening measures, but the budget for this component area was not sufficient to address these needs. The environment for legal aid provision in Timor-Leste is weak, but there is a significant need to extend these services into remote environments. Former justice programs funded by USAID had used a local paralegal approach and this may have been a better practice for long term sustainability in areas where weak justice access is likely to be the norm for an extended period of time. 33 One obvious difference between Ba Distrito and the previous project is the significant difference in resources available to Ba Distrito’s justice components, and previous justice program. A significant challenge is how to improve the quality of legal assistance being provided and contribute to longer-term government commitment to the important role of private legal aid organizations in increasing access to justice, and one that requires in many ways an entirely separately funded program and approach. Based on fieldwork legal aid organization staff and services are critical in all three municipal areas visited. There is an evident need for legal aid information to be made more accessible to marginalized populations, but some of the partners were too under resourced to effectively meet the targets established based on the budgets proposed for these activities. One unintended outcome is the lack of oversight and monitoring of these partner weaknesses and the projects ability to strategically realign or reduce the activity level to meet the resource realities and contextual needs. The project provided legal aid service provision in all five municipalities on a very limited budget, and this in and of itself was a feat. If the project had more significant funding, paralegals may have been an approach to consider and one that seemed to have significant impact in past USAID funded justice efforts. Ba Distrito did not have the budget to allow this. Gender Sensitivity This component clearly sought to address the specified target populations, but in some cases this was not monitored and managed as effectively as it could have been. The legal aid partner in Oecusse only worked with perpetrators or accused parties of domestic violence cases (95% men). Partners of the legal aid organization highlighted how difficult it was to link potential perpetrators to anger management or case management services. This is an area that should be addressed and better coordinated with other service providers to ensure a more holistic and measured approach in this particular component area of the project. This component did seek to target and increase access to justice for women and marginalized groups, and training was provided on issues such as access to legal aid for domestic violence issues, women’s rights relevant to land and property, and training for women suco council members on conflict resolution and mediation. Radio programs also sought to create gender sensitive messaging and most programs tried to incorporate programming specific to the needs of women. One of the issues to consider, however, may be other forms of social media and messaging, as radio access seems more limited in Timor-Leste than other environments. KEY FINDINGS: DISTRICT COURT FUNCTIONALITY Goal 2: Increased Access to Justice Component D: District Court Functionality Objective: Strengthened local justice sector institutions that increase access to formal and informal justice for marginalized citizens and the poor. Implementing and Institutional Partners: Dili Court of Appeals, PACT with Oecusse and Baucau district courts, and the Legal Training Centre. Component Overview: The goal of this component is to increase access to formal justice by strengthening specific district court institutions. This component is being piloted in the Oecusse and Baucau courts. Activities in this component area seek (1) assess needs and design district court training; (2) deliver training for justice and accountability institution staff; (3) advocate for the introduction of a professional development system; and (4) build the capacity of the Legal Training Center (LTC) and other justice sector institutions. Component D activities are integrated with Component C activities to assist in spanning the formal and informal sector and increase the provision of legal information in the municipalities. Challenges and Lessons Learned Finding 1: The sheer difference between Goal 1 and Goal 2 of the project remains a challenge. This was evident in interviews and also in analysis of project documentation. Ba Distrito subcontracted to Tetra 34 Tech DPK for the management of these two component areas and the staff responsible were exceptionally qualified to address and understand the particular needs of these technical areas, but the resources were not sufficient to address some of the complexities, gaps and needs in this component area. In addition, some of the identified project areas as specified in the proposal could not be addressed based on the refusal of the Ministry of Justice to sign and establish a Memorandum of Understanding (MOU) with the project. Finding 2: The Justice Sector in Timor-Leste is weak and suffers from minimal reach into rural areas, insufficiently trained judiciary staff, and a set of laws that are not fully understood by the staff who must implement and create sentencing guidelines, case precedents, and legal analysis based on these legal frameworks. This is the not the fault of the project, but was a current political and practical reality that the project had to face. The Justice sector in Timor-Leste is also critically aligned with the Portuguese system and this stems from Portuguese being the primary language of law in Timor-Leste, but also from the support over the years of Portuguese legal providers and the role and influence of CPLP. Recent tensions over the use of Portuguese Advisors and slow improvements in the relationship with Portugal over these issues had impacted the Ministry of Justice as well as the Legal Training Center. Finding 3: Although the project was not able to establish an MOU with the MoJ, it did succeed in developing coordinated training programs with the Legal Training Centre (LTC), Director of National Directorate for Human Rights and Citizenship and the Director for Legislative Drafting. Research grants and technical and training support for these GoTL partners allowed the project to work with the MoJ and the LTC on justice issues such as training in Administrative Court Management for Judges and Case Management for Lawyers. Finding 4: To mitigate these challenges the project analyzed gaps in the legal aid environment in Timor￾Leste and through technical assistance and research grants published a series of analysis on the deficiencies and issues the justice system faced. These studies focused particularly on analyzing the access to justice issues that minority groups, particularly women, the elderly, and victims of domestic and sexual and gender based violence (SGBV) face in accessing justice. These studies were widely referred to by partners, and by some interviews with district court staff. Program Implementation Finding 5: Based on initial problems establishing a formal partnership with the Ministry of Justice the project established an MOU with the National Court of Appeals and established two pilot programs for the district courts in Baucau and the Special Administrative Region of Oecusse. Two intervention areas were identified: (1) to support improved statistical reporting and case management, and (2) to increase administrative and access issues relevant to enhance court use. There was also training provided on administrative court procedures through short-term technical assistance (STTA). Based on interviews, this was a reasonable pathway for mitigation, but the results were mixed and specific to the particular pilot court interventions. Finding 6: Based on interviews, training provided on Administrative Court Procedures was useful and the use of an international trainer from the U.S. was appreciated, but there was a request for training to be based more on the Portuguese system of court management as this is more conducive to the context and realities of Timor-Leste. The LTC is also an institution with weak documentation of training and modalities used, so the project struggled to create uniformity and a clear pathway for adoption of some of the training methods and materials used. Finding 7: There are procedural issues that impact the functioning of the court in Baucau and Oecusse. Variations in court procedures challenge court reporting, monitoring and case documentation and management. Ba Distrito’s support for statistical reporting is simple, but still requires significant training 35 in data entry and tool management. There are still challenges in recording using the Excel system developed. This is improving, but there was minimal experience with Advanced Excel programming and if not well monitored this may impact the consistency and utility of this tool. Finding 8: There are sentencing structures in Oecusse and irregularities that are a result of insufficient staffing and human resources and Ba Distrito does not have the level of resources it would need to address these issues. The Judge can serve on over 200 cases a year and there is only one judge. The caseload alone is unsustainable in these settings, so irregularities are not surprising based on interviews with court staff in Oecusse as well as the Legal Training Center. There was clear evidence that there is just not time or resources for judges to be trained on court procedures and to serve on cases. Finding 9: JSMP developed a report with funding from Ba Distrito, which was based on the monitoring of the Oecusse court by JSMP. Court cases on domestic violence as reported by JSMP, a partner of Ba Distrito who did court monitoring in Oecusse sites Oecusse as having the highest rates of GBV cases currently in the court system. FFSO, the Oecusse legal aid partner of Ba Distrito, believes that is because of the increased awareness of Suco Chiefs to not mediate domestic violence issues. Staff from FFSO in interviews attributed this to the work done on sensitization on the Law Against Domestic Violence (LADV) and also on past and current awareness training with suco councils on these issues. They felt training, mobile clinics, and strong relationships between legal aid providers, women’s rights organizations, and the suco chief all contribute to these factors, but that it has taken many years. Based on field interviews with suco council representatives the increase in cases is a natural reaction to more training provided for suco chiefs and traditional leaders who with increased awareness refer more cases to the formal justice system. Unintended Outcomes Finding 10: Based on analysis of the statistical reporting tool and interviews with court accountability staff, Ba Distrito’s support for statistical reporting is simple, but still requires significant training in data entry and tool management. It also requires local oversight on a more regular basis as several court staff interviewed still seemed unfamiliar with the excel tool. Statistical reporting services will help improve the functionality of the district courts, only to the extent that staff of the courts uses the analysis. More regularized reporting can assist not only in case management, but also in trend analysis and monitoring. One of the unintended outcomes is that the tool, although simple, may not be adopted due to technical knowledge transfer issues. Gender Sensitivity Finding 11: This particular goal area of the project and this component area focused on access to justice for marginalized groups including women. According to some legal aid partners interviewed in Oecusse, Covalima, and Baucau there is still a need to understand some of the trends evident relevant to GBV cases. For example, Oecusse has one of the highest rates of GBV cases, but there the reasons why seem to not be well researched. There are theories and correlations proposed by partners based on the role of suco chiefs and traditional authorities in referring these cases at higher levels that should be explored. Ba Distrito staff and partners both expressed interest in pursuing the reasons why such high levels of formal case referrals were evident in Oecusse. KEY CONCLUSIONS: DISTRICT COURT FUNCTIONALITY Challenges and Lessons Learned This goal area and its two components were particularly challenging for the Ba Distrito project. The staff was technically strong, but the weak justice sector, resources required and the sheer magnitude of the tasks at hand made these efforts an uphill battle. It is almost as though these goals could have been divided 36 into two separate projects and this may have made more sense in addressing the clear access to justice needs in Timor-Leste. The Justice Sector in Timor-Leste is weak, and political problems persist within the Ministry of Justice and in independent institutions such as the district courts that make access to justice an exceptionally complex challenge that is likely to be embedded in the Timor context for many years. One of the challenges the project faced was establishing ownership of the activities, tools and trainings implemented and planned. There is a strong preference for national or Portuguese trainers as the system is predicated on establishing national capacity and Portugal has a complex and deeply rooted relationship within the justice sector. There is a clear need within the project to address these challenges head on for the remainder of the project performance period. This project component has smart, competent and well-informed staff, but mitigating these challenges with a relatively small budget and weak local partners presents a significant hurdle to overcome. The research and analysis of gaps in the legal aid environment in Timor conducted by the project holds a wealth of information for future project planning, but it would be better to separate these goal areas into two smaller projects rather than merge the two in the future. The context requires separate resources, strategies, and staff approaches. This impacted Ba Distrito and though they did an exceptional job managing these technical disconnects national budgets are small and there are clear challenges ahead the project must face. Research conducted by Ba Distrito has some lessons to share, and one is that there is less of a need for a huge access to justice program, but for more targeted and effective methods to address critical gaps and deficiencies in increasing access for the poor and marginalized. Program Implementation Partnering with the Ministry of Justice (MoJ) proved to be a challenge and implementation mitigation strategies were established to address this issue. Although the project was not able to establish an MOU with the Mo,J it did succeed in coordinating with the Legal Training Centre, the National Directorate for Human Rights and Citizenship and the Director for Legislative Drafting. Research grants and technical and training support for these GoTL partners allowed the project to work with the MoJ on policy issues critical to justice in Timor-Leste that proved to be an effective use of limited project resources. An MOU was signed with the National Court of Appeals and two pilot programs established with the district courts in Baucau and the Special Administrative Region of Oecusse. Based on interviews this was a reasonable pathway for mitigation, but the results were mixed and specific to the particular pilot court interventions. Weak procedural issues impacted the functioning of the court in Baucau and Oecusse. There were significant variations in court procedures and this impacted citizens seeking access to justice in these courts. In the courts there is irregularity in the sentencing, fine structure and remediation offered to victims and perpetrators of domestic violence. There has been an emphasis on increased awareness in communities of GBV by Ba Distrito, but it appears that the courts themselves also need enhanced training. Consistent court monitoring and advocacy to improve judicial practices to protect victim rights are considered critical by most sources interviewed. Judges have been trained in the Law Against Domestic Violence (LADV), but there is not enough awareness at the district level and perhaps experience in establishing guidelines for sentencing that protects victims’ rights. Irregularities in sentencing impact whole communities and also influence victims who may not seek formal remediation for crimes based on past sentencing of the courts and their knowledge of this. Unintended Outcomes There are procedural issues that impact the functioning of the court in Baucau and Oecusse. Variations in court procedures challenge court reporting, monitoring and case documentation and management. Ba Distrito’s support for statistical reporting is simple, but still requires significant training in data entry and 37 tool management. There are still challenges in recording using the Excel system developed. This is improving, but there was minimal experience with Advanced Excel programming and if not well monitored this may impact the consistency and utility of this tool. If not monitored and managed carefully, this tool will not be effectively adopted within the two pilot courts despite the political will and interest in particular partner stakeholders. Gender Sensitivity This component has a weakness that needs to be addressed. Despite significant gains in creating a more gender sensitive and aware environment, the partner in Oecusse needs some additional support and analysis to address their particular program intervention and modality. There is an immediate need to identify methods of intervention to coordinate with groups seeking to rehabilitate, counsel or work with perpetrators to reduce the risk of recidivism for cases covered by this service provider. If done mindfully this can also provide examples for the district courts, as there is a strong relationship between this partner and the district court staff in Oecusse. There is also a need to research more acutely the relationship the relationship between increased cased of GBV in the formal court system and the behavior and knowledge of referrals of local authorities. KEY FINDINGS: HOUSEHOLD SURVEY DATA Baseline and Midline Comparison (Oecusse, Baucau, and Covalima) Table 3: Baseline and Midline Measures for Oecusse, Baucau and Covalima (combined) Ba Distrito Objective Baseline Question Method/ Frequency Baseline Measure Mid￾Term Final Composi te Objective 1 Increased confidence among citizens in target communities in the effectiveness of suco councils in local development planning, infrastructure development and maintenance, and conflict mitigation Survey 63.36% Target 3.75% 3.75% 7.50% Baseline and Final Actual 81.34% Variance 17.98% Objective 2** Increased percentage of citizens who have heard about decentralization related activities in targeted municipalities Survey 48.53% Target 3.75% 3.75% 7.50% Baseline, Mid-Term and Final Actual 34.79% Variance -13.74% Objective 3 Increased in overall percentage of respondents in target areas who report greater access to justice and legal aid information Survey 70.23% Target 3.75% 3.75% 7.50% Baseline and Final Actual 55.71% Variance -14.52% Objective 3 Increased percentage of citizens in target communities who report confidence in the formal justice sector/district courts Survey 59.67% Target 3.75% 3.75% 7.50% Baseline and Final Actual 74.56% Variance 14.89% * All data will be disaggregated by: sex; youth/non-youth; and vulnerable minority population ** This was the only measure specified to be assessed at midterm. General Observations Finding 1: Based on interviews with Ba Distrito staff, it was evident that the variances and predicted increases (targets) from baseline to end line were done uniformly. This method is acceptable and is standard for predicting perception changes measured from randomized surveys, however it also presents some risks, as there are sure to be unpredicted increases and decreases in perception measures. 38 Finding 2: Some variations in the objective areas showed correlation to Ba Distrito interventions as well as other efforts, as there was evidence from interviews conducted that past USAID efforts (such as the 12-year Asia Foundation program) had not been forgotten. The impact of these efforts may also have contributed to the initial high baseline numbers reflected. This was not clearly articulated in the baseline calculations. Variances from Baseline to Midline, by Objective Objective 1: Increased confidence among citizens in target communities in the effectiveness of suco councils in local development planning, infrastructure development and maintenance, and conflict mitigation. The variance from baseline to midline trended upward by 17.98%. The project’s rationale was that training and support would enhance suco council functioning. At the same time, increased constituent engagement would also increase awareness of suco council activities. These interventions are expected to lead to increased confidence among suco council constituents. This tracks with question 13 and 14 of the survey. Figure 1: Effectiveness of Objective 1 (Table 5.3.2, Baseline Q 13) Finding 1: As outlined in Table 5.3.2 above, a majority of the sample indicated at baseline that suco authorities were either “Very effective” or “Effective” at the two tasks of “Local development planning…” (total 61.77%) and “Building and maintaining infrastructure…” (61.63%). However, even at baseline the National Program for Village Development (PNDS) had also impacted this measurement and was cited as one of the reasons for the high response rates. Finding 2: In interviews with suco council representatives, it was evident that training provided by Ba Distrito had assisted suco council members to better facilitate and coordinate community member needs regarding critical services. The upward trend observed in this objective, then, seemed consistent with qualitative data collected from FGDs held with suco council members who demonstrated a significant level of local knowledge about community issues relevant to planning and services. This trend, however, cannot be attributed solely to Ba Distrito as there were other partners working with suco council members targeted by Ba Distrito such as PNDS. However, based on interviews related to training and constituent 39 engagement activities participated in by suco council members this is an indication that the project has plausibly contributed to an increase in confidence of community members in suco council efficacy. Objective 2: Increased percentage of citizens who have heard about decentralization related activities in targeted municipalities. The variance from baseline to midline trended downward by 13.74%. The project’s rationale was that advocacy campaigns would increase awareness of and support for decentralization in participating sucos. This indicator was designed to track the increase in awareness and knowledge of decentralization processes and activities. This tracks with Q9 from the survey. Finding 3: Based on interviews with the Ba Distrito team, it was clear that advocacy efforts had been undertaken thoughtfully and with consistent effort within the project in Component A and B. However, the context of Timor-Leste is challenging and this indicator shows how unpredictable and confusing the decentralization process is at the local level. Based on interviews with suco council representatives, NGOs, and GoTL staff this downward trend observed in this objective seemed reasonable as consistent delays in the passage of laws and resolutions on decentralization in Timor-Leste have spanned three constitutional governments and have had limited civil society and public consultation. Finding 4: Based on qualitative fieldwork, there was also evident confusion as to the status of specific laws relating to decentralization and their impact on local systems of governance, particularly the impact these laws would have on suco council representation and the process of suco elections. This measure illustrates how little information filters down to the community level in Timor-Leste and how weak communication is between the GoTL and local communities on decentralization planning and implementation. Two baseline measures were associated with Objective 3. Objective 3 (a): Increase in overall percentage of respondents in target areas who report greater access to justice and legal aid information. The variance from baseline to midline trended downward by 14.52%. Based on program design for this measurement, advocacy campaigns were meant to increase awareness of available services and training, and technical support as well as legislative improvements were designed to enhance the provision of legal assistance. This tracks with Q17 as well as some information from Q21 from the household survey. Finding 5: This downward trend seems reasonable based on qualitative fieldwork. Extensive interviews held with legal aid partners, justice sector officials, staff of the LTC, and suco council members showed an increased need for greater information on court processes, access to justice issues, and weak information about specific sentencing and legal guidelines relevant to GBV, land issues, and case filing. This did not only extend to marginalized populations, but also to staff working within the court systems. This measure was specific to local populations and based on interviews and analysis of legal aid information available at the suco level, this trend seemed plausible based on the status and capability of some of the legal aid partners chosen for the project and the sheer number of sucos these partners needed to reach. Objective 3 (b): Increased percentage of citizens in target communities who report confidence in the formal justice sector/district courts. The variance from baseline to midline trended upward by 14.89%. The assumption for the project for this objective was that improved justice sector functionality coupled with increased awareness in target sucos would increase confidence in the formal justice system. This is measured by Q18 in the survey. Finding 6: This upward trend seemed plausible based on evidence from field interviews, but also from analysis of past donor efforts on access to justice efforts in overlapping suco and municipal areas of Ba Distrito. In interviews, there was evidence of residual impact from other USAID efforts working on access 40 to justice, so this trend seemed plausible but was derived from high initial baseline numbers reflecting the residual effects of past efforts in this objective areas. Finding 7: There was evidence from interviews with legal aid partners and in FGDs at the suco level that training in court access information had made respondents feel more confident in their ability to access and understand the court system. This was evident in FGDs held in Oecusse, Covalima, and Baucau. KEY CONCLUSIONS: UTILIZATION OF HOUSEHOLD SURVEY AND DATA The baseline tool and data did influence the projects planning, but may not be as useful as a predictive planning tool to design technical interventions. The baseline was used primarily to support the PMEP and although its information and data influenced some of the design of key activities, it was not integrated as a critical planning tool for the project. It was evident from fieldwork and desk research that the variances predicted for the midline and end line measures calculated within the Ba Distrito project did not seem well targeted based on contextual issues and norms. Based on analysis of the methods used during the baseline and the midline with SSD, the targets seemed more random than would have been expected on a project that had an extensive baseline conducted with a considerable amount of analytical detail provided. For example, some variations in the objective areas could not be attributed solely to Ba Distrito as there was evidence that past USAID efforts (such as the 12 year TAF program) and its residual impact may also have contributed to these increased measures. This was not clearly articulated in the baseline reporting and this oversight may impact the ability of the program to meet targets set without taking into account these considerations. The upward trend in Objective 1 by 17.98% is encouraging, showing an increase in awareness and confidence of suco council role and responsibilities. This upward trend was consistent with qualitative data collected from FGDs held with suco council members who demonstrated a significant level of local knowledge about community issues relevant to planning and services. This trend cannot be attributed solely to Ba Distrito based on the high initial baseline numbers and evidence of other donors working on similar training efforts in some suco areas. However, there is evidence that the project has plausibly contributed to an increase in confidence of community members in suco council efficacy, and based on interviews conducted with suco council members they were better informed about suco responsibilities in planning, advocacy, and community conflict resolution and in several cases they referred directly to training provided by Ba Distrito implementing partner staff. Downward trends in Objective 2 from baseline to midline by 13.74% seemed plausible based on the context of decentralization and its impacts on local populations and governance systems. Based on qualitative interviews with suco council members and staff, there was also evident confusion as to the status of specific laws relating to decentralization and its impact on local systems of governance, particularly the impact these laws would have on suco council representation and the process of suco elections. This variance illustrates how weak communication is between the GoTL and local communities on decentralization planning and implementation. For Objective 3a, the downward trend of 14.52% seemed reasonable based on extensive interviews held with legal aid partners, justice sector officials, staff of the LTC and suco council members who felt that there was a greater need for information on court processes, access to justice issues, and weak information provided to local communities on legal guidelines relevant to GBV, land issues, and case filing. This did not only extend to marginalized populations, but also to staff working within the court systems. This measure was specific to local populations, and based on interviews and analysis of legal aid information available at the suco level, this trend seemed plausible. 41 Objective 3b baseline to midline trended upward by 14.89%. This upward trend seemed plausible based on evidence from field interviews, but also from analysis of past donor efforts in these same suco areas on access to justice efforts. In interviews conducted, there was evidence of residual impact from other USAID efforts working on access to justice, so this trend seemed plausible based on the local efforts these projects had extended into some of the same program areas as Ba Distrito. This, however, was not well accounted for in the original baseline calculations. RECOMMENDATIONS: LOCAL GOVERNANCE STRENGTHENING Recommendation 1: Provide more training for suco councils in local governance planning and in prioritizing and sharing need more information on the impact of decentralization within their communities.  Strengthen project efforts working on suco level planning and prioritization of recommendations. When the Suco Law passes and the Local Power Law is implemented, there will be distinct changes in local government coordination with suco councils and there will be a need for this skill transfer. This will be particularly relevant in the areas of participatory planning, project monitoring and management.  Staff from Component A should seek opportunities to assist with this training and planning process within DNAAS and the MSA and other GoTL institutions (i.e. Ministry of Finance) to ensure the inclusion of local input into suco level prioritization and planning.  The integrated planning process being proposed in Oecusse may be a good place for the project to focus as it is unique to other municipalities and Ba Distrito has 100% of suco coverage in this district. Recommendation 2: Strengthen the relationship with DNAAS to ensure national ownership and increased GoTL cooperation.  Ba Distrito Component Managers and Municipal Coordinators need to coordinate more effectively with the MSA staff particularly, the National Directorate for Support and Suco Administration, the Department of Capacity for Community Leader and the Department of Technical Cooperation for Sucos. These efforts will improve this relationship and increase the likelihood of adoption of training module approaches and methods currently used in the project.  Decentralization is a process and Ba Distrito needs to not only monitor pending legislation and what it means for suco councils and local governance, but also define the specific roles, responsibilities and challenges that the Ministry and DNAAS will face in implementing and supporting these changes.  Ba Distrito project staff should share these formally with DNAAS staff in memos, or organized donor coordination meetings organized by the project with the Vice Minister or Minister. Recommendation 3: Enhance government ownership through capacity building partnerships.  Partnership efforts by Ba Distrito staff and USAID should be designed to allow more integration, ownership and exposure with GoTL staff. Ideas include support for Annual Action Planning (AAP) with DNAAS so that it includes actions to address gaps noted in suco level training needs identified by Ba Distrito. Ba Distrito project staff can also support activity planning to ensure that budget requests allow more DNAAS staff partnership in municipal level events and Training of Trainer (ToT) events.  Ba Distrito Component A and B staff managers should continue to work closely with the MSA to identify specific activities that support municipal level planning and the inclusion of recommendations and priorities from the suco councils as well as methods to prioritize recommendations locally. 42  Ba Distrito Municipal Coordinators must ensure that they collaborate with Municipal Secretaries and their staff. There needs to be regular, consistent, and predictable communication and sharing at this level.  This increased formalization of coordination systems within the project will increase GoTL ownership of the project, improve relationships with GoTL staff, and enhance formal learning between Ba Distrito and municipal staff. RECOMMENDATIONS: DECENTRALIZATION AND INPUT OF LOCAL INSTITUTIONS Recommendation 4: Find opportunities to strengthen inclusive planning.  There is a need to share lessons learned at the suco level with Municipal staff so that they can identify pathways for community prioritization based on integrated planning techniques.  Oecusse is a unique example as it was the first district to be decentralized, be it through isolation. It is also a pilot for the whole country so offers a good opportunity for possible future replication in this technical area.  In Oecusse, this needs to happen relatively quickly as there is a draft integrated plan that is pending approval. If this is released in the next 1-2 months, it will be an opportunity for increasing inclusion and representation where Ba Distrito is well represented.  Ba Distrito should look for tangible ways to do this in Oecusse, as there is a political will and appetite for this that is different than other areas.  These efforts could then be replicated based on lessons learned in the process in Oecusse and shared more widely in other municipal areas through research grants or suco to suco exchange forums. Recommendation 5: Support decentralization through innovation.  There has been a considerable amount of work in Ba Distrito on inclusion and advocacy, but this has sometimes been frustrated by the state of decentralization implementation and GoTL institutional capacity. Ba Distrito should look at innovations that may assist in more productive decentralization implementation and planning and integrate this into future efforts that support decentralization. The following actions are suggested: o Staff should look at some of the decentralization pathways that are being proposed within the GoTL such as within the Vice Ministers Office in the MSA and how these ideas can be leveraged and supported. o There are many positive ideas circulating to enhance local level engagement in decentralization planning in Timor-Leste. Ba Distrito staff should look for ways to leverage these efforts creatively and in close coordination with their GoTL counterparts. o The Suco Expo is a good example and more efforts like this are needed, although the scale can vary. This effort was well received by GoTL counterparts, but sustainability and ownership of future events need to be shared within the GoTL, particularly the MSA and ensure there is an annual budget discussion as to how to sustain such forums.  USAID should consider analyzing in the final years of the project the impact of these pending draft laws on local governance issues and their impact on state and citizen accountability. Research grants could be used to look at the future impacts of these and analyze the role that these laws will play on citizen engagement in local planning. o Municipal forums, suco exchanges and suco expos have been appreciated by GoTL staff, particularly in MSA, but the ability of line ministry staff to respond to citizen needs are limited. o In multiple GoTL interviews more services were requested to improve local planning and accountability processes within MSA at the municipal level. 43 Recommendation 6: Increase emphasis on public consultations and GoTL accountability.  Municipal forums conducted by Ba Distrito worked well under the previous administration and partnership with MSA but the current status of decentralization planning has weakened the interest for public consultations. This is likely to change as elections approach as there will be a need to have political platforms that allow more citizen engagement on key issues relevant to planning.  Ba Distrito needs to extend activities that seek to create citizen and government partnership and not just suco to suco exchange. The ET recommends that the Ba Distrito project consider the following actions: o Adapt and innovate some of the new areas of collaboration proposed in the Local Power Law as many of these areas represent good governance practices; o Adapt Ba Distrito training modules to accommodate some of these changes and a slight redesign of local level consultations and increased emphasis on GoTL participation may yield large impact for improved government and citizen engagement; and o Preposition women and youth as contributors to platform issues at the suco level, which are then prioritized in local level planning (i.e. municipal level). The project has done this in limited forums, but events need to be scaled up and more widely publicized. RECOMMENDATIONS: LEGAL AID ORGANIZATION SUSTAINABILITY Recommendation 7: Increase technical capacity building efforts for legal aid partners and increase organizational monitoring of project agreement activities.  To build organizational stability, their needs to be more direct work on organizational management and institutional strengthening with the legal aid service provision partner in Oecusse.  This is not unique to Oecusse, though, and partners in Baucau and Covalima also require significant capacity building support on specific technical areas necessary for organizational stability.  Increase technical assistance in report writing, case analysis, proposal development, and financial management and planning. RECOMMENDATIONS: DISTRICT COURT FUNCTIONALITY Recommendation 9: Monitor the utilization of the statistical reporting tool and staff skills in excel management.  Statistical reporting improvements are critical to tool adoption and require more oversight from Ba Distrito staff both in Oecusse and in Baucau.  Increased monitoring of tool use, adoption and staff skills within the court team will ensure integration of the statistical reporting tool and staff confidence in its use.  Court staff need additional training in ensuring that information is entered consistently. A Data Quality Assurance (DQA) process done by Ba Distrito staff or technical advisors should be conducted in Oecusse and Baucau. Recommendation 10: Utilize research and advocacy grants to improve court performance.  Ba Distrito should provide a research and advocacy grant to develop options for strengthening sentencing guidelines for cases of domestic violence to help judges determine the appropriate penalty. These guidelines could address specific inconsistencies in the Penal Code and the LADV and other human rights violations.  JSMP has done some initial work and a partnership on a policy paper may benefit USAID. Based on research conducted by JSMP, there are some recommendations generated by the Ba Distrito Project such as: 44 o Additional orders could include requiring the convicted person to: (1) periodically appear before a court, social reintegration officer or non-police entity during the duration of the suspension – Article 70(1)(g) of the Penal Code; (2) be monitored by social reintegration services for the duration of the suspension, and to adhere to a social reintegration plan – Article 71 of the Penal Code; (3) undergo treatment (or the court can require that follow￾up support services be provided to the victim) – Article 38(1) of the LADV; (4) have no contact with the victim for a maximum period of 3 years, if this is necessary to prevent a repetition of violence – Article 38(2) of the LADV; and (5) pay maintenance to the victim – Article 29 of the LADV. Recommendation 11: USAID should consider in the future combining these two component areas into a separate project that focuses on access to justice for marginalized groups.  This has been done in the past and the small budget allocated for these component activities and the scope and scale of the targeted implementation area made this challenging for Ba Distrito.  Areas of focus should be on providing more formal support for legislation for customary law and increasing mobile clinics and legal aid information provided by the GoTL.  There should also be more support provided to the LTC to provide more formal training for judges on sentencing guidelines relevant to LADV and human rights issues. This would, however, require a more formal partnership with the MoJ to allow more direct strengthening of LTC as a critical partner in improving training access for court staff.  More effective support for legislation that support legal aid organizations is also an area that should be continued, as these organizations are critical for marginalized and impoverished populations seeking recourse in the judicial system RECOMMENDATIONS: UTILIZATION OF HOUSEHOLD SURVEY AND DATA Recommendation 11: Use baseline and midline data to assess areas for increased or adjusted intervention by the project.  The Ba Distrito M&E team should consider updating end line targets for high-level indicators.  Utilizing the information provided on variances between baseline measures and midline results will allow a greater depth of analysis of these changes and will assist in project planning.  Analysis of these changes should be used to inform staff of specific general changes in knowledge and behavior and to assist in more accurate and informed activity level planning in these specific objective areas where significant downward trends were noted. 45 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK 1. TITLE OF ACTIVITY Mid-Term Evaluation - Ba Distrito Program 2. PURPOSE The purpose of this Task Order is to conduct a mid-term evaluation of the USAID Ba Distrito Program implemented by Counterpart International (Cooperative Agreement No. AID-486-A-13-00007). The activity began in September 2013, and will nm until September 2017. The purpose of the mid-tem1 evaluation is primarily to evaluate progress of the activity against the Baseline Survey conducted in 2014; to conduct the same Baseline Survey for the two municipalities that were added to the activity in November 2014; and to identify challenges faced and present recommendations for addressing them. This evaluation will inform the Mission about the activity's perf01mance and achievements, and will be used by the Mission to make programmatic adjustments to the project if needed. 3. BACKGROUND In September 2013, USAID awarded a four-year project called Ba Distrito (“To the Districts”) Program to Counterpart International. The goal of the activity is to increase institutional and human capacity at local levels to deliver basic services effectively and in a manner that is responsive to citizen needs and expectations. The activity is being implemented in 100 sucos in four municipalities namely Baucau, Covalima, Ermera, Liquica as well as in the Special Administrative Region of Oecusse-Ambeno. Initially the activity had four components, with a Total Estimated Cost (TEC) of $6 million. In April 2015, the Cooperative Agreement was modified to include an additional component to support anticipated local elections and the TEC was raised to $7 million. Below are five components of the activity: • Component A - Local Governance Strengthening • Component B - Decentralization and Input of Local Instih1tions • Component C - Legal Aid Organization Sustainability • Component D - District Court Functionality • Component E - Local Elections Support In 2014, Counterpart International hired a contractor, Social Science Dimensions (SSD), to carry out a Baseline Survey for the activity. The survey was undertaken in 2014 in 22 sucos throughout two targeted municipalities (Baucau, Covalima) and the Special Administrative Region of Oecusse-Ambeno. Data was collected through interviews with 958 randomly-selected community informants. The survey instrument was comprised of25 questions (including a number of sub-questions) divided into the four themes of (1) "Citizens' understanding of their role in political processes," (2) "Citizens' knowledge and awareness of decentralization in Timor-Leste (including suco legislation)," (3) "Citizens' knowledge of the roles and responsibilities of sucos, and their expectation of and satisfaction with suco service provision," and (4) "Citizens' access to justice." A fifth, cross-cutting theme concerning "Citizens' perception, understanding and agreement with concepts of gender equality and the participation of women, youth and minorities in decision making around development priorities" was integrated into the survey design process. The survey results were published in September 2014. They served as a basis for developing and adapting interventions to ensure responsiveness to citizen needs and expectations, as well as provide baseline data to assist with the monitoring and evaluation of the Ba Distrito Program over the term of its implementation. 46 The main beneficiaries of the activity are members of suco councils, legal aid organizations, and court actors (i.e. judges and judicial officers). The Ba Distrito program has been regularly monitored through the activity's quarterly rep01ting to USAID and site/office visits by USAID staff. 4. EVALUATION QUESTIONS The contractor is expected to use the results of the Baseline Survey to assess progress in the two municipalities and in the Special Administrative Region of Oecusse-Ambeno. The Contractor will also use the Baseline Survey Questionnaire (see attached Annex A) to establish the baseline in the two other municipalities (Ermera and Liquica). Any required modification to the questionnaire should be proposed in the work plan. In addition, the contractor is expected to ask following additional general questions: a. What are the main challenges faced by the project and lessons-learned to-date? b. What changes, if any, are proposed regarding implementation? c. What, if any, unintended consequences have occurred as a result of the project? d. 'What evidence exists that the program is contributing to a reduction in Gender-Based Violence? 5. SUGGESTED METHODS OF EVALUATION The Contractor is expected to rely primarily on qualitative methods for collecting data, such as through interviews and conducting a desk review of relevant documents. The evaluation methodology and sampling strategy for the mid-term survey shall be consistent with the methodology and sampling strategy used for the Baseline Survey (see attached Annex B). To the extent possible, the sampling strategy should be as inclusive as possible to acquire a representative dataset. The survey shall be gender-sensitive: the contractor will develop and select the best data collection methods and decide on timing to ensure gender￾sensitive samples. For this purpose, the contractor may need to schedule an interview with the SSD, as well as project implementer Counterpart International. The Contractor may propose other applicable qualitative and/or quantitative methods as appropriate. In addition, the contractor is also expected to conduct key info1mant interviews with suco council members, beneficiaries, staff of the project implementer, Counterpart International, USAID and other donors to allow for a qualitative data set to supplement the survey information. 6. GENDER INTEGRATION Within Timorese society, there is a widespread belief that women should defer to men, not speak out and not be involved in major decisions. Women are under-represented in national and local politics and the prevalence of violence against women is reported at 50%. The GOTL promulgated the Law Against Domestic Violence (LADY) in 2010, but much remains to be done to significantly enhance access to justice for victims of Gender-Based Violence (GBV). As part of its approach, the project applies its strategic framework and operational guidelines, as ·well as other tools to gender integration, which identify key aspects of gender relations across of social and political life and the project makes gender equality a focus across all program components. The Contractor should consider the different needs of men and women. The Contractor should consider how project outcomes and impacts were affected by the participation of women versus men, and specifically how the projects could increase the number of women beneficiaries (either direct or indirect). 7. PLACE OF PERFORMANCE The Contractor shall conduct its work in Dili, as well as travel to the project targeted areas to interview the communities that were interviewed for the Baseline survey (Baucau, Covalima and Oecusse-Ambeno) and communities in two other municipalities (Liquica and Ermera) where the sampling will be proposed 47 by the contractor and approved the COR. Some of the key informant interviews will need to be conducted at the municipality level. Trips outside of Dili will require overnight travel. The Contractor may choose to conduct some activities (such as desk review of documents, report writing) before and after deployment to Timor-Leste, as agreed upon in the Work Plan. The project may plan to interview SSD personnel should it be deemed necessary. This may be done in person or by telecommunication as appropriate. 8. TEAM COMPOSITION The evaluation team shall consist of three members: three technical specialists, a local Logistician/Program Assistant and a local interpreter/logistician. While it would be especially valuable that each of the technical specialists have extensive and documented experience in conducting performance evaluations, it is a requirement that at least the team leader has this experience. See Section F for Key Personnel. The above is a suggested staffing pattern. The Contractor may propose a different staffing pattern that meets the functional requirements described above. 9. DELIVERABLES a. Work Plan. Within 30 days of the award of the Task Order, the Contractor shall submit a detailed Work Plan to the Contracting Officer's Representative (COR) for approval. The Work Plan shall include a list of tasks to be completed including: (1) a brief of explanation of the purpose of the task in connection to the overall objective; (2) the necessary steps to complete the task; (3) the person(s) responsible for the task, and; (4) a timeline for the implementation of the task. The Work Plan should identify any actions required by the COR to support the Work Plan, consistent with Section 10.2 below. One week after award, the contractor participate in a conference call with the evaluation COR and other relevant Mission staff members to review the SOW and answer any potential questions. b. Evaluation Design. Within 30 days of the award of the Task Order, the Contractor shall submit a detailed Evaluation Design to the Contracting Officer's Representative (COR) for approval. The evaluation design will include a detailed evaluation design matrix (including survey questionnaire, key informant questionnaire, sampling strategy, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments, known limitations to the evaluation design, and a dissemination plan based on input from the COR - including with project implementer Counterpart International, as well as USAID Timor-Leste Mission. The final design requires COR approval prior to data collection. c. In-Brief with USAID/Timor-Leste. The Contractor must make arrangements through the COR to provide USAID/Timor-Leste with an introductory oral briefing within three business days of arrival in Dili. At this meeting, the Contractor will be expected to review their Work Plan. The contractor is also expected to arrange an introductory oral brief with the project implementer, Counterpart International. d. Exit Briefing with USAID/Timor-Leste. Before departing Timor-Leste, the Contractor shall provide an oral briefing to the Mission, reviewing preliminary findings. e. First Draft Evaluation Report. On a date agreed upon in the Work Plan the Contractor shall provide the COR (electronically) with a draft version of the evaluation report, using Microsoft Word. The Mission will provide comments on this draft through the COR. The report will meet the criteria outlined in USAID's Evaluation Policy (http://www.usaid.gov/evaluation and http://www.usaid.gov/evaluation/preparing-evaluationreports). 48 In addition to comprehensively addressing the evaluation questions, the final report should include: (I) a discussion of any constraints faced by the Contractor in the performance of the evaluation that were not resolved and that impacted the outcome of the evaluation; (2) an index of all information products (source materials) used by the Contractor; (3) a list of all key informant interview participants with their contact details; ( 4) discussion of methodologies employed; (5) signed disclosures of conflict of interest from each member who worked on any substantive part of the evaluation, and; (6) if applicable, statements regarding any significant unresolved differences in opinion on the part of the evaluation team. f. Second Draft Evaluation Report. On a date agreed upon in the Work Plan, the Contractor shall submit a second draft evaluation report (electronic version, using Microsoft Word or Adobe Document - PDF) to the COR that considers comments received from the COR based on the first draft final report. The Mission will provide comments on this draft. g. Final Evaluation Report. On a date agreed upon in the Work Plan, the Contractor shall submit a final evaluation report in both Microsoft Word and Adobe Document- PDF formats to the COR whose content and format reflects comments received from the COR based on the second draft report. The Contractor must submit the final report (Adobe Document - PDF), as approved by the COR, to the USAID Development Experience Clearinghouse – DEC (http://dec.usaid.1.mv). At that time, COR will advise if viewership of the report on DEC shall be limited to USAID. 10. SUGGESTED TIMELINE An illustrative timeline for the evaluation team to conduct their work and provide a Final Report is shown below. A six-day work week is authorized while in Timor Leste, provided activities are approved by the USAID COR. Task Week Task Week Task Order awarded Week 1 Completion of Pre-departure document deskreview and submission and approval of Eval uation Design and Work Plan. By Week 4 Deployment of Evaluation Team to Timar￾Leste. Commence field work. By Week 5 Analysis and Report Writing By Week 15 Submission of First Draft By Week 17 Submission of Second Draft By Week 19 Approved Final Draft By Week 20 11. LOGISTICS a. Responsibilities of The Contractor The Contractor is responsible for all of their own logistical needs necessary to perform the evaluation including, but not limited to: entry visas, computer access with internet, document printing and copying, lodging, office space, transport, interpretation at meetings with project beneficiaries, and translation of source documents. The Contractor shall communicate directly with the USAID COR, proactively identifying constraints that may impact the perfom1ance of the evaluation. b. Responsibilities of The COR The COR for the proposed evaluation is from USAID/Timor-Leste. As such, the COR will: 49 • Provide overall technical guidance for the evaluation • Provide all necessary documents to the Contractor to enable the Contractor to perform a comprehensive desk review • Ensure that the Contractor has the necessary information, including Letters of Introduction if needed, to conduct interviews in Timor-Leste with project beneficiaries and implementing partners • Meet with the Contractor as necessary to provide technical direction • Schedule opportunities for the Contractor to brief USAID/Timor-Leste on the evaluation • Provide prompt feedback to the Contractor when requested by the Contractor to ensure that the Contractor is able to meet the schedule, as agreed upon in the Work Plan • Serve as the overall Point of Contact between the Contractor and USA ID as well as project's implementing partner. All modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the Contracting Officer. 50 ANNEX II: INTERVIEW PROTOCOL AND TEMPLATES The following are interview templates and protocols for the qualitative fieldwork. This is a guide tool for the evaluation team. As not all respondents will have the same level of knowledge and experience with the Ba Distrito program, respondents will be weighted based on their role with the Ba Distrito Program. Not all respondents will be able to answer all questions, so the team will document each interview and periodically hold TPMs to analyze the data and establish initial findings. Interview Protocol Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are currently working with USAID to gain a better understanding of the Ba Distrito program, which started in 2013. This is a mid-term evaluation and will be used by USAID and implementing partners to inform progress and impact to date, and to also look at any reasonable improvements that can be made to implementation and planning moving forward. Our team has had the opportunity to review documents provided by USAID and the Ba Distrito implementing team to get a better sense of the design and implementation of the Ba Distrito Program. However, such documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how these projects look and function “on the ground.” Please note that, at any time, you may terminate the interview or decline to answer a specific question. You may also decline participation in this interview. This will be noted in the fieldwork notes as well as in the final report in the list of interviewees. Confidentiality Protocol  We will collect information on individuals’ names, organizations, and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report.  We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless the respondent grants express written consent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so.  All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? 51 Interview Template (USAID and BA Distrito Senior Managers) METHODS INTERVIEW QUESTIONS Coded based on role in Ba Distrito Program METHOD NOTES KII, SGD, or FGD 1. Primary: National Field Staff and Managers and Implementing Partners (P-FD) KII, SGD, FGD 1. Document all staff who “decline” an interview 2. Ensure that category 1-3 are triangulated with follow-up questions in Timor-Leste and in DC with other USG staff and in SGDs. 3. Coded categories 1-3 are weighted as these staff had primary tasks relating to activity outcome and impact. 2. Primary: Strategy Planning (P-SP) KII 3. Primary: Strategy Decision and Approval (P-DM) KII, SGD 4. Secondary: Strategy Review only (S￾SR) KII, SGD 5. Secondary: Stakeholder key component/activity area (S-S) KII, SGD 6. Peripheral: USG, External (donor, NGO, etc.) (PP) KII General 1. Before we begin, can you tell us a bit about your association with the Ba Distrito Program? 2. Were you involved in the design of the Ba Distrito program in Timor-Leste? If so, will you describe how the Ba Distrito Program and its intervention in Timor-Leste were designed? a. Did the design/ approach change during implementation? If so, why? 3. Do you feel that the Ba Distrito Program in Timor-Leste is in close alignment with the USAID and National Strategy? In your opinion, how effective has this program been thus far in achieving its stated objectives? a. Are there any factors that facilitated/hindered success? b. Are there any unintended consequences (positive or negative)? 4. How effective have Ba Distrito Staff and Stakeholders been thus far at the following? a. Mapping engagement needs at the national level? b. Identifying potential challenges and obstacles? c. Identifying and prioritizing information or intervention gaps? 5. Knowing what you know today, would you make any changes to the design and/or implementation of the Ba Distrito Program (at the following levels)? a. National level; b. Sub-national Level (municipality level); c. Institutional level (within the USG); and/or at the d. Partner level (NGO and Civil Society). Core Evaluation Questions: Question 1 What are the main challenges faced by the program and lessons-learned to-date? Sub Question 1.1  How was the original program design appropriate/not appropriate for addressing the program goals and objectives? Sub Question 1.2  How were project activities and implementation influenced by the baseline results and data collected? Sub Question 1.3  If so, which activities? Please provide specific examples and geographic focus areas. Question 2 What changes, if any, are proposed regarding implementation? Sub Question 2.1  What changes, if any, seem necessary at this point in the implementation of the Ba Distrito program? Sub Question 2.2  If so, which activities? Please provide specific examples. Sub Question 2.3 52  Are these changes specific to a particular municipality area? If so, why? Please provide specific examples. Sub Question 2.4  Are these changes specific to a particular target group? If so, why? Please provide specific examples. Question 3 What, if any, unintended consequences have occurred as a result of the program? Sub Question 3.1  Have staff or beneficiaries noted any particular “unintended” consequences as a result of Ba Distrito activities? Sub Question 3.2  How have these “unintended consequences” impacted particular component areas? Which ones? Please provide examples. Sub Question 3.3  How have these “unintended consequences” impacted particular target groups? Which groups? Please provide examples. Sub Question 3.4  Are their specific component areas that seem to have had greater impact than others? Which areas? Please provide specific examples. Question 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? Sub Question 4.1  What specific Ba Distrito activities have targeted this? Sub Question 4.2  How has this been monitored at the project level? Sub Question 4.3  What Ba Distrito activities have been most effective in impacting this? 53 Interview Template (KIIs, Timor-Leste Government Counterparts) METHODS INTERVIEW QUESTIONS Coded based on role in Ba Distrito Program METHOD NOTES KII, SGD, or FGD 1. Primary: National Field Staff and Managers and Implementing Partners (P-FD) KII, SGD, FGD 1. Document all staff who “decline” an interview 2. Ensure that category 1-3 are triangulated with follow-up questions in Timor-Leste and in DC with other USG staff and in SGDs. 3. Coded categories 1-3 are weighted as these staff had primary tasks relating to activity outcome and impact. 2. Primary: Strategy Planning (P-SP) KII 3. Primary: Strategy Decision and Approval (P-DM) KII, SGD 4. Secondary: Strategy Review only (S￾SR) KII, SGD 5. Secondary: Stakeholder key component/activity area (S-S) KII, SGD 6. Peripheral: USG, External (donor, NGO, etc.) (PP) KII General: 1. Before we begin, can you tell us a bit about your association with the Ba Distrito Program? 2. Do you feel that the Ba Distrito Program in Timor-Leste is in close alignment with the USAID and National Strategy? In your opinion, how effective has this program been thus far in achieving its stated objectives? a. Are there any factors that facilitated/hindered success? b. Are there any unintended consequences (positive or negative)? 3. How effective have Ba Distrito Staff and Stakeholders been thus far at the following? a. Mapping engagement needs at the national level? b. Identifying potential challenges and obstacles? c. Identifying and prioritizing information or intervention gaps? 4. Knowing what you know today, would you make any changes to the design and/or implementation of the Ba Distrito Program (at the following levels)? a. National level; b. Sub-national Level (municipality level); c. Institutional level (within the USG); and/or at the d. Partner level (government, NGO and Civil Society). Core Evaluation Questions: Question 1 What are the main challenges faced by the program and lessons-learned to-date? Sub Question 1.1  How was the original program design appropriate/not appropriate for addressing the program goals and objectives? Question 2 What changes, if any, are proposed regarding implementation? Sub Question 2.1  What changes, if any, seem necessary at this point in the implementation of the Ba Distrito program? Sub Question 2.2  If so, which activities? Please provide specific examples. Sub Question 2.3  Are these changes specific to a particular municipality area? If so, why? Please provide specific examples. Sub Question 2.4  Are these changes specific to a particular target group? If so, why? Please provide specific examples. Question 3 What, if any, unintended consequences have occurred as a result of the program? Sub Question 3.1  Have staff or beneficiaries noted any particular “unintended” consequences as a result of Ba Distrito activities? Sub Question 3.2 54  How have these “unintended consequences” impacted particular component areas? Which ones? Please provide examples. Sub Question 3.3  How have these “unintended consequences” impacted particular target groups? Which groups? Please provide examples. Sub Question 3.4  Are their specific component areas that seem to have had greater impact than others? Which areas? Please provide specific examples. Question 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? Sub Question 4.1  What specific Ba Distrito activities have targeted this? Sub Question 4.2  How has this been monitored at the project level? Sub Question 4.3  What Ba Distrito activities have been most effective in impacting this? 55 Interview Template (Other Donors and Sub Partners) General: 1. Do you feel that the Ba Distrito Program in Timor-Leste is in close alignment with the USAID and National Strategy? In your opinion, how effective has this program been thus far in achieving its stated objectives? a. Are there any factors that facilitated/hindered success? b. Are there any unintended consequences (positive or negative)? 2. How effective have Ba Distrito Staff and Stakeholders been thus far at the following? a. Mapping engagement needs at the national level? b. Identifying potential challenges and obstacles? c. Identifying and prioritizing information or intervention gaps? Core Evaluation Questions: Question 1 What are the main challenges faced by the program and lessons-learned to-date? Question 2 What changes, if any, are needed? Question 3 What, if any, unintended consequences have occurred as a result of the program? Sub Question 3.1  Have staff or beneficiaries noted any particular “unintended” consequences as a result of Ba Distrito activities? Sub Question 3.2  How have these “unintended consequences” impacted particular component areas? Which ones? Please provide examples. Sub Question 3.3  How have these “unintended consequences” impacted particular target groups? Which groups? Please provide examples. Sub Question 3.4  Are their specific component areas that seem to have had greater impact than others? Which areas? Please provide specific examples. Question 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? Sub Question 4.1  What specific Ba Distrito activities have targeted this? Sub Question 4.2  How has this been monitored at the project level? Sub Question 4.3  What Ba Distrito activities have been most effective in impacting this? METHODS INTERVIEW QUESTIONS Coded based on role in Ba Distrito Program METHOD NOTES KII, SGD, or FGD 1. Primary: National Field Staff and Managers and Implementing Partners (P-FD) KII, SGD, FGD 1. Document all staff who “decline” an interview 2. Ensure that category 1-3 are triangulated with follow-up questions in Timor-Leste and in DC with other USG staff and in SGDs. 3. Coded categories 1-3 are weighted as these staff had primary tasks relating to activity outcome and impact. 2. Primary: Strategy Planning (P-SP) KII 3. Primary: Strategy Decision and Approval (P-DM) KII, SGD 4. Secondary: Strategy Review only (S￾SR) KII, SGD 5. Secondary: Stakeholder key component/activity area (S-S) KII, SGD 6. Peripheral: USG, External (donor, NGO, etc.) (PP) KII 56 ANNEX III: BIBLIOGRAPHY Program Documents USAID/Timor-Leste Country Development Cooperation Strategy (CDCS) 2013 – 2018 Ba Distrito Program Award, Attachment B – Program Description Program Work Plans (Years 1, 2, and 3) Ba Distrito Program Performance Management Plan Ba Distrito progress against Performance Management Plan indicators Ba Distrito Baseline Survey Report (2014) List of Mid-Term Evaluation Sucos Contact List for Justice and Governance Partners and Stakeholders Ba Distrito Program Quarterly Reports (FY14 QI, QII, QIII, and QIV; FY15 QI, QII, QIII, and QIV; FY16 QI) Access to Justice Brief: Legal Assistance in Timor-Leste (Timor-Leste: Ba Distrito Program, 2014), Summary of Assessment Findings and Recommendations. Access to Justice Brief: Functional Assessment of the Oecussee and Baucau District Courts (Timor-Leste: Ba Distrito Program, 2014), Summary of Assessment Findings and Recommendations. The application of alternative sentences in cases of domestic violence at the Oecussee District Court (Timor-Leste: Ba Distrito Program, 2015). Overview of the Justice Sector: JSMP Annual Report (Timor-Leste: Ba Distrito Program, 2014). Suku Council’s Role in Effective Local Governance (Timor-Leste: Ba Distrito Program). Literature Review Damian Grenfell et al., Beyond Fragility and Inequality: Women’s Experiences of Economic Dimensions of Domestic Violence in Timor-Leste (Australia: Nabilan, 2015), Executive Summary. Justice at the Crossroads in Timor-Leste (Timor-Leste: IPAC, 2015), Report No. 22. Dismissal of international officials and advisors in the Timor-Leste judicial sector (Timor-Leste: JSMP, 2014). Justice Sector Strategic Plan for Timor-Leste 2011 – 2013 (Timor-Leste; Government of Timor-Leste, 2010). Timor-Leste Law and Justice Survey (Timor-Leste: The Asia Foundation, 2013). Asia Foundation. 2013. “Timor-Leste Public Opinion Poll” Asia Foundation publication dated September 2013. See http://asiafoundation.org/publications/pdf/1323 Asia Foundation. 2014. “Reflections on Law No. 3/2009: Community Leadership and their Election.” Asia Foundation report dated February 2014. See http://asiafoundation.org/publications/pdf/1319 Chopra, T., Pologruto, S., and Deus, Timotio de. 2009. “Fostering Justice in Timor. Timor-Leste program dated January 2009. Counterpart International. 2013. “Ba Distrito Final Technical Proposal.” Counterpart International project document dated 31 July 2013. Cutter, Ana, Sigrid Gruener, Suniter Vyavaharkar and Luis da Costa Ximenes. 2004. ‘Timor Leste Conflict Assessment: Final Report.’ Center for International Conflict Resolution at Columbia University: New York. 57 Gross, J. 2009. “Timor Leste’s Successful Local Elections: A Positive Sign for the Future?” Asia Foundation report dated October 14, 2009. See http://asiafoundation.org/in-asia/2009/10/14/timor-lestes-successful-local-elections-a￾positive-sign-for-the-future/ Government of Australia. 2012. “Independent Evaluation of the East Timor Justice Sector Support Facility,” AusAID. Government of Australia. 2013. "National Program for Village Development (PNDS) briefing (Timor-Leste) Summary,” DFAT/AusAID report dated March 2013. See: http://aid.dfat.gov.au/Publications/Pages/pnds-briefing￾march-2013.aspx Government of Timor-Leste. 2010. “Population and Housing Census of Timor-Leste, 2010 Volume 3: Social and Economic Characteristics, National Statistics Directorate (NSD) United Nations Population Fund (UNFPA). http://dne.mof.gov.tl/published/2010%20and%202011%20Publications/Pub%03%20English%20web/Publication%203% 20English%20Final_Website.pdf Government of the United States of America, 2013a. “Timor-Leste: 2013 Human Rights Report, U.S. State Department. See: http://www.state.gov/j/drl/rls/hrrpt/humanrightsreport/index.htm#wrapper Government of the United States of America, 2013b. “Country Development Cooperation Strategy 2013 – 2018,” USAID. ICG (International Crisis Group).2013. “Timor-Leste: Stability at What Cost?” ICG Asia Report N°246 dated 8 May 2013. See http://www.crisisgroup.org/en/regions/asia/south-east-asia/timor-leste/246-timor-leste-stability-at￾what-cost.aspx Nixon, Rod. 2009. “Non-State Actors as Agents of Order: Suco Justice and Dispute Resolution Systems in East Timor,” in Timor-Leste: Challenges for Justice and Human Rights in the Shadow of the Past (Ed. William Binchy), pp.1- 79. Clarus: Dublin. Nixon, Rod. 2011. ‘Timor-Leste: Change, Stagnation, and Questions of Political Economy’, in The Political Economy of Economic Reform in the Pacific (edited by Ron Duncan). ADB: Manilla. RDTL. 2004a. Law No. 2/2004 on the Election of Suco Chiefs and Suco Councils (RDTL law promulgated on 10th February 2004 and published on 18th February 2004). RDTL. 2004b. Decree Law No. 5/2004 on Community Authorities (promulgated 01/04/04; published 14/04/04). RDTL. 2009. Law 3/2009, of 8 July 2009 on Community leaderships and their Election. RDTL. 2010. “Justice Sector Strategic Plan for Timor-Leste 2010-2030.” RDTL strategic plan dated 12th February 2010. World Bank. 2006. ‘Strengthening the Institutions of Governance in Timor-Leste’. Dated April 2006. http://siteresources.worldbank.org/INTTIMORLESTE/Resources/Strengthening-Institutions-Full-report.pdf Ximenes, F.N. 2011. “Legal Aid Client Services Survey,” Asia Foundation report dated 2011. ANNEX IV: BA DISTRITO EVALUATION INTERVIEW LIST Dili Field Work Counterpart International Staff 1 Counterpart Carolyn Tanner Chief of Party 2 Counterpart Selma Hayati Organization Development Manager 3 Counterpart Uka Pinto Governance Technical Specialist 4 DPK Nino Ximenes Tt DPK Country Representative and Access to Justice Manager Ministry of State Administration 1 Ministry of State Administration H.E Dionisio Babo Soares Minister of State Coordinator of Affairs of State Administration and Justice, Minister of State Administration 2 Ministry of State Administration Sr Samuel Mendonça Secretary of State for State Administration 3 Ministry of State Administration Sr Abilio José Caetano Director General for Administrative Decentralization 4 Ministry of State Administration Sr Amandio Paulino de Sousa Director of National Directorate for Support and Suco Administration 5 Ministry of State Administration Sra Josefina Antonieta da Silva Chief Department of Capacity for Community Leader, Office of National Directorate for Support and Suco Administration 6 Ministry of State Administration Sr António da Costa Belo Chief Department of Technical Cooperation for Sucos, Office of National Directorate for Support and Suco Administration Justice Sector Partners 1 National Directorate of Human Rights and Citizenship￾Ministry of Justice Flavio Leao MONIZ Director 2 Celito CARDOZO Staff (former Director) 3 National Directorate of Legislation and Judicial Advisory (DNAJL). Also President of Private Lawyers’ Council Nelinho VITAL Director 4 Legal Training Center (LTC) Marcelina TILMAN Director General The Court of Appeal 1 The Court of Appeal Gizelda MOTA Court of Appeal President Secretary 2 Malena PEDADE Superior Secretary of the Court of Appeal National NGOs 1 Liberta Advocacia & Consultoria Aderito DOS REIS Director 2 Natalino LETO Program Coordinator/Lawyer 3 Aniceto DA SILVA Lawyer 4 Maria DO CEU REGO Finance/Admin officer 5 Judicial System Monitoring Programme (JSMP) Luis OLIVEIRA SAMPAIO Executive Director 6 Casimiro DO SANTOS Deputy Executive Director 7 Ana Paula MARCAL Legal Research Coordinator 8 Paulo NUNES Court Observer (Oecusse) 9 Nani DA CRUZ Finance/Admin officer 10 Belun Luis da Costa Ximenes Director 11 Miguel Mao Soares Program Manager Ba Distrito 12 Women Caucus Paula Corte-real Director 13 Fundacao Patria Laura Menezes Lopes Director 59 International NGOs and Multilaterals 1 The Asia Foundation Susan Marx Country Director 3 Sophia CASON Manager- End of Violence Against Women/DFAT 4 OXFAM Sharon ALDER Deputy Country Rep. 5 UN Women Camille Wauters Gender Mainstreaming 6 UN Women Cristina Fernandez Un Women, SEM Advisor 7 JSI Research & Training Institute, Marianne Viatour Chief of Party-Reinforce Basic Health Services Project (TL-Reinforce) 8 Health Alliance International Sarah Meyanathan Communication Specialist 9 WFP Marianne Kearney Communications, Donor Liaison Oecusse Field Work Oecusse Suco Council Members, Naimeco Suco Naimeco,Pante Macassar Check Name Women Representative Naimeco,Pante Macassar Sra. Lusiana Anunu Women Representative Naimeco,Pante Macassar Ms. Lucia Lalus Youth Representative (Female) Naimeco,Pante Macassar Check Name Youth Representative (Male) Oecusse Government Staff and Representatives 1 Special Administrative Region of Oecusse Sr. Francisco Marques Oecusse Regional Secretary for Administration (Acting) Sr. Arsenio Paixo Bano (Acting) 2 Secretary of State for the Economic Promotion of Women Inacio Kebo SEM Focal Point Oecusse Court of Appeal Team and Staff 1 Court of Appeal Team **Project Specific Hon. Judge Joao RIBEIRO Deputy Court of Appeal team president. Also Oecusse Court administrator 2 Amaro JOAQUIM Court of Appeal Team Secretary. Also judicial secretary of Oecusse court Oecusse Legal Aid Provider 1 Fundacao Fatuk Sinae Oecusse (FFSO) Antonio DOS REMEDIOS Director 2 Cesaltina DA COSTA FREITAS Lawyer 3 Arnold COLO Finance/Admin officer Baucau Field Work Baucau SGD, Suco Council Members Women’s Representative, and Youth Representative Name Position Alberto L. Sarmento Xefe Aldeia Virgilio R. Soares Xefe Aldeia Francisco Pereira Xefe Aldeia Jerimeas A. Pereira Xefe Aldeia Joana Casilda de S. Ribeiro Aldeai Member Joanico de S. Sarmento Youth Representative for Men Juscelina b. Ximenes Women’s Representative Manuel Lopes Pereira Suco Member 60 Madalena da P. do Rego PNDS Maria Pereira da Cruz Women’s Representative Amaro X. Pereira EIP Constancio Jose do Rego Xefe Suco Marcelino Pereira Secretary of EIP Domingos Manuel da C. Pereira PAAS Domingos do Rego Soares Xefe Aldeia Government Stakeholders Organization Name Position 1 Ministry of State Administration Sr Antonio Augusto Guterres Baucau Municipal Administrator 2 Court of Appeal Team Sebastiao Soares MARCUS Deputy Court of Appeal Team Secretary 3 Ministry of State Administration Agustinho P. Da Silva Baucau Municipal DNAAS Focal point 4 Secretary of State for the Economic Empowerment of Women Domingas Soares Nunes Baucau Legal Aid Provider, Baucau 1 Justice and Peace Commission (JPC), Baucau Fr. Julio Crispin XIMENES BELO Director 2 Marcal (Asala) XIMENES Program Coordinator 3 Horta RAMOS Lawyer 4 Zelia CORREIA Finance/admin officer Covalima Field Work Government Stakeholders Organization Name Position 1 Ministry of State Administration Sr Jose Pina Cardoso Covalima Municipal Administrator 2 Ministry of State administration Afonso Noguera Nahak Focal Point DNAAS/new Secretary of Municipal New title is Municipal secretary. 3 SEM Francisco Tranjanus G. de Araujo Gender Focal Point for SEM, formerly SEPI Suco Council Member Representatives, Covalima Suco Name Position Ogues, Maucatar Ms. Natalia Cardoso Women Representative Lalawa, Tilomar Ms. Felizada Moniz da Costa Women Representative Maudemo, Tilomar Mrs. Maria Fatima Suco Chief 61 ANNEX V: EVALUATION QUESTION MATRIX The following questions highlight the leading questions that were used to shape the evaluation and ensure that data was collected in a way that allowed Mission and USG priorities to be addressed for data collection and analysis. Lead questions were followed by sub-questions to guide the interview and discussion process and ensure that respondents were provided with enough time and detail to form their perspective and response. Each activity and component area of the Ba Distrito program was also analyzed to assess possible attribution and contribution and also ensure that respondents were knowledgeable of USG inputs and resource contributions. Evaluation questions, sub questions, and data collection methods were noted as well as indications that were used to provide evidence of attribution and/ or contribution to impact. INTERVIEW KEY: METHODS INTERVIEW QUESTIONS Coded based on role in Ba Distrito Program METHOD NOTES KII, SGD, or FGD 1. Primary: National Field Staff and Managers and Implementing Partners (P-FD) KII, SGD, FGD 1. Document all staff who “decline” an interview 2. Ensure that category 1-3 are triangulated with follow-up questions in Timor-Leste and in DC with other USG staff and in SGDs. 3. Coded categories 1-3 are weighted as these staff had primary tasks relating to activity outcome and impact. 2. Primary: Strategy Planning (P-SP) KII 3. Primary: Strategy Decision and Approval (P-DM) KII, SGD 4. Secondary: Strategy Review only (S￾SR) KII, SGD 5. Secondary: Stakeholder key component/activity area (S-S) KII, SGD 6. Peripheral: USG, External (donor, NGO, etc.) (PP) KII CODING QUESTION DATA COLLECTION METHOD P-FD, P-SP, S-S QUESTION 1 What are the main challenges faced by the project and lessons-learned to-date? KIIs and Document Review P-FD, P-SP, S-S, P-DM Sub Question 1.1 Was the original program design appropriate for addressing the projects goals and objectives? KIIs and Document Review P-FD, P-SP, S-S, P-DM Sub Question 1.2 Were project activities and implementation influenced by the baseline results and data collected? KIIs, Document Review and Household Survey Analysis P-FD, P-SP, S-S, P-DM, PP, and S-S Sub Question 1.3 If so, which activities? Please provide specific examples and geographic focus area? KIIs, SGDs, and Document Review CODING QUESTION DATA COLLECTION METHOD P-FD, P-SP, S-S Question 2 What changes, if any, are proposed regarding implementation? KIIs, Document Review and Household Survey Analysis P-FD, P-SP, S-S, P-DM Sub Question 2.1 What changes, if any, seem necessary at this point in the implementation of the Ba Distrito project? KIIs, Document Review and Household Survey Analysis P-FD, P-SP, S-S, P-DM Sub Question 2.2 If so, which activities? Please provide specific examples? KIIs, SGDs, FGDs, and Document Review 62 P-FD, P-SP, S-S, P-DM, and PP Sub Question 2.3 Are these changes specific to a particular municipality area? If so, why? Please provide specific examples. KIIs, SGDs, FGDs, and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 2.4 Are these changes specific to a particular target group? If so, how come? Please provide specific examples. KIIs, SGDs, FGDs, and Document Review CODING QUESTION DATA COLLECTION METHOD P-FD, P-SP, S-S, P-DM, and PP QUESTION 3 What, if any, unintended consequences have occurred as a result of the project? KIIs, SGDs, FGDs, and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 3.1 Have staff or beneficiaries noted any particular “unintended” consequences as a result of Ba Distrito activities? KIIs, SGDs, FGDs, and Document Review P-FD, P-SP, SS, P-DM Sub Question 3.2 How have these “unintended consequences” impacted particular component areas? Which ones? Please provide examples. KIIs, SGDs, FGDs, and Document Review P-FD, P-SP, SS, P-DM Sub Question 3.3 How have these “unintended consequences” impacted particular target groups? Which ones? Please provide examples. KIIs, SGDs, FGDs, and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 3.4 Are their specific component areas that seem to have had greater impact than others? Which ones? Please provide specific examples. KIIs, Document Review and Household Survey Analysis CODING QUESTION DATA COLLECTION METHOD P-FD, P-SP, S-S, P-DM, and PP QUESTION 4 How has Ba Distrito incorporated gender sensitivity into their programming, operations, and management? KIIs and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 4.1 What specific Ba Distrito activities have targeted this? KIIs and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 4.2 How has this been monitored at the project level? KIIs and Document Review P-FD, P-SP, S-S, P-DM, and PP Sub Question 4.3 What Ba Distrito activities have been most effective in this? KIIs and Document Review 63 ANNEX VI: BA DISTRITO 2016 HOUSEHOLD SURVEY QUESTIONNAIRE INTERVIEW CODING District: ☐ Baucau ☐ Covalima ☐ Ermera ☐ Liquica ☐ Oecusse Suco: Baucau Covalima Ermera Liquica Oecusse Suco Fatulia Suco Beco Suco Eraulo Suco Darulete Suco Bobometo Suco Lacoliu Suco Fohoren Suco Estado Suco Fahilebo Suco Malelat Suco Loilubo Suco Lalawa Suco Lauana Suco Lauhata Suco Naimeco Suco Samalari (Laga) Suco Maudemo Suco Lemia Leten Suco Leorema Suco Suni-Ufe Suco Soba Suco Ogues Suco Malabe Suco Leotela Suco Usitacae Suco Triloca Suco Raimea Suco Matata Suco Lissadila Suco Usi-Taqueno Suco Uatolari Suco Ponilala Suco Maubara Lissa Suco Railaco Leten Suco Vaviquinia Aldeia: Date of Interview: Interview code: Enumerator code: Supervisor code: Time interview began: _ _:_ _ Time interview ended: _ _:_ _ INTRODUCTION “Greetings, I am from Social Science Dimensions, an independent research organization. We are conducting an opinion survey among people like you to find out your views on issues of public interest. This is an independent survey about suco councils and development in our country. I am interested in your opinion. Your answers will be kept entirely confidential, your name will not be given to anyone and your views will be analyzed along with those of hundreds of others.” IDENTIFICATION QUESTIONS Gender (Do not ask): ☐ Male ☐ Female Age at last birthday: ☐ 16-30 ☐ 31+ Telephone Number:………………………. What is the highest level of education you have completed: ☐ Not yet completed primary school ☐ Completed Primary School 64 ☐ Completed Secondary School ☐ Completed University ☐ No answer Occupation: ☐ Farmer ☐ Employed by the government ☐ Employed by the private sector ☐ Operate your own business ☐ Student ☐ Other ☐ No work of any kind (unemployed) ☐ No answer Does the respondent have a physical and/or intellectual disability (indicate both if appropriate) that hinders them in their work or home life: ☐ Physical disability ☐ Intellectual disability ☐ No disability Religion: Could you please indicate your religion. ☐ Catholic ☐ Protestant ☐ Muslim ☐ Other (please state)……………………………………………………………. ☐ No religion ☐ No answer Ethnicity/Language: Are you a member of an ethnic/linguistic minority? (someone with a different ethnic/cultural/language background to the majority in the suco) ☐ Yes ☐ No ☐ No answer THEME 1: CITIZEN’S UNDERSTANDING OF THEIR ROLE IN POLITICAL PROCESSES 1. Thinking about the suco where you live, how would you rate the following? Feature Level of satisfaction Excellent Very good Not so good Poor Don’t know 1A. General living conditions 1B. Quality of housing 1C. Quality of school buildings and facilities 1D. Availability of health services 1E. Accessibility of water 1F. Roads 1G. Security from violence 1H. Education opportunities 1I. Job opportunities 2. Thinking about the involvement of different groups in decisions on development priorities in the suco, please indicate your level of agreement with the following statements? Statement Level of agreement with statement 65 Strongly agree Agree Disagree Strongly disagree Don’t know 2A. It is important that elected suco council officials make decisions about development in the suco 2B. It is important that traditional leaders make decisions about development in the suco 2C. It is important that all members of the community are involved in decisions about development in the suco 2D. It is important for men and women to be equally involved in decisions about development in the suco 2E It is important for youth and adults to be equally involved in decisions about development in the suco. 2F. It is important for people living with disabilities to be equally involved in decisions about development in the suco. 2G It is important for members of ethnic and religious minorities to be equally involved in decisions about development in the suco. 3. Over the past year, have you participated in meetings organized by suco council officials to discuss suco development priorities? (a) Yes (b) No 4. Thinking about meetings to discuss development in the suco over the past year, please indicate your level of agreement with each of the following statements. Statement Level of agreement with statement Strongly agree Agree Disagree Strongly disagree Don’t know 4A. I feel I have the right to attend suco meetings 4B. I feel I have the opportunity to have my views heard at meetings organized by suco council officials to discuss suco development priorities 4C. I feel that I can influence suco development priorities by participating in meetings at the suco level. 5. Thinking about development activities sponsored by your suco council, please indicate your level of agreement with each of the following statements: Statement Level of agreement with statement Strongly agree Agree Disagree Strongly disagree Don’t know I am happy to spend some of my time every month working on suco development projects for suco council authorities without any pay. 6. Do you intend to vote in the next suco election? (a) Yes (Go to Q.7 below) (b) No (Go to Q.8 below) (c) No answer / Undecided (Go to Q.9 below) 7. (NOTE SKIP POSSIBILITY) (If “Yes” to Q.6) Please indicate the reasons (as many as applicable) why you intend to vote in the next suco elections. (Enumerator to tick off ALL stated reasons, but NOT to list any reasons in advance or help the respondent.) 66 ☐ Because we fought for independence. ☐ Because I have the right to vote. ☐ Because I want to choose (or change) our leaders. ☐ Because I am a member of a political party. ☐ Because I want to choose or change the development outcomes/priorities in the suco. ☐ Because it is my duty as a citizen (democracy). ☐ My friends and family vote, so I vote. ☐ The church tells me to vote. ☐ The national leaders tell me to vote ☐ The local leaders tell me to vote. ☐ Teachers from educational institutions (school, university, technical college, etc.) tell me to vote. ☐ Other 8. (NOTE SKIP POSSIBILITY) (If “No” to Q.6) Please indicate the reasons (as many as applicable) why you will not vote in the next suco elections. (Enumerator to tick off ALL stated reasons, but NOT to suggest any reasons in advance or help the respondent. Enumerator NOT to accept non-registration as reason.) ☐ Would have to go to / return to another suco to vote ☐ Polling booth too far away ☐ Physical limitations (pregnant, frail, elderly, disabled or sick) ☐ No time because of domestic work commitments (looking after children, cleaning, cooking, etc.) ☐ No time because of other work commitments (i.e. farm work) ☐ Previous negative experience when voting (at the polling station) ☐ Because of security issues / situation ☐ Because friends and/or family don’t vote ☐ Not old enough to vote ☐ Because it makes no difference to their life who is elected to suco council ☐ Other THEME 2: CITIZEN’S KNOWLEDGE AND AWARENESS OF DECENTRALIZATION IN TIMOR-LESTE (INCLUDING SUCO LEGISLATION) Now I would like to ask you some questions about decentralization plans in Timor-Leste. 9. Have you heard about the Timor-Leste Government’s plan to develop and implement legislation on decentralization or pre-deconcentration? (a) Yes (Go to Q. 10 below) (b) No (Go to Q. 12 below) (c) No answer (Go to Q. 12 below) 10. (NOTE SKIP POSSIBILITY) (If “Yes” to Q.9) Have you attended any briefings (at suco, sub-district or district level), concerning the Government’s decentralization or pre-deconcentration plans? (a) Yes (b) No (c) No answer 11. (NOTE SKIP POSSIBILITY) (If “Yes” to Q.9 and Q. 10 above) Have you had the opportunity to provide any input into draft legislation relating to decentralization or pre-deconcentration (for example a meeting in the suco, sub-district or district)? (a) Yes (b) No (c) No answer. THEME 3: CITIZEN’S KNOWLEDGE OF THE ROLES AND RESPONSIBILITIES OF SUCOS, AND THEIR EXPECTATION OF AND SATISFACTION WITH SUCO SERVICE PROVISION 67 Now I would like to ask you some questions about suco councils 12. Could you please list the responsibilities you know about that suco leaders and suco councils are required by state law to perform. (Enumerator to tick off ALL listed responsibilities, but NOT to list any responsibility in advance or to help the respondent) ☐ Peace and social harmony (including facilitating the resolution of minor disputes and calling in security forces when necessary) ☐ Domestic violence problems (prevent domestic violence, help victims, punish perpetrators, etc.) ☐ Population census and registration activities ☐ Civic education ☐ Promotion of the official RDTL languages ☐ Economic development ☐ Food safety considerations ☐ Environmental protection ☐ Education, culture and sports ☐ Maintenance of social infrastructure/public works (eg. housing, schools, health centers, water systems, roads, communications) ☐ Financial reporting tasks ☐ Annual development plan ☐ Promote equality ☐ Promote resect for the sucos customs and traditions ☐ Cooperate with the development plans of other levels of government. 13. Please indicate how effective you think your suco authorities are at each of the following tasks: Task Level of Effectiveness Very effective Effective Not very effective Don’t know 13A. Local development planning (planning new roads & social services, etc.) 13B. Building and maintaining infrastructure (for example buildings, drains and roads) 14. How effective are the dispute resolution / mediation services provided in your suco? (a) Very effective (b) Effective (c) Not effective (c) Don’t know 15. Are there any services helping people affected by Gender Based Violence (for example domestic violence cases where people are hit by members of their own household, such as a woman hit by her husband) provided in your suco? (a) Yes (Go to Question 16 below) (b) No (Go to Question 17 below) (c) Don’t know (Go to Question 17 below) 16. (NOTE SKIP POSSIBILITY) (If “Yes” to Q. 15. above) How effective do you consider these services helping those affected by Gender Based Violence? (a) Very effective (b) Effective (c) Not effective (c) Don’t know THEME 4: CITIZEN’S ACCESS TO JUSTICE Now, I would like to ask you some questions about access to justice in Timor-Leste. 68 17. How much information (posters, pamphlets, etc.) is available in your suco about the district court system, including how to access the courts? ☐ A lot of information ☐ Some information ☐ No information ☐ No answer 18. Thinking about the court system, please indicate your level of agreement with the following statements. Statement Level of agreement with statement Strongly agree Agree Disagree Strongly disagree Don’t know 18A. The court system is fair. 18B. The court system is fast. 18C. The court system is easy to access. 18D. The court system is much better that it was two years ago. 18E. The court system equally respects the human rights of men, women and youth 18F. The court system respects the rights of people living with disabilities and minorities (ethnic, religious and sexual) 18G. The staff of the courts are hard-working officials dedicated to improving justice in the community. 19. Thinking about suco justice and dispute resolution systems, please indicate your level of agreement with the following statements. Statement Level of agreement with statement Strongly agree Agree Disagree Strongly disagree Don’t know 19A. It is important for suco justice / dispute resolution forums to increase their respect for the equal rights of women 19B. Suco justice / dispute resolution forums respect human rights more now than they did two years ago. 20. Which justice / dispute resolution option (suco-forums or the courts) is best for the following outcomes? Outcome Suco forum Courts Don’t know 20A. Which provides a faster outcome? 20B. Which provides a fairer outcome? 20C. Which is easiest to access (transport)? 20D. Which is easiest to use? 20E. Which provides the best outcome for women? 20F. Which provides the best outcome for the community? 20G. Which shows more respect for traditional values? 20H. Which shows most respect for members of minority groups (sexual orientation, religious and ethnic background). 20I. Which provides the best outcome for people living with disabilities? 69 21. Have you ever received any information on who you can talk to if you need help with a justice / dispute resolution matter (legal aid information)? ☐ Yes (Go to Q. 22 below) ☐ No (Go to Q. 23 below) ☐ Don’t know (Go to Q. 23 below) 22. (NOTE SKIP POSSIBILITY) (If “Yes” to Q.21 above) Please indicate the sources of any legal aid information you have received (as many options as appropriate). ☐ Suco meeting, gathering or ceremony ☐ PNTL/police ☐ Suco council member ☐ Community theatre ☐ Radio program ☐ SMS message ☐ Briefing at a health clinic, school, or other government agency ☐ Church ☐ TV program ☐ LBH/Legal aid organization ☐ Other ☐ Don’t know 23. Thinking about the treatment of women and girls, please indicate your agreement with the following statements. Statement Level of agreement with statement Strongly agree Agree Disagree Strongly disagree Don’t know 23A. Hitting women and girls is important to make sure that women and girls perform their duties as wives and daughters. 23B. Hitting women and girls is a crime, and prevents women and girls from reaching their potential and contributing fully to the development of the nation. 23C. There is less hitting of women and girls in the community than there was in the past. 24. Thinking about justice and dispute resolution, have you used any of the following: Yes No 24A. District courts 24B. Suco level justice and dispute resolution forums 24C. LBH (lawyer) 24D. Public defender Now I would like to ask you some questions about how you obtain news about Timor-Leste. THEME 5: CITIZEN’S SOURCES OF INFORMATION 25. Where do you obtain most of your information about Timor-Leste news? (One option only) ☐ Newspaper ☐ TVTL Television ☐ Indonesian Television ☐ RTL Radio ☐ Community / local radio ☐ Websites ☐ Social media (Facebook, Twitter, etc.) ☐ Friends/Family 70 ☐ Educational institution (school/university) ☐ Suco councils ☐ Other 26. What time of day do you most often listen to the radio? (One option only) ☐ Morning ☐ Lunch-time ☐ Afternoon ☐ Night ☐ Never (Go to Q.29) ☐ None of the above 27. (NOTE SKIP POSSIBILITY) (For all those who indicated that they listen to the radio in Q. 26 above) Please list any community radio stations you listen to (as many as appropriate). (Enumerator to tick off ALL community radio stations listed by respondent, but NOT to list any stations in advance or to help the respondent) Municipality Station name Frequency Tick/Check Box Baucau (at Suku Bucoli) Radio Popular Colelemai Bucoli 102.5FM Baucau Radio Lian Matebian Baucau 99.9FM Covalima Radio Cova Taroman 94.5FM Ermera Radio Cafe Ermera 92.3FM Liquiça Radio Tokodede 92.3FM Oecusse Radio Atoni Lifau 93.3FM Dili Radio Lorico Lian 100.5FM Dili Radio Rakambia 99.5FM Manatuto Radio Ili Wai 96.1FM Viqueque Radio Povu Viqueque 97.9FM Lautem Radio Komunidade Lospalos 100.1FM Aileu Radio Rai Husar 97.1FM Ainaro (at Maubisse) Radio Mauloko 89.7FM Ainaro Radio Lian Tatamailau 98.1FM Bobonaro Radio Komunidade Maliana 91.7FM Manufahi Radio Don Buaventura 95.1FM Respondent cannot list a community radio station or does not listen to community radio (Go to Q. 29 below) 28. (NOTE SKIP POSSIBILITY) (If the respondent indicated one or more community radio stations in Q. 27 above) How often do you listen to community / local radio? ☐ Daily ☐ Weekly ☐ Monthly ☐ Other THEME 6: SUCO ADMINISTRATION I now have one final question on suco administration. 29. Thinking about your suco council, do you consider that it operates free from KKN (Korupsi, Kolusi & Nepotisme)? ☐ Yes ☐ No 71 ANNEX VII: HOUSEHOLD SURVEY DATA See separate annex document. ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Sarah Wood Title Monitoring and Evaluation Specialist Organization Social Impact Evaluation Position? Team Leader Evaluation Award Number AID-486-I-14-00001/AID-472-TO-16- 00001 USAID Project(s) Evaluated Mid Term Evaluation of the U.S. Agency for International Development’s Timor-Leste Ba Distrito Program I have real or potential conflicts of interest to disclose. Yes If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 30, 2016 72 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523