September 2014 Printing of this publication is made possible by the generous support of the American people through the United State Agency for International Development's (USAID) “Monitoring and Evaluation of the Timor-Leste MCC Threshold Program for Anti￾Corruption in Timor-Leste,” (TACME) implemented by Social Impact, Inc. The contents and opinions expressed herein are the responsibility of TACME and do not necessarily reflect the views of USAID or the United States Government. PERFORMANCE EVALUATION Final Performance Evaluation of the Fostering Transparency Initiative Program in Timor-Leste ii FINAL EVALUATION OF THE FOSTERING TRANSPARENCY INITIATIVE IN TIMOR￾LESTE AN MCC ANTI-CORRUPTION THRESHOLD PROGRAM September 2014 iii CONTENTS ACRONYMS ....................................................................................................................................................................... v ACKNOWLEDGEMENTS ............................................................................................................................................ vii EXECUTIVE SUMMARY ............................................................................................................................................... viii EVALUATION PURPOSE & QUESTIONS ................................................................................................................. 1 COUNTRY CONTEXT .................................................................................................................................................. 2 EVALUATION METHODS & LIMITATIONS ............................................................................................................ 3 METHODOLOGY ........................................................................................................................................................ 3 LIMITATIONS ................................................................................................................................................................ 4 FINDINGS AND CONCLUSIONS .............................................................................................................................. 4 1. PROGRAM LOGIC ...................................................................................................................................................... 4 A. WERE THE ORIGINAL SIX PROGRAM COMPONENTS RELEVANT TO THE TIMORESE CONTEXT? (E.G. WERE THEY BASED ON INITIAL ANALYSIS OF ASSUMPTIONS, RISKS, AND EXTERNAL FACTORS?)............................................................................................................................................. 4 FINDINGS .................................................................................................................................................................. 4 CONCLUSIONS ...................................................................................................................................................... 5 B. WERE ANY CHANGES MADE TO THE PROGRAM LOGIC DURING THE IMPLEMENTATION? IF SO, WERE THEY APPROPRIATE? ............................................................................. 6 FINDINGS .................................................................................................................................................................. 6 CONCLUSIONS ...................................................................................................................................................... 6 2. ACTIVITIES DESIGNED AND IMPLEMENTED & NEW ACTIVITIES ............................................................ 7 A. WERE PLANNED ACTIVITIES ACTUALLY IMPLEMENTED? WHY OR WHY NOT? ................... 7 B. WERE ANY NEW ACTIVITIES DESIGNED AND IMPLEMENTED? IF SO, WHY? .......................... 7 FINDINGS—COMPONENT A: CAC ................................................................................................................ 7 FINDINGS—COMPONENT B: OPG ............................................................................................................... 10 FINDINGS—COMPONENT C: AUDIT .......................................................................................................... 10 FINDINGS—COMPONENT D: PROCUREMENT ....................................................................................... 11 FINDINGS—COMPONENT E: PARLIAMENT .............................................................................................. 12 FINDINGS—COMPONENT F: NON-STATE ACTORS ............................................................................13 CONCLUSIONS: PLANNED ACTIVITIES ...................................................................................................... 14 CONCLUSIONS: NEW ACTIVITIES ................................................................................................................ 15 A. WHAT WAS THE EFFECTIVENESS OF ACTIVITIES? ............................................................................ 16 FINDINGS—COMPONENT A: CAC .............................................................................................................. 16 FINDINGS—COMPONENT B: OPG ............................................................................................................... 18 FINDINGS—COMPONENT C: AUDIT .......................................................................................................... 18 iv FINDINGS—COMPONENT D: PROCUREMENT ....................................................................................... 19 FINDINGS—COMPONENT E: PARLIAMENT .............................................................................................. 20 FINDINGS—COMPONENT F: NON-STATE ACTORS ............................................................................ 21 CONCLUSIONS ....................................................................................................................................................22 3. INFLUENCE ON CHANGES IN KNOWLEDGE, ATTITUDES, AND BEHAVIORS ............................... 24 DID IMPLEMENTED ACTIVITIES LEAD TO MEASURABLE OUTCOMES, SUCH AS CHANGES IN KNOWLEDGE, ATTITUDES, OR BEHAVIOR? ................................................................................................. 24 FINDINGS ................................................................................................................................................................ 24 CONCLUSIONS .................................................................................................................................................... 25 4. SUSTAINABILITY & LESSONS LEARNED ........................................................................................................... 26 A. HAVE KEY IMPLEMENTED ACTIVITIES BEEN SUSTAINED? .................................................................. 26 FINDINGS ................................................................................................................................................................ 26 CONCLUSIONS & LESSONS LEARNED ........................................................................................................ 27 B. WHAT IS THE LIKELIHOOD FOR FUTURE SUSTAINABILITY? ........................................................ 27 FINDINGS ................................................................................................................................................................ 27 CONCLUSIONS .................................................................................................................................................... 29 RECOMMENDATIONS ................................................................................................................................................. 29 ANNEXES ......................................................................................................................................................................... 32 ANNEX I: EVALUATION STATEMENT OF WORK ....................................................................................... 32 ANNEX II: DATA SOURCE MATRIX, EVALUATION METHODS, AND LIMITATIONS ..................... 40 ANNEX III: DATA COLLECTION INSTRUMENTS ......................................................................................... 52 ANNEX IV: SOURCES OF INFORMATION ...................................................................................................... 55 ANNEX V: FOTI RESULTS FRAMEWORK ......................................................................................................... 59 ANNEX VI: FOTI ORIGINAL COMPONENT ACTIVITIES ............................................................................ 61 v ACRONYMS AA Assistance Agreement AC Anti-Corruption ACC Anti-Corruption Commission (also CAC, below) Casals Casals & Associates, DynCorp International CAC Comissão Anti-Corrupção [Commission on Anti-Corruption] (also ACC, above) CCN Cooperating Country National CEPAD Center of Studies for Peace and Development COP Chief of Party COR Contracting Officer’s Representative CP Condition Precedent CPP Corruption Prevention Plan CSO Civil Society Organizations FIU Financial Intelligence Unit FOTI Fostering Transparency Initiative GOPAC Global Organization of Parliamentarians against Corruption HATAC High Administrative Tax and Audit Court IG Inspector General JISP Justice Institutions Strengthening Program KII Key Informant Interview M&E Monitoring and Evaluation MAT Midterm Assessment MCC Millennium Challenge Corporation M&E Monitoring and Evaluation MOF Ministry of Finance MOU Memorandum of Understanding MP Member of Parliament NGO Non-Governmental Organization NP National Parliament OECD-DAC Organisation for Economic Co-operation and Development-Development Assistance Committee OIG Office of Inspector General OPG Office of the Prosecutor General OPM Office of the Prime Minister vi PG Prosecutor General PNDS Program for Accelerating Suco Development RDTL Democratic Republic of Timor-Leste RRT Rating Results Tables SEFI Secretariat of State for Institutional Strengthening SI Social Impact SIAK Sistema Informasaun Anti-Korrupsaun SOP Standard Operating Procedures SOW Statement of Work TNA Training Needs Assessment TOT Training of Trainers TPA Threshold Program for Anti-Corruption UN United Nations UNCAC United Nations Convention Against Corruption USAID United States Agency for International Development USG United States Government vii ACKNOWLEDGEMENTS The team wishes to extend its gratitude to the many people who made this work possible. First, the team thanks our current USAID/Timor-Leste Contracting Officer’s Representative (COR), Ana Guterres for her assistance throughout the course of this effort, from the design of the evaluation, to the data collection, analysis, and report writing. Her consistent support over the past few months has been critical to the success of this work. The team also thanks Lisa Whitley of USAID/Timor-Leste for her valuable inputs along the way. In addition, the team is indebted to our former COR, James Coy, for his willingness to provide detailed historical information and rich contextual information that helped us understand and interpret the story of this FOTI program. Finally, the team also wishes to thank Ms. Berta Heybey for her inputs in the design of the evaluation to make ensure that the report also serves MCC’s needs. Next, the team thanks the various staff members from Casals & Associates that helped us to collect all the relevant information through various rounds of phone calls, emails, and perusal of program files for the information needed to inform our analysis of the program. In particular, former COP Eduardo Flores-Trejo’s commitment to the success of both the program and our evaluation was of critical importance. Likewise, the team is grateful for the inputs of his current and former colleagues, particularly the former FOTI advisors who graciously offered to speak with us months and, in some cases years, after the conclusion of their assignments. The team also wishes to thank and express its appreciation to all of the FOTI counterparts both within and outside of the Government of Timor-Leste who met with us over the course of its interviews and arranged for us to speak with their colleagues. This evaluation would not have been possible without them and the team benefited greatly from their time and thoughtful responses. The team would like to thank its interpreter, driver, and logistics coordinator, Pedro Da Silva who graciously filled these dual roles, arranging interviews and drawing upon his networks and contextual knowledge to ensure that the team met with as many relevant stakeholders as possible, and also helped to ensure that the team came away with an appropriate understanding of the context and the players. viii EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This final performance evaluation is intended to provide an objective, rigorous, and evidence-based analysis of the Fostering Transparency Initiative’s (FOTI) overall performance. Specifically, the evaluation will look to identify, review, and assess four key aspects by utilizing the following evaluation questions: The primary evaluation questions include: 1. Program Logic a.Were the original six program components relevant to the Timorese context? (e.g. were they based on initial analysis of assumptions, risks, and external factors?) b.Were any changes made to the program logic during the implementation? If so, were they appropriate? 2. Activities Designed and Implemented a.Were planned activities actually implemented? Why or why not? b.Were any new activities designed and implemented? If so, why? c.What was the effectiveness of activities? 3. Influence on Changes in Knowledge, Attitudes, and Behaviors a. Did implemented activities lead to measurable outcomes, such as changes in knowledge, attitudes, or behavior? 4. Sustainability & Lessons Learned a. Have key implemented activities been sustained? b.What is the likelihood for future sustainability? c. What are the key aspects of FOTI that should be continued in the near term? d. Which aspects of FOTI would be appropriate to replicate in other contexts? PROJECT BACKGROUND In September 2010, the Millennium Challenge Corporation’s (MCC) Board of Directors approved the Democratic Republic of Timor-Leste’s (RDTL) proposal for a $10,496,000 Threshold Program1. The Timor-Leste Threshold Program was funded by MCC with oversight provided by its partner, the United States Agency for International Development (USAID). Implemented by Casals & Associates (Casals), FOTI focused on Component I of the Threshold Program, Strengthening Anti-Corruption (AC) Efforts, and accounts for $6,961,340 of the total Threshold Program’s budget. The program worked to reduce corruption and emphasized building human and institutional capacity. FOTI was divided into six major components: (A) Anti-Corruption Commission (CAC); (B) Office of the Prosecutor General (OPG); (C) Internal and External Audit; (D) Procurement; (E) National Parliament; and (F) External Watchdog and Public Advocacy. The program operated from February 1, 2011 to March 31, 2014. 1 See “Timor-Leste Threshold Program,” MCC Website. http://www.mcc.gov/pages/countries/program/timor-leste-threshold-program. Accessed 4/16/2013. ix COUNTRY CONTEXT The Portuguese colonial era from 1500 to 1974 provided the Timorese with virtually no experience or capacity in self-governance. Shortly after Portugal’s change in government and subsequent declaration of independence by Timor-Leste, Indonesia invaded and ultimately incorporated the country as a province. The Indonesian period from 1975-1999 was characterized by brutality, and a pattern of corruption instigated by the occupying power, the Indonesian military. Resistance movements were active in rural areas. Some Timorese went into exile. In 1998, 78% of the Timorese electorate voted for independence during a United Nations (UN)-sponsored referendum. Pro-Indonesia militias launched a systematic scorched-earth campaign2 . The UN administered the country until its restoration of independence in 2002. Donors provided thousands of advisors to help the RDTL keep the peace, promote transitional justice, reconstruct infrastructure and establish new institutions. Capacity building was slow, as human resources were a significant constraint. Lack of a common language and a poor understanding of corruption also have slowed progress. Geographic and social divisions plagued the country. Elections held in 2007 and 2012 allowed a transfer of power. The government relies on income from the Timor-Leste Petroleum Fund, supplemented by donor funding, for almost all of its revenue. The almost total absence of tax revenue creates problems of government accountability to Timorese citizens. A small private sector, weak civil society institutions, and the absence of a culture of investigative journalism limit the ability of the population to check government abuses. EVALUATION METHODS & LIMITATIONS The mixed methods evaluation was carried out by a four-member team from Social Impact, Inc. during field visits in February and subsequently in June-July 2014. The evaluation team planned to utilize the following methodologies: document review; technical evaluation of “deliverables”; pre- and post-test data analysis; key informant interviews; and observations during a field visit. It proved impossible to utilize all of the proposed research methodologies due to logistical and bureaucratic realities in Timor-Leste. Specific methodological limitations included: challenge in measuring qualitative outcomes; logistical and timing challenges; recall and attribution biases; selection bias; and the use of an interpreter. Despite the limitations, the team feels that the information presented in this report is an accurate depiction of the FOTI program. FINDINGS & CONCLUSIONS PROGRAM LOGIC: WERE THE ORIGINAL SIX PROGRAM COMPONENTS RELEVANT TO THE TIMORESE CONTEXT? (E.G. WERE THEY BASED ON INITIAL ANALYSIS OF ASSUMPTIONS, RISKS, AND EXTERNAL FACTORS?) The Threshold Program for Anti-Corruption (TPA), selected six relevant components focusing on important AC actors within the RDTL; however, the design was predicated on assumptions that did not hold true. Generally speaking, low capacity, competing organizational priorities, and lack of leadership buy-in negatively affected program activities. The TPA design was decidedly ambitious in its assumptions about how receptive institutions would be to receiving TPA assistance. The design overestimated the program’s political capital, particularly regarding its objective of increasing cooperation among 2 "Indonesia/East Timor: Forced Expulsions to West Timor and the Refugee Crisis". Human Rights Watch. December 1999. x organizations. The TPA’s emphasis on investigation among several institutions, while justified by global best practices, was especially challenging in the Timorese context. In some cases, the design relied too heavily on external dependencies (e.g. establishment of institutions) that were difficult to manage or predict. WERE ANY CHANGES MADE TO THE PROGRAM LOGIC DURING THE IMPLEMENTATION? IF SO, WERE THEY APPROPRIATE? All of the inconsistencies in program design necessitated slight modifications to the program logic. However, the program was—at least in the case of the corruption survey—able to take corrective actions that made it more responsive to the target institution’s priorities. If the program had a better sense of the needs and desires of the target organizations at the outset, it seems likely that the activities would have launched more quickly and decreased the likelihood of needing to substantially modify the Scope of Work. ACTIVITIES DESIGNED AND IMPLEMENTED & NEW ACTIVITIES: WERE PLANNED ACTIVITIES ACTUALLY IMPLEMENTED? WHY OR WHY NOT? WERE ANY NEW ACTIVITIES DESIGNED AND IMPLEMENTED? IF SO, WHY? FOTI delivered a broad suite of activities to build corruption fighting capacity, in the form of routine coaching and mentoring, often delivered on an ad hoc basis. Factors that influenced the extent to which planned activities did or did not occur vary by institution:  CAC- FOTI primarily supported the Investigations Unit with the placement of a long-term advisor as well as limited amount of formal, external trainings. The emphasis of support shifted considerably from what was originally envisioned from the Prevention and Investigations Units to the Education, Outreach, and Research Unit due to evolving needs of and relationships with the CAC.  OPG- Despite some initial setbacks in launching the activities, the final evaluation team found that FOTI conducted a significant amount of work in line with what was originally envisioned for the component once an advisor with sufficient relevant experience was hired.  Audit- Most of the anticipated work was unable to occur due to an overly ambitious condition precedent regarding the establishment of the external audit authority. Instead, FOTI supported the Office of the Inspector General (OIG) with the hiring of a long-term advisor.  Procurement- Due to overambitious program design, this component did not operate as envisioned although the program managed to conduct a few of the originally planned activities, primarily with the CAC Prevention Unit.  Parliament- FOTI’s work with Parliament suffered during the first half of the program. In the second half of implementation, the program successfully shifted to supporting the development and enactment of AC legislation.  Non-state actors- FOTI supported few activities that contributed to Component F’s two original objectives, but did complete a variety of new activities in a civic education campaign. EFFECTIVENESS OF ACTIVITIES FOTI assistance built the capacity of staff at target organizations to detect, investigate, and/or punish corruption cases through various means. Insufficient baseline data is available to determine the precise level of capacity built. Limited pre- and post-test scores indicate successful knowledge transfer; however, there is some concrete evidence to confirm these findings. Effectiveness varied by component:  CAC- Assistance helped staff at CAC to build capacity in their lines of work. FOTI provided formal and informal assistance to the Investigation Unit to build skills in corruption investigation and also by contributing to the development of a case management system; the program xi strengthened the capacity of the EOR unit by assisting them to conduct outreach work and develop the ability to design and conduct three corruption surveys; FOTI aided the Prevention Unit to educate RDTL institutions and the public on corruption, particularly in procurement, and also about the CAC itself.  OPG- FOTI trainings enhanced capacity to investigate corruption crimes. The lack of a protocol for coordination between CAC and OPG contributed to distrust, although inter-agency trainings did make progress in building relationships with other stakeholders.  OIG-Despite the inability to help establish an external audit authority due to its non-operational status, the component was effective insofar as it contributed to the overall purpose of increasing audit capacity of the OIG staff.  Procurement- The program was unable to develop a procurement certification course and deliver it to relevant RDTL staff as envisioned, nor was it able to assist with the establishment of the Procurement Review Commission. However, it did succeed in helping counterparts to develop the ten principles of procurement and a training on corruption in public procurement, both of which were rolled out to various institutions within the RDTL.  Parliament- There is insufficient data to comment on the effectiveness of the assistance provided during the first two years of implementation. The quantity and quality of the assistance increased substantially in the final year, which led to some legislative victories, including the amendments to Act 17/2011 and the decree law of the Financial Intelligence Unit (FIU).  Non-state actors- Work with journalists achieved limited success due to the relatively small amount of programming in this component, as well as externalities, such as fear of retribution for publishing investigative stories. The civic education campaign developed with the Center of Studies for Peace and National Development (CEPAD) appears strong, but it is too early to judge actual effectiveness of the campaign. The beginning of the program was plagued by poor quality of several FOTI staff charged with forging relationships with counterparts. All of these individuals were replaced by the end of the program and subsequent staff received high quality ratings from counterparts. The technical content of trainings, manuals, and other deliverables was judged to be highly relevant to the ongoing work of partner institutions. INFLUENCE ON OUTCOMES: DID IMPLEMENTED ACTIVITIES LEAD TO MEASUREABLE OUTCOMES, SUCH AS CHANGES IN KNOWLEDGE, ATTITUDES, OR BEHAVIOR? The structure of the outcome indicators included in the original contract limited the program’s ability to report high-level results. Most of the planned outcomes were overly ambitious within the short timeframe due to low levels of institutional and human capacity, tensions between target organizations, and the social context. The program was also slow to gain traction, due to both personnel and context issues, and very few activities were successfully completed in the first two years of implementation. It is difficult to detect higher-level outcomes of the program given the fact that evaluation was completed only three months after the end of implementation. SUSTAINABILITY AND LESSONS LEARNED: HAVE KEY IMPLEMENTED ACTIVITIES BEEN SUSTAINED? WHAT IS THE LIKELIHOOD FOR FUTURE SUSTAINABILITY? The program left behind both physical resources, such as documents, and intangible resources in the form of skills. The evaluation team found that three months after FOTI ended, program activities are being sustained to a certain extent. Continuation of activities across most of the components takes three forms: continuation of advisors in the CAC Investigations Unit and the OIG; continuation of xii independent trainings and outreach in CAC; and continued use of written deliverables by CAC, OPG, and Non-State Actors. Sustainability has not yet been achieved with the OIG deliverables, as none of the products have been formally approved. The CAC electronic case management system (SIAK) is not operational. Parliament reports a lack of internal dissemination of key reports. Lessons learned include: 1) CAC will need several more years of support and experience to meet its mandate; 2) External advisors need to have an exit strategy that pro-actively builds sustainability from the beginning; 3) Donors should consider the structural barriers to conducting investigative work, particularly in the media; 4) On-demand or upon-request training often produces better results, counterpart ownership and sustainability than training identified and planned by donors with minimal counterpart input; 5) Donors should invest heavily in establishing an understanding of division of responsibility and interagency procedures before building the capacity of relevant anticorruption agencies. RECOMMENDATIONS Component Specific Recommendations:  CAC will require further program support for the short to medium-term. Investing in the finalization of the SIAK system may facilitate improved case management of corruption investigations. USAID should support CAC’s campaign to educate districts and sucos in preventing corruption. USAID or other donors should encourage the CAC Education, Outreach, and Research Unit to continue planned research such as survey activities.  Donors should consider bolstering the capacity of OPG to investigate and prosecute complex crimes, including possible trainings in financial investigative techniques.  Donors should continue to invest in strengthening internal and external audit capacity within the RDTL. The importance of audit skills will increase as the country moves toward decentralization, especially as movement towards decentralized governance escalates.  Public procurement continues to be a perceived source of corruption, which has yet to be addressed. USAID or other donors should leverage the procurement Training Needs Assessment produced by the program as a starting point for designing a comprehensive approach to this issue.  Donors should consider providing technical assistance in the forms of training or mentoring to Parliament Committee A to fill legislative gaps that facilitate various forms of corruption.  Civic education is important for building a citizenry that holds public officials accountable. Ba Distrito could increase its potential impact by adding funding for a civic education component that used the work of CEPAD and the Global Organization of Parliamentarians Against Corruption to reach out personally to citizens in the target districts. USAID could institutionalize the AC storybook by funding the printing of another set for 2016. USAID should support civil society and journalists to undertake an advocacy program on freedom of information at both the national level and the target Ba Distrito districts. Overarching Recommendations  Future AC threshold programs should carefully consider organizational readiness of beneficiary organizations before committing to support them and/or designing an approach.  Future programs should also consider whether a focus on investigation is appropriate given the social and historical context.  Donors in Timor-Leste should be moving away from the external advisor model and towards a more sustainable model of pro-active capacity-building. 1 EVALUATION PURPOSE & QUESTIONS EVALUATION PURPOSE & QUESTIONS This final performance evaluation provides an objective, rigorous, and evidence-based analysis of the Fostering Transparency Initiative’s (FOTI) overall performance. The primary evaluation questions were identified following requests from key stakeholders at The Millennium Challenge Corporation (MCC) and its partner, the United States Agency for International Development (USAID). 1. Program Logic a.Were the original six program components relevant to the Timorese context? (e.g. were they based on initial analysis of assumptions, risks, and external factors?) b.Were any changes made to the program logic during the implementation? If so, were they appropriate? 2. Activities Designed and Implemented a.Were planned activities actually implemented? Why or why not? b.Were any new activities designed and implemented? If so, why? c.What was the effectiveness of activities? 3. Influence on Changes in Knowledge, Attitudes, and Behaviors a. Did implemented activities lead to measurable outcomes, such as changes in knowledge, attitudes, or behavior? 4. Sustainability & Lessons Learned a. Have key implemented activities been sustained? b.What is the likelihood for future sustainability? c. What are the key aspects of FOTI that should be continued in the near term? d. Which aspects of FOTI would be appropriate to replicate in other contexts? The primary intended purpose of the evaluation is to 1) provide lessons learned for MCC, RDTL, USAID, and other stakeholders in implementing future anti-corruption (AC) activities; 2) uphold MCC’s commitment to maintaining accountability and measuring program results; 3) provide practical lessons for current and future Threshold partners. In September 2010, MCC’s Board of Directors approved the RDTL’s proposal for a $10,496,000 Threshold Program to 1) strengthen AC efforts and 2) improve access to immunization services, demonstrated by improved performance in MCC’s Control of Corruption and Immunization Rates indicators3. The Timor-Leste Threshold Program was funded by MCC with oversight provided by USAID. Implemented by Casals & Associates (Casals), FOTI focused on Component I of the Threshold Program, Strengthening Anti-Corruption Efforts, and accounts for $6,961,340 of the total Threshold Program’s budget. The program, which operated from February 2011 to March 2014, worked to reduce corruption by strengthening a network of functioning, effective AC institutions and actors with the 3 See “Timor-Leste Threshold Program,” MCC Website. http://www.mcc.gov/pages/countries/program/timor-leste-threshold-program. Accessed 4/16/2013. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 2 capacity to deter, detect, and sanction corruption. It emphasized building human and institutional capacity and developing linkages among select institutions and actors to create a coordinated approach to fighting corruption. FOTI contained six components with each of their objectives explained below: Component A: Support to the CAC Establish the CAC as the lead agency for anti-corruption education, prevention, and investigation by ensuring that the CAC is able to successfully complete investigations of cases of alleged corruption and, if warranted, refer its findings to the Office of the Prosecutor General. Component B: Support to the OPG Build OPG’s investigative and prosecutorial capacity, particularly for the investigation and prosecution of corruption cases and economic crimes. Component C: Improve Internal and External Audit Capacity Strengthen the auditing capabilities of internal government audit entities and establish a functioning external audit authority to improve the Government’s ability to prevent and detect instances of corruption and, where necessary, assist OPG with criminal indictments. Component D: Improve Procurement Policies, Regulations, and Practices Increase the knowledge and use of internationally-accepted procurement standards in relevant line ministries by having Government contracting authorities train procurement officers and establish an in-service training system for contract officers on international and Government legal procurement standards. Support the establishment and operations of an independent procurement review commission empowered to adjudicate procurement complaints in accordance with its legal mandate. Component E: Strengthen Fiscal Oversight Skills of the NP Strengthen NP’s role as a check on corruption and collusion through increased oversight of the government’s fiscal management practices, including using its legislative oversight authority to probe alleged corruption or misuse of funds, or follow up on recommendations of independent agencies. Component F: Enhance External Watchdog and Public Advocacy Efforts Increase the capacity and public advocacy skills of civil society organizations, media, academia, and private sector associations in order to improve their effectiveness in monitoring, advocating, and disseminating information on corruption-related issues, including through AC themed public advocacy campaigns and watchdog activities. COUNTRY CONTEXT Timor-Leste has a unique set of historical, cultural, social, political, and economic features that shape its potential for experiencing and fighting corruption. As a Portuguese colony from the 17th century until 1975, its infrastructure was devoted to extraction of resources, while its population was left uneducated, in poor health, with limited political and social rights, and hardly any experience or capacity in self-governance. When the Portuguese left abruptly in 1975, their departure gave the opportunity for indigenous movements to fight for independence, only to be crushed by the invasion of the Indonesian army. Timor-Leste was formally incorporated as Indonesia’s 27th province in 1976 and was administered by the Indonesian Government. This period was characterized by brutality, neglect, and a widely documented pattern of petty and large-scale corruption instigated by the Indonesian military. Resistance movements were active in rural areas, occasionally engaging in urban skirmishes. Some Timorese went into exile, where they attended universities and organized support for the resistance. After the fall of Indonesian President Soeharto in 1998, the United Nations (UN) organized a Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 3 referendum in which 78% of the Timorese electorate voted for independence. Pro-Indonesia militias then launched a planned, systematic scorched-earth campaign that killed civilians, displaced up to 30,000 persons4 , coerced more than 200,000 refugees into Indonesia, and destroyed approximately 80% of Timorese infrastructure5 . The UN administered the country until its independence in 2002, and its peacekeepers remained until December 2012. Donors provided thousands of advisors to help the RDTL keep the peace, promote transitional justice, reconstruct infrastructure and establish new institutions. Capacity building was slow, as human resources were a significant constraint, given that very few Timorese held senior positions during the Indonesian period or were educated abroad. Lack of a common language and a poor understanding of corruption also slowed progress. Traditional hierarchies and the history of both Portuguese and Indonesian top-down governance have exacerbated the post-conflict tendency for former combatants to be seen as “entitled” to economic benefits from the government. Geographic and social divisions plagued the country, leading to periodic fighting that resulted in dozens of deaths and tens of thousands of displaced persons. Presidential and parliamentary elections in 2007 and 2012 allowed a transfer of power, building the credibility of these new institutions. The government relies on income from the Timor-Leste Petroleum Fund, supplemented by donor funding, for its revenue. While the Fund is believed to have been well-managed, the government finds it increasingly difficult to maintain the discipline needed to withdraw an annual sum that will conserve capital after petroleum reserves are depleted. The virtual absence of tax revenue creates problems of government accountability to Timorese citizens and creates the potential for poor budget decisions. A small private sector, weak civil society, and the absence of a culture of investigative journalism limit the ability of the population to check government abuses. The RDTL aims to decentralize governance in the coming years. The Program for Accelerating Suco Development (PNDS) will grant local councils (sucos) considerable sums of money to manage community development programs. EVALUATION METHODS & LIMITATIONS METHODOLOGY The mixed methods evaluation was carried out by a four-member team from Social Impact, Inc. (SI) during field visits in early 2014. In February 2014, while the FOTI project was still active, the Team Leader completed preliminary data collection during an evaluation design trip. In July 2014, the other three team members completed data collection during a two week field visit to Timor-Leste. The evaluation utilized ongoing data analysis, an iterative methodology, and careful triangulation of data sources throughout the research process. The evaluation team utilized the following methodologies: document review; technical evaluation of “deliverables”; pre- and post-test data; key informant interviews; and a field visit to Baucau. Please see Annex II for a detailed description of each method. 4 Silva and Ball, “The Profile of Human Rights Violations in Timor-Leste, 1974-1999: A Report by the Benetech Human Rights Data Analysis Group to the Commission on Reception, Truth and Reconciliation of Timor-Leste,” February 2006. 5 "Indonesia/East Timor: Forced Expulsions to West Timor and the Refugee Crisis". Human Rights Watch. December 1999. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 4 LIMITATIONS Although the evaluation team attempted to complete a rigorous examination of the approved research questions, it proved impossible to utilize all of the proposed research methodologies due to logistical and bureaucratic realities in Timor-Leste. For example, although the team initially planned to conduct mini-surveys and focus group discussions, the team did not gain access to enough program beneficiaries to make data obtained from such informants meaningful. Other limitations include: challenge in measuring qualitative outcomes; logistical and timing challenges; recall and attribution bias; selection bias; and use of interpreter. For further details, please see Annex II. While these factors limit the quantity and quality of the data collected, the team made every effort to fill gaps in the data, particularly by examining all available program documentation. Through this triangulation of sources, the team feels that the information presented in this report is an accurate depiction of the FOTI program. FINDINGS AND CONCLUSIONS 1. PROGRAM LOGIC A. WERE THE ORIGINAL SIX PROGRAM COMPONENTS RELEVANT TO THE TIMORESE CONTEXT? (E.G. WERE THEY BASED ON INITIAL ANALYSIS OF ASSUMPTIONS, RISKS, AND EXTERNAL FACTORS?) FINDINGS The Threshold Program for Anti-Corruption (TPA) program objective mirrored the corruption-related “result” of the Assistance Agreement (AA) between MCC and the RDTL. This objective sought to reduce corruption through a two-fold approach of building the capacity and increasing coordination of “key actors” within the RDTL to “prevent, detect, and punish the misuse of government funds6.” In order to achieve this objective, the program selected six intermediate results that looked to develop the capacity of the six institutions (“components”) identified in the program solicitation. The TPA design was informed by a comprehensive Corruption Assessment of the context in Timor￾Leste, conducted by Management Systems International in 2009. The Assessment identified low capacity and weak oversight as two of four core issues influencing corruption in the country; specifically, the study found that the CAC, the external audit body, the OIG, the justice system, the NP, and the media were all “nascent institutions [which] need further support.” The designation of the six components reflects an adoption of the Assessment’s recommendations to support these institutions. The original solicitation for the program touted Timor-Leste’s political will to combat corruption, citing the establishment of the CAC and a suite of other signals of the RDTL’s commitment to fight corruption. However, the TPA design contained flaws in its determination of the institutional readiness to receive FOTI assistance in the forms laid out in the TPA. Interviewees at the CAC, OPG, the Office of the Inspector General (OIG), and the NP cited two factors that limited uptake of advisors’ services: competing organizational priorities and lack of trust of the program’s advisors and/or leadership. In the cases of the external audit authority and National Procurement Commission, the organizations themselves were not operational for the majority of the program period. 6 Program Results Framework (see Annex VI) Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 5 TPA’s emphasis on investigation of corruption cases, while compelling for donors, did not seem to carefully consider some realities of the social context in the country. Timor-Leste is a small island nation with a close, hierarchical society in which many residents are connected through family, clan, ethnic, or political ties. Furthermore, it is a post-conflict society, in which a great deal of violence was perpetrated including as recently as 2006. Much of the violence has subsided; however, political party affiliation or other personal affiliation governs many social interactions. Interviewees at the OPG, the OIG, and the media conducting work of an investigative nature cited personal barriers to retrieving evidence as needed and worse, a risk of retaliation from the accused or members of their social networks. In the case of investigative newspaper article written under the mentorship of the investigative journalism advisor, the editors ultimately declined to publish it, for fear of backlash or retaliation. Although the program was not informed by a specific gender assessment, the solicitation required that implementers “ensure the integration of gender concerns” and address how the program would incorporate men and women into design and implementation; how program activities would differently affect men and women; and how the program would mitigate potential tensions created by program activities. Casals endeavored to address gender issues, primarily by working with women partners, particularly, parliamentarians and civil society groups, but also by integrating gender into CAC surveys and advancing gender-responsive budgeting and gender-based violence policy agendas7. The AA included a condition precedent (CP) that required a “binding written instrument specifying and allocating the respective levels of authority and responsibility for the investigation and prosecution of economic crimes among the Office of Prosecutor General, the Anti-Corruption Commission, the National Police, the Banking Payment Authority, the Office of the Inspector General, and any other relevant institutions or entities within the Government” to be in place before programming began. This was designed to minimize the risk that training provided by FOTI would aggravate the challenge of ambiguous jurisdiction in the investigation of corruption cases. Although a formal MOU was not signed, both the CAC and the OPG agreed that the actual working arrangements and agreements between the CAC and the OPG, along with the plain language of laws and constitution of Timor-Leste, demonstrated that formal roles were identified. According to the agreements approved by MCC and USAID, the Constitution clearly articulated the OPG's sole authority to direct criminal investigations (as well as articulating the responsibility of everyone, presumably including the CAC, to notify the OPG when they suspected a crime had been committed. Despite these formal agreements, interviews with staff of the CAC and OPG revealed some continuing misunderstandings between the two organizations on how cases are handled, particularly once the CAC passes cases onto the OPG. CONCLUSIONS The TPA, informed by an assessment of the corruption context in Timor-Leste, selected six relevant components each focusing on important AC actors to target within the RDTL; however, the design was predicated on several assumptions that did not hold true in the life of the program. The TPA sought to leverage the momentum seemingly created by the series of AC legislation passed between 2009 and 2011. While not insignificant, these signals existed in the legal and policy realm and did not translate to readiness of the TPA-targeted institutions to receive assistance as designed by the program. Generally speaking, low capacity, competing organizational priorities, and lack of leadership buy-in negatively affected program activities. The TPA design was ambitious in its assumptions about how receptive these 7 Year 1 Work Plan, 2011 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 6 institutions would be to receiving TPA assistance. Likewise, the design overestimated the program’s political capital, particularly regarding its objective of increasing cooperation among organizations. The TPA’s emphasis on investigation among several institutions, while justified by global best practices, was especially challenging in the Timorese context. This emphasis on building investigative capacity in OPG and CAC also exacerbated existing tensions between them. The complicated social and political dynamics in post-conflict Timor-Leste, in which hierarchy dictates that lower ranking people keep silent, also call into question the apparent over-emphasis on the TPA on the act of investigation. B. WERE ANY CHANGES MADE TO THE PROGRAM LOGIC DURING THE IMPLEMENTATION? IF SO, WERE THEY APPROPRIATE? FINDINGS Slight modifications to the program logic were permitted through a contract modification of the scope of work (SOW) shortly before the end of the program in December 20138. These were primarily to match the required activities to the altered contextual realities. For example, neither the external audit authority nor the Procurement Review Commission were operational during the majority of the program. Therefore, requirements to work with them were removed in the SOW modification. One of the most significant departures from planned activities was the original requirement that the program assist non-state actors in conducting three iterations of a corruption survey, which was likely in response to one of the recommendations made by the 2009 Assessment9. Once implementation commenced, the CAC requested that FOTI provide technical assistance on corruption surveys to its Education, Outreach, and Research Unit. Consequently, the CAC participated in the first public perception survey, which was conducted by a private firm. With guidance from FOTI advisors, the CAC implemented the second and third surveys without the involvement of non-state actors. FOTI’s assistance to the CAC, rather than to non-state actors, on surveys was documented in a contract modification. CONCLUSIONS In some cases, the design relied too heavily on external dependencies (e.g. establishment of institutions) that were difficult for Casals to manage or predict. In retrospect, the lack of existence of the external audit authority and Procurement Review Commission suggests that targeting these nascent organizations was premature. Similarly, the program design made some assumptions about the political will among target institutions that did not hold. In hindsight, the emphasis on investigation also seems premature at best. All three of these factors necessitated slight modifications to the program logic. However, the program was—at least in the case of corruption survey—able to take corrective actions that made it more responsive to the target institution’s priorities. If the program had a better sense of the needs and desires of the target organizations, it seems likely that the activities would have launched more quickly and decreased the likelihood of needing to substantially modify the SOW. 8 Task Order AID-486-TO-11-00001, Mod 4, 2013 9 Task Order AID-486-TO-11-00001,2011 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 7 2. ACTIVITIES DESIGNED AND IMPLEMENTED & NEW ACTIVITIES A. WERE PLANNED ACTIVITIES ACTUALLY IMPLEMENTED? WHY OR WHY NOT? B. WERE ANY NEW ACTIVITIES DESIGNED AND IMPLEMENTED? IF SO, WHY? For the purposes of this evaluation, “planned activities” refer to activities encompassed by the original SOW included in the contract between USAID and Casals (see Annex VI). These activities were deliberately broad, noting that activities are “preliminary, and not exhaustive.” “New activities” refers to activities not explicitly captured by the original SOW. The contract between Casals and USAID was modified in December 2013 and updated the SOW to permit significant departures from the originally planned activities, based on the realities faced by the program during the course of implementation. FINDINGS—COMPONENT A: CAC CAC Investigations Unit FOTI’s support to the Investigations Unit at the CAC occurred on both a formal and informal basis including the following planned activities:  A FOTI advisor provided an “induction training program,” which provided a basic overview of subjects such as criminal investigations, the Criminal Procedural Code, the Penal Code and the United Nations Convention against Corruption (UNCAC). Two FOTI advisors provided training on AC￾related money laundering and financial crimes. In addition, two of CAC’s investigators participated in a four-month training course on corruption investigations organized by Singapore’s Corrupt Practices Investigations Bureau.  A FOTI advisor assisted with the design of the Sistema Informasaun Anti-Korrupsaun (SIAK), an information management system to host CAC’s caseload, which was initially designed and developed by an information technology specialist from USAID’s former Justice Institutions Strengthening Program (JISP10); however, the system is currently not operational.  Additional trainings provided to group on an ad hoc basis in response to investigator’s needs. All other training provided by the FOTI advisor was provided to groups on an ad hoc basis in response to the investigators’ needs. CAC investigators also reported that FOTI advisor provided training and mentored mainly on basic police work, such as general crime investigation. No new activities were added to the SOW in the area of investigations. With the fulltime advisor in place, the Investigations Unit received a steady flow of technical assistance along the lines prescribed by the SOW. Prevention Unit FOTI provided various forms of support to the Prevention Unit, including the following activities:  In Year 1, FOTI advised and mentored the Unit in discussions with the Customs Service, development of a Customs Integrity Plan, and two Custom corruption prevention workshops11. No additional Customs activities took place in Years 2 and 312. 10 USAID’s Justice Institutions Strengthening Program (JISP) was implemented by Management Sciences for Development, Inc. from May 2005 to March 2012. 11 October-December 2011 Quarterly Report Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 8  In late Year 2, FOTI prepared the training of trainers (TOT) manual on prevention. It led a training of Prevention staff through 11 workshops, using the two manuals and other curricula. Of 119 persons attending all trainings, 14 or 12% were female13. FOTI trained Unit personnel to train on specific subjects, so that they could collaborate in training other agencies.  With FOTI’s mentoring, trained Unit staff undertook a campaign to build the corruption prevention capacity of other public institutions, including SERVE, the one-stop shop for business registration and the Ministry of State Administration, in Year 3. They engaged International Airport personnel in a “participant observation” exercise in the arrivals terminal. The majority of the gains in prevention were accomplished in Year 3, after the arrival of the second FOTI advisor. Multiple stakeholders reported that this advisor was much more successful than his predecessor due to his relevant previous experience and background as well as his collaborative working style and engaging pedagogy. Two new prevention-related activities were added to the CAC component of the SOW:  “…provide technical assistance and guide CAC in designing and conducting corruption surveys and disseminate the survey’s findings to the public;” and  “…develop and deliver a competency-based certification course on procurement for CAC staff and government procurement officers.” The latter activity initially appeared in FOTI’s procurement component and was targeted at the Ministry of Finance (MOF) and local training institutions; however, due to the lack of traction with the MOF, FOTI ultimately shifted this activity to the CAC, where more progress could be made. The program administered a workshop for CAC staff on preventing corruption in public sector procurement. This workshop served as the basis for a new training manual to help CAC staff train other public officials on mechanisms to reduce and manage procurement corruption risks14. In Year 3, FOTI prepared the TOT based on the procurement manual, which took into account survey findings on the different perceptions and experience of women and men15. It was used to train CAC staff in a day-long workshop in December 2013. Of 22 persons attending the training, 14% were female, compared to 30% female participation in the Civil Service16. Concurrently, FOTI helped the Unit develop a Corruption Prevention Plan (CPP) for 2014-2015, which identified procurement as one of its four priorities17. This took place through facilitating the identification of lessons learned from the Investigations Directorate, consultations with other agencies and stakeholders, and internal CAC workshops to gain a consensus on priorities, which include new support to the Customs Service and expanding support for decentralization18. The CPP was developed as a measure to build the CAC’s strategic planning capacity, which was an initial goal of the component. 12 FOTI Final Report, 2014 13 CAC Prevention Training attendance sheets 14 FOTI Final Report, 2014 15 Training Manual on Preventing Corruption in Public Sector Procurement, 2014 16 CAC Prevention Training attendance sheets 17 CAC Prevention Plan, 2014 18 Ibid Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 9 CAC Education, Outreach, and Research (EOR) Education and research were not a major focus of the original activities, as the SOW only made reference to “developing education campaigns.” Initial work with the EOR was also limited due to challenges with the first advisor assigned to work with EOR; CAC staff reported that the original advisor did not have appropriate background or experience to work effectively with the CAC. In addition, CAC staff had a challenging relationship with the first Chief of Party (COP) of FOTI, which had “adversely affected the CAC’s appetite for FOTI’s assistance” at the time of the Midterm Assessment (MAT)19. Despite these setbacks, the following activities took place in the first half of the program:  The FOTI advisor supported a public information campaign by the EOR Unit in Year 1, which allowed it to make presentations and hold meetings in all of Timor-Leste’s 13 districts, reaching 120 stakeholders20. One of the EOR unit staff told the team, “When CAC was established, it was not known to the public. We did a lot of outreach activities to increase public awareness.”  FOTI also assisted CAC in the development of their public website, which has received over 40,000 visits to date21. The site regularly posts updates about the CAC’s work in both English and Tetun, provides contact information for denouncing cases of corruption, and includes links to the websites of other relevant organizations22. FOTI conducted a number of “new activities” in response to the EOR’s needs. Following the recommendations made in the 2009 Corruption Assessment, the original FOTI program design called for three iterations of a survey to be conducted by non-state actors23. However, this was renegotiated at the program’s inception in response to CAC’s desire to undertake this work themselves. In support of the request, FOTI mobilized advisors to support the CAC in developing research plans and survey methodologies, creating data collection tools, and conducting the data analyses. FOTI supported the CAC to design a public perception survey in 2011, one of the first of its kind. For this survey, the CAC engaged a local consulting firm to carry out the data collection24. The survey had two purposes as an interviewee in the CAC unit noted, “We wanted to measure whether [our outreach] activities [in FOTI’s Year 1] were sufficient to make citizens aware of the CAC’s existence.” The first purpose of the household survey was to measure public awareness of the CAC, while the second purpose was to measure awareness of corruption. The original SOW called for three iterations of the same survey but, the CAC elected to change the focus of the second and third surveys. A second survey, the 2012 Integrity Survey served two basic purposes: first, “to better understand what parts of public service are already working well and which improvements are needed to better serve the citizens of the country25.” Second, the CAC wanted to use this opportunity to train a core group of staff to specialize in survey methodologies. With additional support from FOTI, the trained staff applied this knowledge in the development and execution of the third survey, a “snap survey” that gauged public perceptions of corruption in the civil service. 19 FOTI Midterm Assessment Report, 2012 20 Ibid. 21 CAC website, accessed August 11, 2014 22 Ibid. 23 MSI Corruption Assessment Report, 2009 24 CAC Public Perception Survey Report, 2011 25 Integrity Survey of Public Servants in Timor-Leste Report, 2012 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 10 FINDINGS—COMPONENT B: OPG The program conducted trainings on various topics pertaining to investigation and prosecution of corruption cases, including, but not limited to money laundering and asset tracing; financial investigation; and on-the-job training, particularly from the second advisor. Notably, the program conducted joint trainings with the CAC, OIG, the national police, and other relevant stakeholders in topics such as corruption prevention, money laundering, and other financial crime investigations in order to develop relationships and increase collaboration among AC-fighting institutions. The program also produced manuals including, the “Financial and Corruption Investigations Manual,” the “Prevention and Combating Money Laundering in Timor-Leste” manual, and an annotated version Law No. 17/201126. FOTI had a slow start, but the OPG acknowledged that partly it was because the OPG was not ready for the program. At the time, the OPG was preoccupied dealing with its priority goal of reducing the backlog of cases, a problem which fell outside the scope of FOTI27. Various FOTI program documents cite difficulty making inroads with the OPG, particularly in getting them to accept mentoring support due to the confidential nature of their work. The OPG advisor was replaced in 2012 and, whereas the first advisor primarily had an academic background, the new advisor had many years of practical experience as a prosecutor in Portugal. The pace of activities in this component increased significantly and this component was considered to be “performing” at the time of the midterm assessment, even when other components were deemed to be behind28. The district prosecutors interviewed indicated that confidentiality issues were not problematic when working with the second advisor. No “new” activities were provided for this component. There was however an amendment to the job description for advisor as follows: “The long term OPG Corruption Advisor shall provide daily technical assistance and skills transfer to the Prosecutor General, Deputy Prosecutor General and line staff on developing corruption-specific cases for prosecution. Areas of focus include: investigations analysis; evidence gathering; and prosecutorial techniques. The Advisor shall also be responsible for assisting the National Parliament and others (e.g. CAC, Council of Ministers) in drafting outstanding pieces of anti-corruption legislation” 29. Although no new activities were formally added to the SOW in support of the OPG, the following unplanned activities took place:  OPG held a national seminar on 2014 on corruption prevention and involved many institutions such as MOF, FOTI, CEPAD, and journalists.  A training was delivered on Asset Recovery Regulations and Mechanisms. These activities both occurred at the suggestion of the OPG based on the organization’s new initiatives and priorities. FINDINGS—COMPONENT C: AUDIT The FOTI program was unable to undertake work in support of the external audit agency as it did not exist in practice during the program’s period of performance. Rather than facilitate the establishment of the High Administrative, Tax, and Audit Court (HATAC), the program worked with the fully operational OIG, the body charged with conducting internal government audits. 26 OPG Indicator #3 27 Corruption Assessment Report, 2009 28 FOTI Midterm Assessment Report, 2009 29 Task Order AID-486-TO-11-00001, Mod. 4, 2013 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 11 Initially, work was delayed on this component because the RDTL had not fulfilled one of the relevant CPs for this component, i.e. the establishment of the HATAC. USAID deemed the audit-related CPs to be met in the second half of 2011, at which point the program carried out a Training Needs Assessment (TNA) per the terms of the SOW. The TNA found that state auditors understood the need for comprehensive, quality audit capacity building, and that auditors “could benefit from additional numeracy and literacy courses that develop critical thinking skills30.” FOTI initially proposed an advisor who was rejected by the OIG due to lack of relevant experience. In early 2012, FOTI engaged a former Casals advisor who had previously resigned after three months, citing difficulties in working with the Casals COP. The advisor agreed to return in May 2013 once the program was under the leadership of a new COP. The advisor provided classroom-based courses and training manuals as well as coaching and mentoring on subjects pertinent to the auditors’ work. Training sessions covered a wide variety of subjects including, but not limited to: auditing, public administration, financial crimes, and legal procedures. With regard to the third set of activities, the advisor worked on increasing coordination with the inspector generals in 14 other ministries by holding joint meetings and standardizing audit definitions while hired by FOTI. This was an important need identified by the TNA. In December 2013, USAID modified Casals’ contract revising Component C’s title to “Audit Authority” and removing the first activity and all other references to the external audit authority. Aside from the change in the targeted organization, no substantially “new” activities were conducted. As referenced in the previous section, FOTI helped to create a number of written resources during his latter tenure with the FOTI program. These include, but are not limited to: Decree Law for the Special Regime for the Career of Public Auditors in the Government; Internal Audit Chapter and Handbook of the OIG; and a guide on dealing with accusations against public officers. These products were all created in collaboration with the OIG based on their self-determined organizational needs. FINDINGS—COMPONENT D: PROCUREMENT At the time of the FOTI program design, public sector procurement was concentrated in the MOF. Work in this component did not launch for the first few months of the project due to delays in meeting the procurement CPs, one of which was related to the procurement law, which underwent several drafts and was not passed during the life of the program. Once the work was authorized to commence, the procurement advisor conducted a TNA of the MOF staff. The advisor observed that the MOF was uninterested in taking advantage of FOTI support, reporting that RDTL counterparts did not signal any desire to leverage the skills or resources offered by the program. FOTI discontinued support to the MOF after the approval of the TNA report, as the training directorate at the MOF—the supposed FOTI counterpart organization—folded. Although a National Procurement Commission had been established by law, it was not operational at the time, so the program established a relationship with the newly￾created Secretariat of State for Institutional Strengthening (SEFI), a seemingly natural candidate for building government capacity in preventing corruption in public procurement31. FOTI stationed the procurement advisor at SEFI on a part-time basis; however, given that it was a new institution, SEFI was not prepared to begin working on the procurement activities as laid out in the TPA32. The FOTI advisor assisted SEFI to draft a document codifying the ten principles of public sector procurement in Timor-Leste. This document was socialized among all the government ministries; in 30 Audit Training Needs Assessment Report, 2011 31 FOTI Midterm Assessment Report, 2012 32 Ibid Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 12 addition, the advisor began working with SEFI to draft SOPs guiding public sector procurement. After the departure of the advisor, FOTI decided to focus on other programmatic priorities and consequently, activities under this component ceased altogether. Substantial modifications in the program’s activities in the procurement component occurred due to the changing circumstances within the RDTL about the procurement mandate. In the absence of a procurement law designating a specific organization for the training mandate, USAID modified Casals’ contract to specify that the competency-based certification course on procurement for government be delivered to CAC staff in addition to government procurement officers33. As such, all of the work related to procurement was done under the auspices of the CAC component. As described in the CAC prevention section, a FOTI advisor assisted the CAC to hold workshops, develop a manual, and design and roll out a training on preventing corruption in public procurement. Interviewees at SEFI were pleased with the progress of this training, citing the growing frequency and value of public procurement, particularly with foreign firms. FINDINGS—COMPONENT E: PARLIAMENT FOTI engagement with Parliament in the first half of the program focused on developing Parliament’s oversight skills. To this end, the FOTI program assisted with the following activities:  Parliament’s AC sub-committee held town hall meetings on the design and contents of anticorruption legislation. The first town hall meeting was held in Year 1 attended by 116 local opinion leaders representing stakeholders from across Timor-Leste34;  A joint Global Organization of Parliamentarians against Corruption (GOPAC)/UNDP event held in Year 2 on the UNCAC self-assessment35;  Development of a Desk Reference (second Volume) for Committee C of Parliament36; and  Delivery of 368 person hours of training for members of the National Parliament. FOTI provided assistance in the planning and drafting of three pieces of AC legislation37. Parliamentary elections took place in 2012, which resulted in a change the ruling party. In addition, responsibilities for AC legislation shifted from Committee C to Committee A, which effectively required the program to start over. At the time of the MAT, this component was deemed to be “non performing…due to the departure of the technical expert most recently assigned to this section who was well respected and appreciated for all her contributions in the past38.” However, during the course of the final evaluation field work, interviewees from the Parliament reported difficulties working with FOTI initially, namely a poor relationship with the program’s first COP, who instead of initiating contact coordination with the President of the Parliament, approached the Chair of Parliamentary Committee C’s Sub-Committee responsible for AC. In addition, there were issues related to the limited practical experience of the first advisor, whose background was primarily academic. The Contracting Officer’s Representative from USAID later met with the President of the Parliament and re-established the relationship, which heralded a more productive second half of the program. 33 Task Order AID-486-TO-11-00001, Mod 4, 2013 34 Monthly Progress Report No.4, p.1 (June 2011) 35 Monthly Progress Report No. 15, p.3 (April 2012) 36 Monthly Progress Report No. 15, p.3 (April 2012) 37 Midterm Assessment Report, 2012 38 Ibid, 17 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 13 Although the MAT displayed skepticism about the suitability of the second advisor’s credentials, she was able to re-build the relationship with the Parliament and undertook a large volume of work. During her tenure, the primary focus of activities was provision of technical expertise to drafting AC legislation and to working with the Parliamentary Committee A, which included the following activities:  Development of a “Compilation of Legal Standards on the Oversight Function of Parliament in Timor-Leste” in coordination with the National Parliament. This publication was distributed among all Members of Parliament;  Development of “Report on Legislative Priorities to Strengthen the Legal Framework Against Corruption39;”  Seminar on “Using Anti-Money Laundering Investigation to Combat Corruption40;”  Provision of expert technical analysis on various pieces of legislation pertaining to AC, including Act 17/2011, the Timorese Penal Code and Penal Procedural Code, the draft FIU decree-law, and an amendment of the jurisdiction of the Chamber of Accounts. Training activities conducted by the second FOTI advisor to the National Parliament including the number of activities and the number of reported person hours of capacity building represent achievement the component’s activities. No new activities were formally added to the SOW41. All of the activities conducted under this component were in line with the broadly defined activities outlined in the original scope. FINDINGS—COMPONENT F: NON-STATE ACTORS While the Casals contract was signed in February 2011, the first activities supporting the original two objectives didn’t take place until June 2012. The program then undertook the following activities:  Six workshops for young journalists on watchdog reporting, which involved “journalism skills contests, a brief lecture from one of the local media leaders and a TPA Advisor42.”  Assistance to other non-state actors, e.g. private lawyer members of the Advocates Association of Timor-Leste were trained in financial investigations through twice weekly two-hour lunch seminars over eight weeks43;” and  Delivery of an intensive theoretical and practical course on investigative journalism for 15 radio, television and print journalists. Under its first COP, FOTI made several unsuccessful efforts to organize AC training for civil society organizations (CSOs) and think tanks. The first and only FOTI-supported watchdog and advocacy activities for civil society took place in November 2013 and February 2014, both trainings on monitoring and analyzing the state budget. The CSO/think tank Lao Hamutuk provided basic and advanced training for 20-22 persons, all representing advocacy CSOs. The classes addressed issues such as receipts and withdrawals from the Petroleum Fund; donor funding; the development, review and execution of the budget; and procurement. They also showed the differential impact of the budget on men and women. Although not included in the modification to the SOW, the Year 3 work plan included a major “new” activity, to “Conduct civic education campaigns to raise the understanding of corruption and its pervasive effects44.” The evaluation team found that the program devoted more resources to achieving 39 FOTI Final Report, 2014 40 FOTI Raw Data & Data Sheets 41 Task Order AID-486-TO-11-00001, Mod 4, 2013 42 Quarterly Report for July-September 2011 43 Quarterly Report for October-December 2012 44 Work Plan Year 3, 2013 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 14 this objective than to building watchdog and advocacy capacity, but most of these efforts began 15 months after its initiation. FOTI teamed with the Center of Studies for Peace and Development (CEPAD) and the Global Organization of Parliamentarians against Corruption (GOPAC) to support media and citizen outreach programs that raise awareness of corruption. Events included international and national fora that aimed to raise awareness on corruption and efforts to fight it. Beginning in 2014, FOTI partnered with CEPAD on a civic education campaign on corruption, collusion, and nepotism. The objective was to: 1) raise awareness and understanding on the causes and consequences of corruption and 2) engage different stakeholders in the discussion and promotion of AC measures45. FOTI helped CEPAD prepare civic education material, including an AC action manual, a storybook, a music CD, three radio interviews and two TV debates. The program also supported the distribution and broadcast of these products. CEPAD organized community events to introduce these materials to civil society leaders at its three regional Peace Houses. The radio programs and music CD were broadcast by 23 community radio stations multiple times. The Ministry of Education has agreed to include the storybook in its curriculum. FOTI funded the publication of 64,000 copies for the first year to be distributed in January 2015 in order to coincide with the beginning of the first academic trimester. The Ministry will seek funding for the book to remain in the curriculum for subsequent years. The evaluation team visited the Baucau Peace House to assess its CEPAD-supported AC meeting. The two-day event was attended by 38 local leaders from Baucau and its six sub-districts. Peace House staff focused on logistics and publicity, but relied on CEPAD to present and discuss its civic education products. Local human rights organizations also made presentations. Beginning in May 2012, FOTI co-produced 18 monthly radio interviews with GOPAC focusing on different aspects of corruption and the roles of those fighting against it. According to GOPAC/Timor￾Leste’s Director, FOTI staff identified the themes, selected the participants, arranged production, and assured broadcast by 23 community and national radio stations, while GOPAC’s role was to provide feedback on the themes and participate in the interviews. According to FOTI’s second COP, the program chose the themes not only to educate citizens, but also to focus on specific policy issues. CONCLUSIONS: PLANNED ACTIVITIES FOTI delivered a broad suite of activities to build corruption fighting capacity among various relevant actors within the RDTL as required by the original SOW. Although the program experienced challenges in program start-up, most of the activities laid out in the program’s original SOW were ultimately completed in some fashion. Generally speaking, the activities laid out in the SOW were broad enough to accommodate various trainings, mentoring, and other forms of technical assistance, primarily by embedding long and short term advisors in the beneficiary institutions. Much of the assistance was in the form of routine coaching and mentoring delivered on an ad hoc basis in response to the institutions’ evolving needs. There were also several formal trainings, though these were typically conducted as one￾off activities. Factors that influenced the extent to which planned activities occurred vary by institution:  CAC- The emphasis of support among the Prevention and EOR Units shifted considerably in response to mixed results from the original advisors, changing circumstances and a better understanding of the needs of CAC counterparts. The original program design was almost totally focused on building investigation and prevention capacity at the CAC, containing few activities in the EOR sub-component which focused on primarily on outreach. While outreach was important 45 FOTI Final Report, 2014 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 15 to help socialize the CAC in its early years of existence, this ultimately comprised a small part of the overall assistance to the CAC, as the CAC requested support for its research agenda.  OPG- Despite some initial setbacks in launching the activities and contrary to prior relative lack of results in FOTI program’s interventions with the OPG, the final evaluation team found that FOTI conducted a significant amount of work in line with what was originally envisioned for the component. An analysis of the training activities conducted by FOTI for the OPG including number of activities and reported person hours of capacity building, the quality of the training material, and fact that joint activities and effective non-planned activities were delivered are indicative of the OPG’s satisfaction with the assistance delivered by the close of the program.  Audit- Most of the anticipated work was unable to occur due to an overly ambitious CP regarding the establishment of the external audit authority.  Procurement- The original design was ambitious and optimistic, as the procurement component was predicated on assumptions about the organizational mandate that did not align with the realities during the course of the program. For this reason, the component was unable to operate as envisioned, even once the CP was fulfilled. Despite the setbacks, the program managed to conduct a few of the originally planned activities, though not to the extent planned.  Parliament- Work with the NP suffered during the first half of the program due to non￾adherence to the proper protocols in engaging the national counterparts. However, once the relationship was repaired, the shift from focusing on developing Parliament oversight skills during the first half to supporting the development and enactment of AC legislation on the second half of the program was partially in response to the change in the political environment that took place after the 2012 elections.  Non-state actors- FOTI supported few activities that contributed to Component F’s two original objectives of building the capacity of non-state actors to undertake fact-based, targeted, and sustained advocacy campaigns and watchdog activities and building coalitions of non-state actors devoted to public outreach and education on corruption. CONCLUSIONS: NEW ACTIVITIES Where more specific activities were prescribed by the SOW, the program sometimes had to introduce “new” activities, which were often just modifications of the original activities redirecting the intended beneficiaries or focus of the content. The explanations for why some of the planned activities needed to be substantially changed had largely to do with the operational context of the organizations that FOTI worked with. Generally, changes were needed to match the organizational priorities of target organizations and/or to compensate for underperformance in other components:  CAC- At the insistence of the CAC, FOTI significantly changed its intended set of activities targeting the EOR Unit by mobilizing technical support for the Unit’s survey work. FOTI implementation in this sub-component adapted early on to better serve the CAC counterparts. In addition, through the process of developing the CPP, FOTI helped the Prevention Unit to understand the importance of addressing corruption in public procurement. Despite an unclear division of responsibility within the RDTL for procurement policy and training, FOTI developed and delivered a competency-based certification course on procurement for CAC staff to fulfill the requirements of the procurement component that was unable to reach its objectives.  OPG- The relative dearth of new activities suggests that the originally planned activities were relevant to the OPG’s capacity building needs once an appropriately qualified advisor was appointed. The addition of these two activities is indicative of both OPG’s satisfaction with the technical assistance provided by the FOTI advisor as well as FOTI’s ability to adapt and respond to the institution’s evolving needs. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 16  Audit- The decision to create written resources for OIG staff was an appropriate activity in light of the component’s updated purpose of “upgrad[ing] basic audit skills of selected government auditors, mainly within the OIG.” The written resources—if approved—may end up to be an important contribution to the OIG’s organizational development. The OIG did not provide the evaluation team with a copy of key documents developed by the FOTI Advisor, such as the Organic Law for the “National System of Control for the Government” and the Decree Law for the “Special Regime for the Career of Public Auditors in the Government.” Several of these documents were developed by the advisor during previous assignments with the OIG and were revised during his tenure with the FOTI program.  Procurement- Some new activities were undertaken due to the fact that changes in the procurement mandate among various government agencies limited the possibility of realizing the activities and results associated with this component as laid out in the SOW. Instead, the program adapted to the circumstances surrounding the changing landscape by making slight modifications to the required activities which replicated the originally envisioned activities to the extent possible. These activities were strategically appropriate in that they aligned somewhat with the feasible recommendations made by the TNA and leveraged existing relationships with the SEFI and CAC.  Parliament- The fact that activities did not need to be substantially modified suggests that planned activities were appropriate for the Parliament’s capacity building needs.  Non-state actors- FOTI devoted more resources to civic education than to building advocacy and watchdog capacity or strengthening coalitions of non-state actors. This change in strategic approach was made due to the recognition that citizens, civil society, and journalists required an adequate understanding of corruption and the roles of Timorese AC institutions in order to provide effective oversight. Implementation of civic education activities compensated for the lack of progress on advocacy and coalition-building in the first year of the program. While this change in strategic approach was made in efforts to compensate for the lack of progress in the first year of the program, it also reflected an assumption that citizens, civil society and journalists lacked an adequate understanding of corruption and the roles of Timorese AC institutions. By providing direct assistance to CEPAD, FOTI supported the CSO’s ongoing efforts to lead a civic education campaign on AC, thereby building its capacity. A. WHAT WAS THE EFFECTIVENESS OF ACTIVITIES? FINDINGS—COMPONENT A: CAC CAC Investigations CAC investigators benefited from a long tenure with a FOTI advisor. Interviewees in the Unit reported satisfaction with FOTI advisor’s work on standard police operating procedures in criminal investigations. The three manuals, prepared by the advisor under JISP, are considered very useful. Investigators reported that they have printed copies of the manuals and refer to them in their work. At the time of the field work, no investigations were occurring due to the fact that there was no permanent Commissioner in place at the CAC; however, the team was told that investigations would resume once a permanent Commissioner was appointed46. 46 A new Commissioner was appointed shortly after the team concluded data collection and it is anticipated that investigations will resume as planned. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 17 In meetings with the OPG, the OIG, and the SEFI, each of them rated CAC investigators’ expertise and capacity as low, despite the training, mentoring, and tools provided by FOTI. The evaluation team was informed that evidence in cases referred by CAC to OPG often lacked focus; that CAC investigators’ lack of solid legal foundation and knowledge of the penal procedural code prevented them from knowing what questions to ask witnesses, and what evidence is relevant to the investigation, resulting in gathering a great deal of irrelevant evidence and sending it all to the OPG without establishing a reasonably clear evidentiary basis for prosecution. CAC Education, Outreach, and Research CAC staff spoke emphatically about the quality of the technical assistance received for the Public Integrity and snap surveys. One individual trained remarked “we wish [all of our other advisors] were all like [the survey advisor],” who was reportedly “experienced” and “easy to talk to.” As with other popular advisors, CAC staff noted that the survey advisor worked collaboratively with his counterparts. Three rigorous surveys were conducted by the CAC with decreasing amounts of FOTI advisor involvement, representing a major increase in staff capacity. The FOTI COP noted: “they wanted to do [the surveys] all by themselves…I thought this was too much for them since they hadn’t done anything of that sort in the past, but they proved me wrong…they designed the questionnaires by themselves with [FOTI’s] inputs…If you speak to them, they will say they did it and I would say this is a fair assessment”. The staff not only know how to design and conduct the surveys, but also how to use electronic tablets to increase productivity. Both the Public Perception and Public Servant Integrity surveys furnished information on corruption that are being used by the CAC to raise public awareness of corruption in the country as well as about the CAC’s functions. These reports are publically available in both English and Tetun, which makes them more accessible to people in Timor-Leste. The surveys were created to collect data that is relevant for the design of prevention programming and strategy development, though the evaluation team did not observe any instances of the CAC using survey data for these purposes yet. CAC Prevention FOTI conducted various activities in support of building the capacity of Prevention Unit personnel to lead interactive training and engage in dialogue with public institutions and stakeholders. The first effort to train another public institution, the Customs Service, on prevention stopped after the first year due to an unproductive relationship between FOTI and Customs leaders. It has taken time to rebuild confidence between Customs and the CAC. Despite the inclusion of Customs among the CAC’s four priorities identified in the CPP, the Unit has not yet initiated support for Customs as of July 2014. FOTI’s second effort at building the CAC’s capacity to train other institutions on corruption prevention resulted in participatory development of the CPP and two training manuals. According to the CAC’s leadership, two workshops built their capacity to engage in dialogue with stakeholders on the development of the training manuals. The evaluation team reviewed the procurement manual. Written in Tetun, it is more accessible to CAC staff than if it was written in Portuguese or English. The manual is laid out to train and provide tools that guide training of other agencies by Prevention staff. It provides a general approach to identifying whether there might be corruption, the risks for corruption and actions to manage risks. The manual takes into account incentives for corruption and focuses on systems rather than individuals. It goes beyond fraud and corruption policy to integrate risk management policies into operations. It is grounded in Timorese legislation and is adapted to some degree to Timorese culture. It is interactive, uses case studies and employs small group exercises. FOTI organized 13 training workshops for CAC personnel in 2013-2014. A total of 510 hours of training were provided. The team sampled the pre- and post-tests for seven of the ten prevention Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 18 workshops. These tests showed that 72% of trainees received a score of 60% or higher in the post￾tests47. CAC leaders reported that the training improved the confidence of CAC personnel. The high number of technically appropriate trainings and the relatively high post-test scores were attributed to the strength of the second FOTI advisor. This is in direct contrast to the reports about the first advisor. FOTI’s capacity building of the Unit on prevention and procurement has enabled it to initiate two campaigns to train public institutions at the national and local level. On corruption prevention, it has begun training three institutions: SERVE- the one-stop shop for business registration; Ministry of State Administration; and the Secretariat of State for Youth and Sports. It is awaiting a signal from SEFI to begin training high level officials as part of the decentralization process. On procurement, it has collaborated with the Chamber of Commerce to train private sector institutions in four districts and plans to train in the remaining nine districts. FINDINGS—COMPONENT B: OPG Under the second advisor, training sessions mixed traditional training techniques with elaborate practical case simulations. The training programs used case studies and group discussion methodology. In some cases, training programs also helped create evidentiary document templates for use throughout the investigation and prosecution processes. Interviewees from the OPG reported that the second FOTI advisor provided valuable coaching support to prosecutors during corruption-related trials, including a high-profile trial leading to the conviction of former Minister of Commerce, Industry and Environment. In that case, judgment included the recovery of State assets amounting to $500,000. There was evidence of high level of performance of beneficiaries of the training programs delivered by the FOTI advisor during the post-training tests administered by the program. One example of this is the post-test results for the “Financial Corruption Investigations” training program carried out in January 2014 where the average test results increased by 30 percentage points. Pre- and post-test data is unavailable for activities conducted by the first FOTI advisor at OPG. A review of the manual on “Preventing and Combating Money Laundering in Timor-Leste” was conducted by the evaluation team, who found it to be a technically sound training tool and reference material for prosecutors and investigators. The manual utilizes an annotated methodology to the Timor￾Leste anti-money laundering law presenting it in the context of international law and best practices in this area. It was provided in Portuguese with a Tetun translation attached48. An inter-agency training program delivered by FOTI helped identify areas of jurisdiction and responsibilities among the institutions. National counterparts consulted—particularly at the CAC—felt that it did not have the collateral result of fostering interagency collaboration to any measurable or noticeable degree. FINDINGS—COMPONENT C: AUDIT Although the component was deemed to be “moderately performing” at the time of the MAT, the component reached targets for all of its output indicators by the end of the program. Assistance provided from the FOTI advisor was well-received by targeted staff. As the advisor had been embedded within the OIG for many years prior to FOTI’s inception, he had already built key relationships and worked to build the institution’s capacity, which contributed to his perceived effectiveness within the organization, though this effectiveness is generally attributed to the advisor personally, rather than the 47 Pre and post-test data sheets 48 OPG Indicator #3 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 19 FOTI program. Recipients of his trainings reported high levels of satisfaction with the advisor’s approach, pedagogy, and accessibility49. The reported number of person training hours exceeded the target by nearly 50%50. The majority of activities occurred in the last year of the project; over 50% of the total person hours of training reported occurred in the final four quarters of the program51. However, no concrete measures of effectiveness for his training or mentoring exist, as many of the activities did not meet the minimum three hour threshold for the pre and post-test requirement. As the majority of these capacity building sessions were conducted in a relatively informal, ad hoc basis, there is little material to review to make a sound judgment regarding the quality of instruction. Although the Inspector General asserts that the capacity of 27 technical staff has been built under FOTI, he stated that further training is required. One of the accomplishments in the audit component was the drafting of the internal audit handbook. The evaluation team obtained an English draft copy of the handbook, which covers “internal audits for ministries and their departments and subordinate offices, and other entities wholly funded by the budget52.” The handbook provides a comprehensive introduction to auditing. In addition, the handbook provides practical tools, such as checklists, in the annexes to help auditors conduct work on their own. The language of the manual as translated into English is simple and relatively easy for someone without an auditing background to understand. However, one visible difference from other FOTI-sponsored resources is the fact that the handbook is text-heavy, containing no illustrations or other visuals. Although the handbook references Timorese law, there are no real or fabricated examples of how the concepts would apply in the context of Timor-Leste. The evaluation team did not get a sense of how user friendly the manual is from the intended users, as the manual is still under review by the OIG and, as such, is not currently in use. Likewise, none of the other written guides listed have been approved by the appropriate entities, therefore, it is too early to understand the effectiveness of these tools. FINDINGS—COMPONENT D: PROCUREMENT Little evidence exists to document the effectiveness of the completed activities under the procurement component. The TNA identified some salient capacity building needs within the RDTL that also applied to other government actors (e.g. challenges with language, internet connectivity, etc.53). The program undertook some of the activities recommended by the TNA report, such as the development of a TOT and code of ethics (though it was not procurement specific, as prescribed by the TNA); however, other recommendations were far beyond the scope of the program54. Ultimately, relatively few activities occurred under this component, which limited the potential to achieve results. The quality of technical assistance provided to SEFI and CAC on the creation of the ten principles of procurement and procurement manual/TOT was reportedly high from the counterparts. The process of creating these products was said to be very collaborative and both advisors were described as being “proactive.” The development of the ten principles and the manual/TOT was important not only for the creation of the products themselves, but also served as valuable learning opportunities for counterparts. All of the interviewees from SEFI and CAC who worked on these products noted that preparing these 49 Final RRT, 2014 50 Ibid 51 Ibid 52 Internal audit handbook (English draft copy) 53 Procurement TNA 54 ibid Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 20 products with their respective advisors prepared them to conduct similar activities in the future. However, given that no pre- or post-tests were conducted for this technical assistance, evidence of the quality is purely anecdotal. Nevertheless, the ten principles of procurement and the manual/TOT have both been used to train relevant stakeholders, including national-level ministries and district-level private sector firms without FOTI assistance, which suggests some level of effectiveness. FINDINGS—COMPONENT E: PARLIAMENT The final evaluation team also conducted a thorough review of the technical products developed by the second FOTI advisor to the National Parliament. Among the documents reviewed was the FOTI advisor’s technical analysis of the amendments to the Anti-Money Laundering legislation; the evaluation team found the review to include comparative best international practices from jurisdictions with similar legal framework as that of Timor-Leste55. Relevant considerations included: International Financial Action Group recommendations; Portuguese legislation; articles of the UN Convention Against Transnational Organized Crime, UNCAC, and innovative proposals from the European Parliament and Council regarding asset recovery in the European Union. Likewise, a review of the technical analysis of the Financial Information Unit (FIU) decree law and recommended amendments similarly revealed an incorporation of best practices from international conventions such as UNCAC and others. The team also reviewed the advisor’s annotated manual on Law 17/2011 (the anti-money laundering law) which is a tool not only for reference for Parliamentarians and legal practitioners but also for teaching at law schools and at the Judicial Training Centre. The criteria used in evaluating the manual included looking at the technical quality of the annotated comments made by the author at the end of each of the articles of Law 17/2011, the comments and comparisons with international conventions and best practices, and the fact that it is provided in both Portuguese and Tetun. To the evaluation team’s knowledge, this is the first Timorese law published in an annotated manual format. A testament to the effectiveness of the second advisor’s work was that all but one of FOTI program’s recommendations made during the review of the FIU regulations were incorporated into the decree law on the FIU. The only major FOTI recommendation that was not approved by the Council of Ministers was the one granting autonomy to the FIU56. Similarly, the advisor mentored Committee A during its review of the legislation recommending substantive changes to Act 17/2011 to Prevent and Fight Money Laundering and Terrorism; as a result of this support, the final document sent by Committee A to Parliament was approved unanimously by Parliament on July 12, 2013 without further amendments57. The evaluation team was not able to obtain much data in terms of activities carried out by FOTI in the area of legislative oversight, as no training materials or pre- and post-tests were available for the team’s review. In addition, the team gathered little anecdotal information about the quality of the first advisor, as her work was overshadowed by that of the second advisor. Interviewees at the Parliament emphatically described the second advisor as a “gold mine” for FOTI and for Parliament counterparts. A sample of pre- and post-tests conducted for trainings conducted by the second advisor indicate the trainings successfully imparted knowledge in participants. 55 Parliament indicator #2 56 QR 2013 Q4 7 57 FOTI Final Report, 2014; QR 2013 Q4 7; Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 21 FINDINGS—COMPONENT F: NON-STATE ACTORS The effectiveness of the monthly investigative journalism workshops in 2012 was limited by the length of time between sessions. The evaluation team was told by several interlocutors that the number of corruption-related articles had indeed increased, but few of them produced outside of a training environment were well-researched. The four-week classroom and practical training in investigative journalism was much more effective in building the capacity of journalists. A review of the syllabus revealed that the subjects were essential for building skills in investigative journalism. The pre- and post￾tests for the course as a whole showed that all participants increased their knowledge and that 13 out of 15 achieved a score of 60% or higher on the post-test. The evaluation team met with some of the training attendees who reported that they thought this training truly increased their investigative journalism capacity and built the skills they needed to find information for stories on corruption. They were especially motivated by the coaching of the FOTI expert. They said that now they are better able to interview sources and whistleblowers, use the Internet and public databases, gather adequate evidence and maintain their credibility. Journalists who participated in the training were faced with many formal and informal obstacles to accessing public information. Publishers and media owners, whose publications and broadcasting stations are highly dependent upon government funding, prevented any of the stories from being published or broadcast. They recognized that the barriers to getting stories published were unnecessarily high, including harassment from family members of sources and opposition from publishers and owners. The trainer noted, that, “All working groups were denied access to basic public information, such as state contracts, registry information of companies, licenses, subsidies, permissions awarded by public entities, lists of public purchases, public payments, etc. In every single visit to a state institution the access to public information was denied58.” The evaluation team found that the two courses on monitoring and analyzing the state budget provided a clear and succinct analysis of the Timor-Leste budget process, particularly of the intricacies of the Petroleum Fund. The team met with CSO trainees, who expressed their strong appreciation for the knowledge and budget analysis skills they gained from the courses. They said that the training was very useful for their advocacy work, particularly in reforming the education sector and promoting transparency and accountability. The pre- and post-tests for the basic course showed that all 16 participants increased their knowledge, scoring 60% or higher on the post-test. The pre- and post-tests for the advanced course showed that all participants increased their knowledge, but that only 2 out of 10 achieved a score of 60% or higher on the post-test. To understand the effectiveness of the civic education campaign, the evaluation team assessed the quality of several of the products and interviewed some of the users. CEPAD was already committed to civic AC and welcomed FOTI support. It is committed to using the civic education products on corruption for a long-term conflict-mitigation campaign. Its storybook, “Baino and Binoi,” written in Tetun, is about a boy and a girl who want to become pilots. One studies hard, while the other uses corruption, collusion, and nepotism. The book ends with definitions of corruption, collusion and nepotism that most children will be able to understand. CEPAD’s AC action manual, written in Tetun, defines corruption and its causes, provides an overview of the AC institutions, and identifies each of the corruption crimes contained in the penal code. The 58 Investigative Journalism Consultant Report Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 22 appendix includes a model format for complaining about corruption and three examples of how the form would be filled out for different complaints. The manual would be useful in guiding training of CSOs and grassroots organizations on understanding and taking action on corruption. However, it is unlikely to build analytical skills for engaging in the policy process. The team viewed two TV programs and found them to be germane for civic education. In one program, the Director of CEPAD moderated a discussion in Tetun among the CAC Commissioner, the Vice President of Parliament and the Prosecutor General on what is being done to combat corruption. Several average male and female citizens were asked to define corruption, identify AC institution and to make recommendations on how to prevent and deal with it. While the team was unable to assess the quality of the GOPAC radio programs, the participants and themes appear similar to those of CEPAD. GOPAC is planning to use the radio programs to develop videos and visual presentations for high school and university students in 13 districts. The evaluation team found that CEPAD has reached less than 200 people through its presentations of AC civic education products at its Peace Houses. Only 38 persons attended the Baucau Peace House event. However, the campaign is nascent and plans to expand its reach. The Peace House staff reported that the quality of the materials and presentations was appreciated by the attendees. CONCLUSIONS FOTI assistance built the capacity of staff at target organizations to detect, investigate, and/or punish corruption cases through various means. By and large, insufficient baseline data is available to determine the precise level of capacity built under FOTI among all components. The limited pre- and post-test scores available largely indicate successful knowledge transfer as a result of FOTI trainings; however, as the majority of the technical assistance activities occurred on an ad hoc, informal basis, there is little concrete evidence to concretely demonstrate successful knowledge transfer. Various types of evidence exist to show FOTI’s effectiveness across programs:  CAC-Generally speaking, FOTI assistance helped staff at CAC to build capacity in their lines of work. o Investigations Unit: The little data available suggests that FOTI was largely successful in helping investigation unit staff with limited previous technical background to begin developing relevant skills. FOTI missed an opportunity by providing technical assistance on investigation of complex crimes, rather than shoring up gaps in basic legal knowledge which could have facilitated interactions with other AC stakeholders. The continued use of FOTI materials suggests that technical assistance provided to the Unit was technically appropriate and useful to staff. o EOR Unit: The technical assistance provided by FOTI was one of the most visible contributions to the organization. The program provided high-quality, appropriate assistance, which left a positive mark on the unit. It is evident that that the assistance provided not only built hard skills to undertake similar efforts, but also produced valuable data about corruption which is available to citizens and the public sector. The efforts also had the added value of educating the public about corruption and the CAC itself. o Prevention Unit: The program built the capacity of personnel to lead interactive training and engage in dialogue with public institutions and stakeholders. FOTI has built much of the strategic planning, knowledge, skills and tools needed by the Unit personnel to assist other public institutions in taking steps to prevent corruption. It also has built their confidence and created a sense of ownership of CAC’s prevention mission.  OPG-The FOTI advisor not only developed technically strong curricula for training programs delivered to OPG, but also used training methodology that was effective in knowledge transfer and capacity building. In addition, the advisor developed manuals that provided practical guidance Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 23 and sound technical counsel on the law. The results of the post-test results reflect improvement of knowledge and skills in the area of asset recovery methods and mechanisms by those in attendance. The training on asset recovery was provided late in the program which did not allow for further re-enforcement and follow-up mentoring. The lack of a coordination protocol between CAC and OPG, setting out the jurisdiction of each of these two institutions, contributed to distrust and turf protection at various points in the life of the program. However, the inter-agency trainings did make progress on this result more so than other efforts to promote collaboration among AC institutions in the country.  OIG-The component was effective insofar as it contributed to the overall purpose of increasing audit capacity of the OIG staff; however, the extent of this increase is difficult to discern. The component only gained traction in the last year of the project; given this short timeframe, perceived effectiveness by beneficiaries should be considered bearing in mind the long history of the advisor’s involvement with the OIG in Timor-Leste. The demand-driven approach to training and mentorship content is another factor contributing to his perceived effectiveness. As with other components, a more consultative approach to meeting the target organizations’ needs appears to have yielded better results in terms of the beneficiaries’ satisfaction with the assistance. Although the various products and activities conducted in this component appear to be of high quality, there is insufficient physical evidence such as pre- and post-tests or training curricula to make a sound generalization about the overall quality.  Procurement- The program made some initial progress in the goal to “improve procurement policies, regulations and practices” through the development of the ten principles of procurement and the procurement manual; however, it did not succeed in setting up an independent commission, creating linkages with other institutions, or even improving policies, though much of these limiting factors were beyond the program’s control. Overall, the dearth of activities under this component resulted in a small contribution to the procurement space through the development of the ten principles of procurement with SEFI and the procurement manual and TOT with CAC. The extent of this contribution is not known, as only a handful of trainings have occurred and no concrete measures of effectiveness are available.  Parliament-Insufficient data was available to come to a conclusion about the work done in the first half of the program, particularly in regard to the development of MPs’ capacity in exercising its legislative oversight authority to probe alleged cases of corruption. However, various sources indicate that FOTI was very successful during the second half of the program. FOTI program support to Parliament was instrumental to ensuring that several pieces of draft AC legislation became effective pieces of AC legislation by incorporating good international practices.  Non-state actors- FOTI conducted several high-quality activities with non-state actors towards the end of the program. o The structure of the media in Timor-Leste, with its dependence on government funding, severely limits the impact of investigative journalism training on establishing the media as a watchdog on corruption. In its four week course, FOTI provided high quality training and coaching, resulting in well researched articles with conclusions based on strong evidence. The decision not to publish a participant’s article is symptomatic of “deeper challenges” in this area, i.e. fear of retaliation from government. o FOTI provided superb training to CSOs on monitoring and analyzing the state budget. The CSOs benefited from the training, understand how to use it for their work in different sectors, and are excited to use it during the budget process that begins in October. The outcome of this training provides evidence that several CSOs at the national level have the capacity to become strong watchdogs and advocates against corruption. FOTI made a strategic error in delaying training until almost the end of the program and investing no funding in coalition building, the two original Component F activities. It also made an error in diverting resources away from Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 24 watchdog and advocacy training to civic education. While civic education on corruption is essential, it is an investment in a longer-term process of building watchdogs and advocates. o FOTI supported the development of important AC civic education products through CEPAD and GOPAC. Many of them are very high quality. It is unknown how many people were reached through these messages, whether they understood them, and how they acted on them. However, corruption has become a big topic of conversation among many Timorese, an outcome that could be the result of FOTI efforts. Through these examples, the evaluation team observed several contributing and limiting factors to perceived effectiveness. Though detectable amount of effectiveness varies by institution, generally speaking, two major factors of technical assistance have a direct influence on the effectiveness of program activities: Quality of FOTI staff and advisors: The beginning of the program was plagued by poor quality of FOTI staff. The first COP failed to build strong relationships with the partner organizations, which limited the program’s traction in all components. Hardly any of the original personnel were appropriate choices for the program. Interviewees at the CAC, OPG, OIG, and Parliament all cited difficulties with the original advisors assigned to their organizations primarily due to lack of relevant experience. All of these individuals were replaced by the end of the program, which ushered in higher demand for and satisfaction with technical assistance. Relevance of technical content: Interviewees at all organizations targeted by FOTI stated that one of the main contributors to effectiveness was the relevance of the technical content to their workloads. Program advisors responded to requests for assistance either on an ad hoc or formal basis, which significantly increased the perceived utility of the material to direct beneficiaries. The fact that—for the most part—interviewees cited continued use of knowledge or resources beyond the life of the program is a testament to the suitability of the technical assistance’s content. 3. INFLUENCE ON CHANGES IN KNOWLEDGE, ATTITUDES, AND BEHAVIORS DID IMPLEMENTED ACTIVITIES LEAD TO MEASURABLE OUTCOMES, SUCH AS CHANGES IN KNOWLEDGE, ATTITUDES, OR BEHAVIOR? FINDINGS The original contract included a list of outcome indicators that were not systematically tracked during the life of the project through the RRT. The program reported on only 9 out of 23 of these indicators in its Final Report and did not report on any of the OPG indicators, stating, “FOTI could not directly influence the list of outcomes identified for this component given the OPG’s refusal to receive mentoring support59.” They did not report on outcomes for Parliament component due to the change in 59 FOTI Final Report, 2014 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 25 focus of the activities, which rendered the outcome indicators irrelevant. These indicators were not updated before the end of the program. Some of the outcome indicators were more quantitative, such as “30 or more corruption cases that meet international standards for quality are indicted and brought to trial, with at least 15 cases resulting in convictions,” whereas others were more qualitative, e.g. “ACC [CAC] staff performing job functions predominately without technical assistance60.” Although FOTI reported on fewer than half of the outcome indicators, the evaluation team found some evidence of outcomes, particularly regarding changes in knowledge, attitudes, and behaviors. Changes in knowledge were captured for some of the longer trainings with pre- and post-tests, generally showing increased scores between tests; however, these changes are generally acknowledged as lower-level results. Given that higher-level results were not systematically tracked, we only have anecdotal evidence to demonstrate achievement at this level. Nevertheless, the evaluation team found, through discussions with staff at beneficiary organizations, indications of some promising effects of FOTI programming. In terms of attitudes, interviewees at the CAC, OPG, OIG, SEFI, civil society, and the media all cited increased confidence in their professional performance as a product of FOTI training and advising. Interviewees at OPG, the NP, SEFI, and CAC lamented that the FOTI program closed when it did, as they were all learning a great deal from the advisors and felt that the work was cut short. Informants from each of the organizations agreed that FOTI’s capacity building activities increased their skills, though when asked to provide examples of how such skills have been applied, most interviewees—regardless of organization or stature—had trouble identifying specifics. However, as indicated in the “sustainability” section, there are several notable cases of behaviors that are being sustained without FOTI assistance, such as the survey work conducted by the CAC. CONCLUSIONS The structure of the outcome indicators limited the program’s ability to report high-level results. Furthermore, most of the planned outcomes were overly ambitious within the short timeframe due to the low levels of institutional and human capacity, tensions between target organizations, and the social context. The program was also slow to gain traction, due to both personnel and context issues, and very few activities were successfully completed in the first years of implementation. Although it is unrealistic to think that the program could have achieved outcome-level results after only one year of robust programming, the prescribed outcome indicators could have been adjusted to measure the work actually completed, which would have allowed for a more nuanced picture of the results actually achieved. A more systematic tracking of outcomes, particularly changes in knowledge, attitudes, and behavior at the individual and organizational levels would be more appropriate indicators than tracking the number of investigations or prosecutions, as the program had a limited ability to contribute to those results. As FOTI built upon the work of and used some of the same advisors as JISP, more sensitive tools would be required to parse out the specific results of each program. Given the above, as well as the timing of the evaluation, which occurred merely three months after the end of the program, it is hard to detect higher-level outcomes of the program at this time. 60 Task Order AID-486-TO-11-00001, 2011 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 26 4. SUSTAINABILITY & LESSONS LEARNED A. HAVE KEY IMPLEMENTED ACTIVITIES BEEN SUSTAINED? FINDINGS The evaluation team found that three months after FOTI ended, program activities are being sustained to a certain extent. Continuation of activities takes three forms: Continuation of advisors: After the program ended, the OIG and Investigation Unit at CAC hired the FOTI-sponsored advisors as full-time staff through their own budgets. The advisors continue to support these organizations through continued training and mentoring on an ad hoc basis as well as by conducting line functions. Continued trainings and outreach: Certain activities such as trainings and other forms of outreach supported by FOTI are still occurring in the absence of the program. The number and scope of these interventions is impressive in the short period of time since the program ended:  CAC: The Prevention Directorate has delivered procurement corruption prevention training to the Chamber of Commerce and in four of the country’s districts. The Prevention Directorate plans to work with the Chamber of Commerce to roll out the training in the remaining nine districts by the end of the 2014. Similarly, it has used FOTI’s corruption prevention manual to train the Secretariat of State for Youth and Sports, the Ministry of State Administration and SERVE—the one stop shop for business registration. It has plans to replicate this work with the Customs Service and other institutions. While outreach and research activities have slowed down since the end of FOTI, the EOR Unit plans to continue the work in the future.  Procurement: SEFI continues to distribute the ten principles of procurement to ministries in Timor-Leste to guide procurement processes and remains interested in advancing procurement reform. SEFI is keen to continue working with donors to improve performance of government bodies across the country, both on procurement and other institutional strengthening issues.  Civil society: CEPAD is collaborating with media organizations to replay radio and TV programs and to reach a wider audience. Community radios and TVTL have agreed to gather data on the number of listeners and watchers.  Parliament: GOPAC will use its FOTI-supported radio programs as the basis for television programs and visual presentations, which it plans to use for meetings with high school and university youth throughout the country. Use of written deliverables: The program developed a wide variety of written resources that continue to be used by them, including:  CAC: procurement and corruption prevention training manuals serve as references to both CAC and the organizations it trains through TOT.  OPG: Financial Investigations Manual is being utilized by prosecutors when they are investigating or prosecuting corruption cases.  Civil Society: CEPAD has identified alternate funding for widespread distribution and discussion of the FOTI-supported AC action manual it prepared with FOTI funding. The AC storybook has been included by the Ministry of Education in its curriculum. The NGO already has printed 60,000 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 27 copies for distribution in 2015, and the Ministry is planning to include the book in its budget for subsequent years. It appears that uptake of written and other physical resources have been slower in other organizations:  CAC: the electronic case management system (SIAK) started under the USAID’s JISP program and continued under FOTI is not operational.  OIG: None of the products produced for the audit component have been formally approved for use at the OIG.  Parliament: FOTI’s report on “Preventing and Combating Money Laundering in Timor-Leste” was on the shelf but it appears that much of the information had not been disclosed internally and more work would need to be done in order to disseminate it.  Media: one of the investigative articles produced with FOTI support has still not been published due to “lack of evidence,” though the author and program staff suspect that this may actually be out of fear of backlash. CONCLUSIONS & LESSONS LEARNED The program left behind both physical resources, such as documents, and intangible resources in the form of skills. By and large, FOTI program activities have been continued beyond the life of the program among all components. Activities are sustained in organizations where there is budget and political will to do so. Frequency of these activities has slowed, most likely due to the absence of external support, which suggests that beneficiary organizations may require external pressure to sustain the momentum created by the program, particularly in its final year when most of these activities took place. The incidence and absence of sustained activities offers a few relevant “lessons learned”:  CAC will need several more years of support and experience to meet its mandate. Unlike AC commissions in other countries, it does not appear to be politicized so far. However, donors should be careful in their expectations for assisting new AC commissions, especially in post￾conflict environments.  When placing full-time advisors into organizations, donors should build sustainability into the program from the beginning through creating domestic capacity for training in order to avoid overreliance on the advisors.  Donors should consider the structural barriers to conducting investigative work, particularly in the media.  On-demand or upon-request training often produces better results, counterpart ownership and sustainability than training identified and planned by donors with minimal counterpart input.  Donors should invest heavily in establishing an understanding of division of responsibility and interagency procedures before building the capacity of relevant anticorruption agencies. B. WHAT IS THE LIKELIHOOD FOR FUTURE SUSTAINABILITY? FINDINGS The sustainability of the program’s results varies by component. There were several notable positive enablers to sustained results noted by the evaluation team:  Sense of ownership: Program beneficiaries—particularly at the CAC—signaled high degrees of satisfaction regarding activities that were responsive to their needs and conducted through a highly participatory approach. Prevention Unit staff cited both the CPP and the procurement manual examples of ideas they had which were realized with the support of FOTI advisors. Both of these documents were produced through a series of workshops, which provided staff Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 28 opportunities to inform the content of the documents. CAC staff reported that they “know the process” and “own the process,” which will enable them to replicate it.  Transferability of skills: Creation of certain deliverables resulted in other positive by-products, such as increased confidence to replicate the processes without an advisor. Staff at CAC reported that they developed public speaking skills and also learned generally how to go about preparing a TOT. Interviewees at SEFI also noted a similar result from participating in the development of the ten principles of procurement and code of conduct.  Gender integration: From the outset in its first work plan, Casals planned to promote gender equality primarily through working with women’s groups and advancing women’s issues such as gender responsive budgeting and domestic violence61. However, the program reported a lack of interest as one of several reasons why it was unable to advance as robust of an agenda as initially planned62. Nevertheless, the program did include women in program activities where possible63 and contributed to the body of knowledge about the dynamics surrounding women and corruption through the surveys conducted by the CAC64.  Tetun language: Most if not all of the written resources are available in Tetun, which makes them more accessible to the people of Timor-Leste. Although Portuguese is the country’s other official language, many citizens have limited proficiency in it, particularly those lacking legal training.  Use of technology: Some of the resources produced by FOTI are available on the internet or on discs provided by the program, as with the case of the “Preventing and Combating Money Laundering in Timor-Leste” report for the OPG and Parliament. In addition, FOTI supported the development of the SIAK system after USAID’s JISP supported its design and initial development. While it is not yet operational, small tweaks to the system could make it useable. At the CAC, the program left tablets that the researchers can use to conduct surveys in the future, which is a more cost-effective and error-limiting way to collect raw data. Finally, the civic education campaign conducted with CEPAD made use of media such as radio and television; although these are not new forms of technology, they do help disseminate information to a wider audience in the country, particularly people living outside of Dili and/or those that may be illiterate. Likewise, certain factors threaten the sustainability of the program’s results:  Lack of written guidelines: Although the program created and left behind many physical resources as noted above, some of the interventions did not generate written guidelines on how to perform tasks or develop skills, as is the case with the CAC’s Investigations and Education, Outreach, and Research Units. In the absence of a defined training program, staff that leave these units will likely take their knowledge on these subjects with them.  Reliance on external advisors: All organizations interviewed stated a need for additional support from advisors, though the OPG notably stated its preference for national experts. In the case of the OIG and CAC Investigations Unit, FOTI-sponsored advisors served the organizations previously and show no signs of leaving their positions in the near future.. The evaluation team did not observe any evidence that these advisors are systematically capacitating their counterparts in order to cascade the skills and wean the organizations from them. 61 Work Plan Year 1, 2011 62 Work Plan Year 2, 2012 63 FOTI Final Report, 2014 64 Integrity Survey of Public Servants in Timor-Leste Report, 2012 Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 29  Limited effective cooperation: As described in previous sections, FOTI achieved mixed results in its objective of promoting cooperation among AC actors in Timor-Leste. However, the CAC recognizes the need for increased cooperation as mentioned above. The team observed lingering tensions between the organizations due to lack of clarity in their mandates. CONCLUSIONS Despite the challenges of creating sustainable results in such a short amount of time, the FOTI program laid the groundwork for continuation of progress made during the program’s lifespan. However, as with any program, there are several barriers to sustaining the results, particularly in the long-run. The fact that the program ended shortly after completing its busiest and most successful year limited the program’s ability to address the barriers and bolster the enablers of sustainability and create the conditions for continuation of results. It is unfortunate that the program could not be extended in order to mitigate some of the threats enumerated above. Nevertheless, current and future programs might consider how to learn from or capitalize on the enablers to sustainability. RECOMMENDATIONS Recommendations on Logic 1. Future AC threshold programs should carefully consider organizational readiness of beneficiary organizations before committing to support them and/or designing an approach. It is vital to first understand the political will within the organizations and the stage of their development. Donors should consult with the target institutions during the program design phase in order to develop a relationship at the outset and better understand their organizational priorities and modalities. 2. Although best practices for AC threshold programs dictate that “it is imperative that countries have the institutional capacity and political will to formally investigate and, where appropriate, prosecute those crimes,” it is important to understand the social dynamics of a country, particularly if it is coming out of a recent conflict65. It is important that there be an environment in which people feel safe conducting investigations and furnishing evidence, and further, that prosecutions not be used to exact political revenge on opponents. A focus on corruption investigation may or may not be appropriate depending on the social context of a country. Recommendations on Activities Planned & Implemented CAC 1. CAC will require further program support for the short to medium-term in order to solidify the gains it has made as a corruption-fighting institution in Timor-Leste. CAC staff would benefit from basic Portuguese language training to enhance their understanding of the legal AC and penal procedural law framework of Timor-Leste. A program focused on legal foundations and the criminal procedural code could be modeled from the prosecutors training at the Judicial Training Centre and adapted to CAC. In addition, investing in the finalization of the SIAK system may facilitate improved case management of corruption investigations. 2. The CAC Prevention Directorate identified decentralization as a top priority in its Corruption Prevention Plan. It also has demonstrated the capacity to train other public institutions in 65 USAID TCA Lessons Learned Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 30 corruption prevention, both generally and with regards to procurement. In support of the Ba Distrito Project, USAID should support CAC’s campaign to educate districts and sucos in preventing corruption and implementing transparent and accountable procurement systems. 3. USAID or other donors should find appropriate channels to encourage the CAC EOR Unit to continue planned research as survey activities, in order to ensure that skills cultivated by the FOTI program are maintained and put to best use. In particular, future iterations of the public perception and public servant integrity surveys would be valuable in measuring changes across time, which could help to inform future AC programming, either by external donors or by actors within the RDTL. OPG 1. Given the OPG’s imminent establishment of a unit dedicated to corruption cases, donors should consider bolstering its capacity to investigate and prosecute complex crimes. Prosecutors noted a number of areas where they require further training and technical assistance on basic and advanced training courses in financial investigative techniques such as: financial crimes (i.e. money laundering; anti-terrorism financing); transnational crimes (i.e. drug trafficking; human trafficking; organized crimes); and asset recovery. Such trainings would also be beneficial for other actors in the judicial system, including judges, the judicial police, and CAC investigators, in order to increase the likelihood of mutual understanding, cooperation, and effectiveness. AUDIT 1. The audit component was not deemed to be one of the great successes of the FOTI program; however, donors should continue to invest in strengthening internal and external audit capacity within the RDTL. The importance of audit skills will increase as the country moves toward decentralization, especially during implementation of the Program for Accelerating Suco Development. Although the Ba Distrito program will teach suco council members financial management skills, building capacity to audit these expenditures is beyond the current scope of the program. It would be wise to leverage the TNA report prepared by the FOTI advisor, as it contains valuable information on what the audit needs are and how they can be addressed. PROCUREMENT 1. Public procurement continues to be a source of corruption, which has yet to be addressed. Large contracts are being awarded to foreign firms, which some perceive to be a bigger risk of loss in terms of dollar values. USAID or other donors should leverage the procurement TNA produced by the program and update the information based on new institutional realities as a starting point for designing a comprehensive approach to this issue. In particular, donors may consider training the CAC, OIG, and its line ministries on monitoring public works projects, including technical advice on how to do evaluations of construction projects. PARLIAMENT 1. Given the short duration of FOTI support to the National Parliament both in drafting AC legislation and serving in a financial oversight capacity, MPs could use follow-on support to bolster their competence in these areas. Donors should consider providing technical assistance in the forms of training or mentoring to Committee A on: financial oversight of budgetary expenses, government contracts and high-level government appointments; technical analysis of AC-related legislation; and/or holding debates and public hearings held on AC-related legislation. NON-STATE ACTORS 1. FOTI has supported the development of excellent civic education material and media focusing on corruption, but did not take steps to share them in an interactive manner with many Timorese. Civic education is important for building a citizenry that holds public officials accountable. Ba Distrito could increase its potential impact by adding funding for a civic education component that used the work of CEPAD and GOPAC to reach out personally to citizens in the target districts. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 31 2. FOTI, through Lao Hamutuk, led an excellent training of CSOs on budget monitoring and analysis, which was well received by national CSOs. While CSOs at the local level lack the capacity of these CSOs, it would be very useful to train them on monitoring the budgets of district administrators and sucos. This activity could build on the corruption-focused civic education proposed for Ba Distrito target districts in the recommendation above. 3. While the Ministry of Education has officially included CEPAD’s storybook on concepts of corruption, collusion and nepotism in its curriculum as a reference for the upcoming school year, the budget for its printing post-2015 is not guaranteed. USAID could institutionalize the book by funding the printing of another set for 2016. 4. Both journalists and CSOs suffer from unofficial and official barriers to access to public information. The Parliament has identified the need for a freedom of information law, but has not made it a priority. Improved access to public information would benefit citizens at both the national and local levels. USAID should support CSOs and journalists to undertake an advocacy program on freedom of information at both the national and district levels. RECOMMENDATIONS ON OUTCOMES 1. Future programs should set realistic indicators and targets that can track the progress of programs’ results against realistic outcomes. Results should also be timely, useful for program management, and attributable to the program’s activities. Indicators should be periodically reviewed and modified as needed and appropriate, which is likely to be the case in an evolving context, such as that in Timor-Leste, where institutions are in flux. RECOMMENDATIONS ON SUSTAINABILITY 1. It may be time for international agencies supporting capacity development programs in Timor￾Leste to consider a paradigm shift regarding the model of hiring long-term advisors. Donors should consider imposing limits on how long they should support individual advisors within a given ministry. UNDP in Timor-Leste has experimented with having advisors develop “exit strategies.” It might be worthwhile for other donors to embrace similar practices. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 32 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK Social Impact (SI) is pleased to present this evaluation work plan to the United States Agency for International Development (USAID) and the Millennium Challenge Corporation (MCC). This document presents three approaches for conducting a final performance evaluation66 of the “Fostering Transparency Initiative” (FOTI) Threshold Program in Timor-Leste. This document was produced following the findings from SI’s Senior Monitoring and Evaluation (M&E) Technical Advisor, Patricia Delaney’s, final evaluation planning and design trip to Dili, Timor-Leste between February 23 and March 1, 2014. During this field visit, Delaney consulted closely with technical representatives from USAID, FOTI, FOTI counterparts, and MCC headquarters. Please see Annex A for the Scope of Work (SOW) from the planning and design trip. The three evaluation approaches outlined herein were designed following the trip in close consultation with MCC technical representatives and have drawn on sample evaluation provided by MCC. The document includes our proposed evaluation approaches; primary evaluation questions; data collection methods; team composition; and anticipated evaluation timeline. Please see Annex B for a detailed evaluation design matrix with relevant indicators and data sources for each evaluation question and broken out by FOTI component. Please see Annex C for our proposed evaluation calendar and Annex D for FOTI’s Results Reporting Table (RRT). SI’s M&E Support In March 2011, SI was contracted by USAID and MCC to assist RDTL in the M&E of the implementation of the six program components of FOTI. During the program’s start-up, SI’s Senior M&E Technical Advisor worked with FOTI staff to development the overall monitoring system and provided as needed M&E technical assistance throughout the contract life. Please see Annex D for FOTI’s RRT developed by SI in close consultation with USAID, MCC, and FOTI staff. Midway through its three year project life, SI completed a comprehensive Midterm Assessment of FOTI’s performance to date. In August 2013, Delaney completed an oversight visit to informally assess program progress subsequent to the mid-term evaluation. In February 2014, Delaney conducted final evaluation planning and design trip. Evaluation Purpose This final performance evaluation is intended to provide an objective, rigorous, and evidence-based analysis of FOTI’s overall performance. Specifically, the evaluation will look to identify, review, and assess the following four aspects: I. FOTI’s Program Logic. II. Activities Designed and Implemented. III. Influence on Changes in Knowledge, Attitudes, and Behaviors (if any). 66 N.B.: This document uses the terminology preferred by USAID’s Evaluation Policy. The evaluation type outlined in this document could also be called a “summative” or “ex-post” “process” evaluation. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 33 IV. Sustainability and Lessons Learned. The primary audiences of this evaluation are interested stakeholders within USAID, MCC, and the RDTL. The secondary audiences are FOTI and Casals staff, as well as other interested stakeholders within Timor-Leste and the US. The evaluation will (1) provide lessons learned for MCC, RDTL, USAID, and other stakeholders in implementing future anti-corruption activities; (2) uphold MCC’s commitment to maintaining accountability and measuring program results; (3) provide practical lessons for current and future Threshold partners. Evaluation Questions The primary evaluation questions are organized along the four aspects outlined above. These purposes were identified in accordance with international best practice identified by the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD-DAC) and following requests from key stakeholders at MCC and USAID. Key Questions % of Focus Relevant Components Comments 5. Program Logic a. Were the original six program components relevant to the Timorese context? (E.g. were they based on initial analysis of assumptions, risks, and external factors?) b. Were any changes made to the program logic during the implementation? If so, were they appropriate? 5% All six components We know that FOTI underwent multiple revisions to the program logic and project design during the three years of implementation. During preliminary interviews in February 2014, several stakeholders expressed strong opinions on the efficacy of the original and subsequent project design. 6. Activities Designed and Implemented a. Were planned activities actually implemented? Why or why not? b. Were any new activities designed and implemented? If so, why? c. What was the effectiveness of activities? 75% 5 of 6 components The bulk of the evaluation effort will be focused here. While no significant activities were completed in Component D: Procurement, all other components had multiple activities throughout project implementation. 7. Influence on Changes in Knowledge, Attitudes, and Behaviors a. Did implemented activities lead to measurable outcomes, such as changes in knowledge, attitudes, or behavior. 10% A: Anti￾Corruption Commission E: National Parliament F: Watchdog & Advocacy (Civil Society) Due to the long-term nature of behavioral change and the timing of the close-out of the FOTI project, we anticipate that we will only be able to capture a limited amount of data assessing this level of influence. Based on the initial document review and the key informant interviews conducted in February 2014, it appears that Components A, E, and F are most likely to be the places where short-term impacts are seen. 8. Sustainability & Lessons Learned a. Have key implemented activities been sustained? 10% All six components USAID and MCC have requested a detailed analysis of both the lessons learned and the current prospects Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 34 b. What is the likelihood for future sustainability? c. What are the key aspects of FOTI that should be continued in the near term? d. Which aspects of FOTI would be appropriate to replicate in other contexts? for sustainability for each component, including an analysis of likely or possible continued funding from RDTL or other donor sources. An analysis of lessons learned will be conducted for all six components. A detailed analysis of sustainability and recommendations for near term programming continuation will focus on Components A, E, and F. Please see Annex B for a detailed evaluation design matrix which breaks down each of the primary evaluation questions by component and identifies relevant indicators and data sources. Possible Evaluation Approaches Below we identify a menu of three possible evaluation approaches for consideration by USAID and MCC. We are confident that we can effectively implement any of these evaluation options and have sufficient budgetary resources to operationalize any one of these scenarios. We look forward to the opportunity to discuss the risks and opportunities associated with each of these proposed approaches. Proposed Methodology & Comments Risks & Opportunities Option 1: Comprehensive Assessment Evaluation of all four aspects in all six component areas. Requires three international consultants (Delaney+2). Includes fieldwork outside of Dili to measure impact of “civic education campaign.” Thorough examination of all components, including those with limited (or no) actual implementation. Fieldwork to measure impact of civic education campaign will be difficult to conduct due logistical consideration; however, preliminary data indicates that this is the precise area where the biggest positive impacts from the project may be found. Option 2: Prioritized Assessment Evaluation of all four aspects in all six component areas. Requires two international consultants (Delaney+1). All fieldwork conducted in Dili. Smaller sample sizes than Option 1 (e.g. fewer interviews, focus group participants). Less comprehensive analysis of Components B, C, and D (Prosecutor General, Audit, and Procurement). Thorough examination of the highest performing components (Parliament, Anti-Corruption Commission, and Civil Society). Less detailed evaluation of lower performing components. Ability to measure “behavior change” is more limited than in Option 1. Cost effective option. Option 3: Targeted Assessment Evaluation of all four aspects in only three components (Anti￾Corruption, Civil Society, and Thorough examination of the highest performing components (Parliament, Anti-Corruption Commission, and Civil Society). Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 35 Parliament). No field visits outside of Dili. Requires two international consultants (Delaney+1). Smaller sample sizes than Option 1 (e.g. fewer interviews, focus group participants). No evaluation of lower performing components. Ability to measure “behavior change” is more limited than in Option 1. Most Cost effective option. Data Collection Methods The evaluation will consist of three components: 1) a comprehensive desk review of relevant documents and analysis of key informant interviews conducted during design trip; 2) three weeks of field data collection in Timor-Leste; and 3) analysis, report writing, and final presentations. Overall, the team proposes to use five methods of data collection: 1) document review and analysis using project performance data, 2) key informant interviews, 3) small group discussions, 4) focus groups, and 5) structured and unstructured site observations. This mix of data collection sources will allow SI’s team to obtain an in-depth understanding of the multiple phenomena present in FOTI’s implementation, mitigate the bias inherent in any one specific research method, support sound analyses, and draw accurate conclusions. o Desk Review Prior to fieldwork, the team will prepare and complete a detailed desk review. As an initial step, the team will compile and organize the project documents reviewed during the life of SI’s contract. The team will then work with Casals headquarters staff to identify and compile any additional documents important for the team to review to get a complete sense of FOTI operating context and implementation. The desk review findings will be used to update the evaluation work plan (as needed), develop draft interview protocols, and for decision-making concerning sampling. The Team Leader, Patricia Delaney, will work closely with the local staff already identified to prepare meeting schedules and document translation in advance of the arrival in the field. Primary sources of data for the desk review include:  Design and overview documents.  Monitoring, evaluation, and reporting documents.  Training pre and post tests.  Training materials and manuals produced by FOTI.  External publications related to corruption in Timor-Leste. o Fieldwork Fieldwork will consist of three phases: previously conducted field interviews (conducted by Delaney in February 2014); future Timor fieldwork; and follow-up DC-based and remote interviews. Initial Timor-Based Interviews Following her initial document review, the Team Leader began with a series of in-country interviews. These interviews were intended both to inform the design of the final evaluation and to capture important qualitative data from key stakeholders prior to FOTI’s close-out. This step was critical as several key stakeholders will not be available in-country during the proposed final evaluation timeframe and consequently, these interviews were the only opportunity to gather their perspectives. Interviewees who will not be in country during the final evaluation phase are marked with an asterisk (*). Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 36 These initial interviews revealed clearer understandings of the issues, challenges, and prioritizations/re￾prioritizations that occurred during implementation. The results from these interviews have been utilized directly in the drafting of this evaluation work plan. These initial interviews also provided an important opportunity for SI to pilot and refine data collection instruments. Timor-Based Fieldwork We anticipate that this field-based portion of the evaluation will take three weeks to complete. Most work will be conducted in the capital, Dili, although one approach also envisages a fieldtrip to one district location outside the capital. The Team Leader and the rest of the team will work in parallel during this phase of the evaluation work. They will meet regularly during the data collection period to triangulate, iterate, and continue to refine the research questions and methodological instruments. The team will check in regularly with MCC headquarters and with relevant counterparts at USAID/Timor-Leste. Follow-Up DC-Based Interviews Following their Timor-based fieldwork, the team will interview any DC-based stakeholders, additional stakeholders identified during fieldwork, and likely also following up with certain key informants to help the team fill in any information gaps. These will be conducted remotely via Skype or phone. Analysis and Writing Analysis and Report Writing The team will use parallel analysis to examine the evidence from its document review, key informant interviews, focus group discussions, and observations. In this “methods triangulation” analytical approach, the team will analyze data related to an evaluation question, and relevant indicators, obtained using different methods in parallel, and then across our data collection methods. Analysis and Report Writing (July 2014) SI will follow MCC guidelines concerning the process of soliciting feedback from the implementing partner, Casals, and key RDTL partners. SI will allow three weeks to receive comment and feedback from these partners and will take one additional week to incorporate revisions based on said feedback.67 Submission of Draft Evaluation & Revisions (August-September 2014) SI will submit a draft final evaluation to the MCC technical team early in August 2014. This timeline should allow sufficient time for review, revision, and resubmission before the end of the SI contract on September 30, 2014. Evaluation Team Composition and Roles The evaluation team is composed of the following individuals:  Patricia Delaney, Team Leader. Dr. Delaney will be responsible for ensuring the overall technical delivery of the contract. This includes managing the development all of written deliverables and providing final review and approval of all deliverables’ technical quality. Dr. Delaney will be involved in all phases of the evaluation including the desk review, data collection, and the analysis and report writing. Delaney will take the technical lead for data collection for Component F: Civil Society. She will lead on writing a majority of the final report but is also 67 Please note that this requirement does not appear in our contract; however, SI believes this follows good evaluation practice and will follow MCC’s suggestion. Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 37 expected to delegate certain sections to other evaluation team members, reviewing and approving their written contributions before including them in the report.  Richard Gold, Anti-Corruption and Civil Society Expert. Mr. Gold brings more than a decade’s experience working with civil society, anti-corruption, and rule of law programming. In Timor-Leste, he helped initiate a seven-year pilot program for the legal empowerment of the poor and provided technical leadership to a justice institution strengthening program. In Bangladesh, he managed an anti-corruption program with civil society, journalism, legislative and audit components. In the Democratic Republic of Congo, Mr. Gold provided technical assistance on judicial independence to the Bar and developed the monitoring and evaluation plan for USAID’s post-conflict governance, anti-corruption and civil society program. In Egypt, he led an 11-person team to analyze, design, procure, manage, monitor and evaluate a $250 million portfolio of dozens of programs in democracy, civil society, anti-corruption and rule of law, including improving criminal, civil, family, and administrative courts, and integrating women into the judiciary. In Jordan, he assessed the country’s progress establishing the rule of law, working closely with its Anti-Corruption Commission. Mr. Gold holds a Master of Public Affairs degree from Harvard University’s Kennedy School of Government, as well as a Master of Public and International Affairs degree from University of Pittsburgh.  Carl DeFaria, Public Administration and Anti-Corruption Expert. Mr. DeFaria is a public policy and anti-corruption expert, and a member of the UNDP Democratic Governance Roster of Experts in Public Administration and Anti-Corruption. He brings over fifteen years’ experience in the justice reform sector, having served as the lead technical expert on projects throughout Southeast Asia, the Pacific region, Africa and Eastern Europe, including extensive experience in Timor-Leste (over 9 assignments). For example, working with UNDP’s security sector reform project, he drafted the Organic Law and the Internal Organic Structure Regulations for the Civil Security Police Guards of Timor-Leste. As the Director for the Democratic Governance Unit for UNMIT in Timor-Leste, he set up a new unit looking to integrate transparency, anti-corruption, civil society, media, M&E, and governance programming. Mr. DeFaria also has extensive experience leading evaluations of development projects. As lead researcher for the Perception Index Study for UNDP’s Civic Education Project, he developed an impact study methodology, to collect both quantitative and qualitative data. Mr. DeFaria holds a J.D. from York University Law School in Canada and was a member of the Canadian Parliament for fifteen years. Mr. DeFaria is a fluent Portuguese and English speaker.  Local Logistician: Pedro Da Silva. This person will lead the coordination of in-country data collection efforts. They will be asked to advise on possible data collection sites; arrange interviews; advise the team on local customs and appropriateness of interview protocols; and alert the team if they suspect any interviewees may be misrepresenting or not fully disclosing accurate information.  Mathias Kjaer, Program Manager. Mr. Kjaer will be responsible for the contractual and administrative delivery of the contract but will also be expected to provide substantial support to Dr. Delaney to ensure optimal technical quality. He will participate in all phases of the evaluation and will be expected to substantially contribute to the final report.  Dr. Dennis Wood, Senior Technical Advisor. Based at SI HQ, Dr. Wood will be responsible for the overall quality assurance of the contract. Dr. Wood will participate in the desk review, advise on data collection and analysis techniques, and review and comment on draft evaluation reports.  Gabrielle Plotkin, Program Assistant. Based at SI HQ, Ms. Plotkin will support Mr. Kjaer in the contractual delivery of the contract. She will lead on administrative and logistical aspects of Final Evaluation of USAID’s Fostering Transparency Initiative in Timor-Leste 38 the contract, including, but not limited to, arranging travel, setting up interviews, reviewing invoices, and also providing copy-editing and technical comments on the draft reports. Design Trip SOW Final Evaluation Planning & Design Trip Timor-Leste Threshold Program for Anti-Corruption I. Purpose: Social Impact (SI) is currently providing the monitoring and evaluation (M&E) of the implementation of the six program components of the MCC/USAID Timor-Leste Threshold Program for Anti-Corruption. Midway through its three year project life, Social Impact completed a comprehensive Mid-Term Assessment. In August 2013, Senior M&E Technical Advisor Delaney completed an oversight visit to informally assess program progress subsequent to the mid-term evaluation. The SOW of work for this February 2014 visit builds upon these past TDYs to develop a plan for the final evaluation plan. II. Statement of Work: Senior M&E Technical Advisor Delaney will be responsible for carrying out a TDY to prepare for the final evaluation of the FOTI project (to be completed after the program has officially ended all technical work). Delaney will work directly with program staff (especially advisors and the M&E team), USAID counterparts, and multiple stakeholders in each component area. Delaney will be in country from February 23 to March 1. Specifically, during the follow-up TDY, Delaney will focus on the following key areas: Pre-Departure Preparation  Review quarterly, annual, and other reporting produced by FOTI during the life of the project special attention to the final year of implementation);  Review MCC evaluation templates and guidelines Data Collection & Rapid Analysis of Existing FOTI Data Sources  Examination, analysis, and collection of data sources including pre and post tests and advisor performance reports from FOTI databases and archival records  Interview COP concerning quarterly, annual, and other reporting produced by FOTI during the life of the project special attention to the final year of implementation);  Review of external data sources including database; OPG database; Judiciary database; and media archives.  Identify additional external data sources that might be utilized for the final evaluation (e.g. World Bank, Asia Foundation, or other donor-funded databases in the D/G sector) Interviews with Key Stakeholders  Preliminary Data Collection while FOTI staff is still available including detailed interviews with all international advisors concerning project accomplishments, shortcomings, and predicted impacts;  Interview with USAID partners concerning project accomplishments, shortcomings, and predicted impacts as well as recommendations for the methodological design of the final evaluation;  Follow-on Interviews with Key Stakeholders Final Evaluation Planning & Design  Complete discussions with USAID concerning timing, structure, and organization of final evaluation activities;  Discussions with USAID, FOTI staff on intended use of final evaluation findings 39  Identification of key areas of interest to USAID FOTI  Prepare “options memo” for final evaluation design. Post-TDY Activities  Complete discussions with MCC concerning timing, structure, and organization of final evaluation activities; Complete discussions with MCC concerning priorities for evaluation design III. RESULTS AND DELIVERABLES The Senior M&E Technical Advisor will be expected to produce: 1. Draft Final Evaluation Plan (featuring mixed methods approach and containing multiple options for consideration by MCC, USAID, and other stakeholders). a. Description of options will contain (at a minimum): i. Specific Methodology ii. Data Sources iii. Key Questions and Related data iv. Administrative Matters 1. LOE 2. Consultant expertise 3. Budget for Data Collection and Report Drafting 40 ANNEX II: DATA SOURCE MATRIX, EVALUATION METHODS, AND LIMITATIONS Component A: Anti-Corruption Commission Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the political context? -with respect to the creation of the CAC? -with respect to the relationship between CAC & OPG? 2. How appropriate were the original proposed activities given -the fact that CAC was a brand new organization? -the original staffing proposal? -the structural reality of CAC (investigators in￾house)? % of respondents agreeing that FOTI activities were appropriate to the Timor-Leste context # of key activities deemed “appropriate” by key stakeholders  Interviews with USAID, CAC, and other stakeholders  Project Logical Frameworks  FOTI Quarterly, Annual, and Final Report  41 Activities 1. How effective was the capacity-building in the investigation unit? 2. How effective was the capacity-building in the prevention unit? 3. How effective was the capacity-building in the research unit? 1. In a sample of ten percent of post-test competency assessments, percent with score of 60% or higher 2. # and % of respondents who report that training was useful to them or their staff 3. Technical score awarded by expert on quality of investigation manual 4. Technical score awarded by expert on quality of prevention manual 5. # and/or % of respondents who report that they or their staff use FOTI-supported manuals, surveys, curricula and tools 6. #of trainings provided by CAC to outside agencies 7. # of surveys or other research projects completed by CAC 8. Training subjects identified as “most helpful” (qualitative) 9. Reported quality of FOTI assistance (qualitative, e.g. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Pre and Post Test Data  Interview with Deputy Commissioner of CAC  Interviews with CAC staffers  Technical Assessments of Training Materials by Anti-Corruption Expert  Observations of CAC Activities  Behavior Change 1. Are the team members in the prevention unit using the new manuals? 2. Are team members in the prevention unit providing trainings for RDTL partners? 3. Are team members in the research unit conducting surveys? 4. Do team members in the investigation unit express confidence in their knowledge and abilities to conduct investigations? # of examples of manual use #of training provided to outside agencies # of surveys or other research projects completed % of investigators reporting they feel “confident” or “very confident” in their abilities (Likert 5 point scale)  Observations of CAC staff work performance  Interviews with CAC senior staff  Interviews with CAC investigators  FOTI Quarterly, Annual, and Final Reports 42 Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for a nascent Anti￾Corruption Commission? What key factors affect the transferability of these lessons? 2. What are the key activities at CAC that USAID or other donors should continue to support? What are the likely targets of opportunity for CAC’s prevention role as the RDTL moves towards pre-de￾concentration (de￾centralization)  Triangulation of all data sources  FOTI Quarterly, Annual, and Final Reports Component B: Office of the Prosecutor General Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the political context? -with respect to the relationship between CAC & OPG? 2. How realistic were the original proposed activities given -the original staffing proposal 3. Were the changes in program logic in this component appropriate given the changing political realities on the ground?  Interviews with USAID, OPG, and other stakeholders  Project Logical Frameworks  FOTI Quarterly, Annual, and Final Reports  RRTs 43 Activities 1. Why did the Project fail to implement most activities in this component? 2. How effective was the capacity-building provided to prosecutors in the final year of implementation? 1. In a sample of ten percent of post-test competency assessments, percent with score of 60% or higher 2. #/ or % of respondents who report that training was useful to them or their staff 3. Technical score awarded by expert on quality of training material [curricula, investigation manual(s), prevention manual(s), testing material]. 4. # of respondents who report that they or their staff use FOTI-supported manuals, surveys, curricula and tools 5. # and/or % of prosecutors met by team who report they feel “confident” or “very confident” in their abilities (Likert 5 point scale) 6. # reported person hours of capacity building. 7. # reported person hours of inter-institutional capacity building joint trainings 8. Training subjects identified as “most helpful” (qualitative) 9. Reported quality of FOTI assistance (qualitative, e.g. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Pre and Post Test Data  Interviews with Prosecutors  Technical Assessments of Training Materials by Expert Behavior Change 1. Are the prosecutors using the asset recovery regulation manuals developed by the project? # of examples of prosecutors using manuals and other training materials  Interviews with Prosecutors 44 Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for the Prosecutor General’s Office? What key factors affect the transferability of these lessons?  Triangulation of all data sources  FOTI Quarterly, Annual, and Final Report Component C: Office of the Inspector General (Internal Audit) Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the organizational reality of the Office of the Inspector General? 2. How realistic were the original proposed activities given -the original staffing proposal 3. How appropriate were the changes in program logic in this component?  Interviews with USAID, OIG, and other stakeholders  Project Logical Frameworks  FOTI Quarterly, Annual, and Final Report Activities 1. Why did the project substantially modify the activities in this component? 2. How effective were the capacity￾building activities completed with the auditors? 3. What is assessment of the technical content of the audit manual? 4. Did the continued funding of a long-term advisor lead to measurable results that are consistent with the project? 1. In a sample of ten percent of post-test competency assessments, percent with score of 60% or higher 2. # of respondents who report that training was useful to them or their staff 3. Technical score awarded by expert on quality of audit manual 4. % FOTI-sponsored products that are approved 5. Training subjects identified as “most helpful” (qualitative) 6. Reported quality of FOTI assistance (qualitative, e.g. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Pre and Post Test Data  Interviews with Auditors  Observations  Technical Assessments of Training Materials by Expert  Interviews with USAID & Other Donors 45 Behavior Change 1. Has the audit manual been formally adopted? Yes or No Indicator  Observations of OIG  Focus Group or Interviews with Auditors Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for the Inspector General’s Office? What key factors affect the transferability of these lessons?  Triangulation of all data sources  FOTI Quarterly, Annual, and Final Report Component D: Procurement Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the political context? 2. How appropriate were the original proposed activities given -the original staffing proposal 3. Were the changes in program logic in this component appropriate given the changing political realities on the ground? # of key issues/items/areas identified as “missed” by USG, Donors, Grants, RDTL, and implementing partners  Interviews and Focus Group Discussions with USAID, OPG, and other stakeholders  Project Logical Frameworks  RRTs 46 Activities 1. Why did the Project fail to implement most activities in this component? 1. In a sample of ten percent of post-test competency assessments, percent with score of 60% or higher 2. # of respondents who report that training was useful to them or their staff 3. Technical score awarded by expert on quality of CAC procurement manual 4. # of respondents who report that they or their staff use FOTI-supported manuals, surveys, curricula and tools 5. # of respondents who report using FOTI￾supported manuals, surveys, curricula and tools 6. Reported quality of FOTI assistance (qualitative, i.e. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Key Informant Interviews with USAID and Other Donors  FOTI Quarterly, Annual, and Final Report  Triangulation of all data sources Behavior Change N/A Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for the procurement function? What key factors affect the transferability of these lessons?  Triangulation of all data sources  FOTI Quarterly, Annual, and Final Report 47 Component E: Parliament Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the political context? 2. How appropriate were the original proposed activities given -the political nature of Parliamentary discussions -Parliamentary elections in the middle of the project -the lack of professional staff supporting MPs  Interviews with USAID, MPs, and other stakeholders  Project Logical Frameworks  FOTI Quarterly, Annual, and Final Reports  RRTs Activities 1. What is the legal efficacy of the major anti-corruption legislation passed with program support? 2. How effective were the capacity￾building activities completed with Members of Parliament? 1. # of respondents who report that they or their staff use FOTI-supported manuals, surveys, curricula and tools 2. Technical Score Awarded by Legal Expert on the quality of technical analysis of legislation and other legal products, and the quality of legislation and regulations drafted and/or enacted. 3. # reported person hours of capacity building 4. #unaddressed gaps in legal framework 5. Training subjects identified as “most helpful” (qualitative) 6. Reported quality of FOTI assistance(qualitative, i.e. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Technical Assessment of Laws by Evaluation Team  Interviews with MPs, Parliamentary Staff  FOTI reporting documents 48 Behavior Change 1. Do MPs demonstrate a greater interest in anti-corruption issues? 2. Are MPs more knowledgeable about key anti-corruption issues? # of examples of MPs using information from the training in their work (e.g. presentations) # of examples of MPs discussing anti-corruption knowledgably in the media and public settings  Interviews with MPs, Parliamentary Staff  FOTI reporting documents Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for the Parliament? What key factors affect the transferability of these lessons?  Triangulation of all data sources Component F: Watchdog & Advocacy (Civil Society) Questions Indicators Data Sources Program Logic 1. Did the original program logic effectively assess the context? -with respect to nascent state of independent journalism in Timor￾Leste? -with respect to the role of civil society in Timor-Leste? -with respect to the level of general societal understanding of corruption in Timor-Leste? 2. How appropriate were the original proposed activities given -the original staffing proposal -the original budget 3. Were the changes in program logic in this component appropriate given the changing political realities on the ground?  Interviews with USAID, OIG, and other stakeholders  Project Logical Frameworks  FOTI Quarterly, Annual, and Final Report 49 Activities 1. How effective were the activities conducted with journalists? 2. How effective was the “civic education” campaign conducted with CEPAD? (music CD, radio shows, TV shows & children’s book) 3. How effective was the collaboration with the watchdog NGO, Lao Hamutuk? 1. In a sample of ten percent of post-test competency assessments, percent with score of 60% or higher 2. # of respondents who report that training was useful to them or their staff 3. # of respondents who report that they or their staff use FOTI-supported manuals, surveys, curricula and tools 4. # of persons reached by civic education campaign 5. # of story books printed 6. # of “anticipated readers” (multiplier effect) 7. # of radio and TV broadcasts of FOTI￾supported programs 8. Training subjects identified as “most helpful” (qualitative) 9. Reported quality of FOTI assistance (qualitative, i.e. relevance of material, appropriateness of pedagogical method, quality of mentorship)  Focus groups with journalists  Post-Test Data  Focus Group discussions with general public (about music, radio, books) Behavior Change 1. What is the likely impact of the distribution of children’s books through the Ministry of Education? 2. What are the reported early indicators of changes in knowledge or attitudes about corruption that can be attributed to the civic education campaign? # of books distributed # of “anticipated readers” (multiplier effect) # of examples of changes in knowledge or attitudes  Interviews with Ministry of Education senior staff  Interviews with staff and beneficiaries at CEPAD Peace Houses (in districts)  Observations in schools and Peace Houses 50 METHODOLOGY Document Review: All relevant FOTI, USAID, MCC, and RDTL reports including monthly, quarterly, and final reports, annual work plans, Results Reporting Tables (RRTs), and background documents were reviewed. (See Annex III for a complete list of documents.) The team used their findings from the desk review to finalize their data collection instruments, interviewee selection, and as a data source for the final report. Technical Evaluation of “Deliverables”: Relevant FOTI products including technical content of legal documents (e.g. AC legislation), training curricula and manuals, and implementation guidance (e.g. CAC prevention training modules), and the materials in the civic education campaign (e.g. music CDs and a children’s book about corruption). The content of these deliverables was analyzed using the evaluation’s indicators, listed in Annex II and Annex IV for a complete list of deliverables analyzed.) Pre- and Post-Test Data: A random sample of monitoring data from 17 trainings was analyzed to assess both the strength of the technical questions in the tests and also the level of trainee learning as captured by the post-tests. Key Informant Interviews: The Team Leader interviewed 14 key informants interviewed during the preliminary fieldtrip in February 2014. A total of 44 informants were interviewed by the full field team during the field visit in July 2014. Four additional stakeholders were interviewed via phone after the fieldwork was completed (see Annex IV for a complete list of informants and Annex III for interview protocols.) Interviews were completed in Portuguese, English, and Tetun and a translator was used in most interview settings Field Visit to Baucau: While the majority of the fieldwork took place in Dili, the July fieldwork included a brief field visit to Baucau. This visit included a visit to one of the Centre of Studies for Peace and Development’s (CEPAD) peace houses to interview three staff members, a visit to a school and interview with a teacher, and an interview with a district coordinator from the Ba Distrito program. This visit provided the team with an opportunity to ground truth their preliminary findings, particularly about the potential impact of FOTI’s media campaigns and to minimize the “capital city bias” that is so common in this kind of evaluative work. Sustainability & Lessons Learned 1. What lessons does this experience provide about the support for civil society in anti-corruption work? What key factors affect the transferability of these lessons? 2. What are the key activities among civil society that USAID or other donors should continue to support? What are the likely targets of opportunity for civil society engagement as the RDTL moves towards pre-de-concentration (de￾centralization)?  Triangulation of all data sources  FOTI Quarterly, Annual, and Final Reports 51 LIMITATIONS Challenge in measuring qualitative outcomes: The intended outcomes of the FOTI project were mostly measurable in qualitative ways, including enhanced capacity of nascent RDTL institutions and the strength of AC legislation drafted with FOTI support. The evaluation team addressed the potential bias inherent in such qualitative data through extensive triangulation of data sources. Logistical & timing challenges: The majority of the data collection took place three months after the end of the FOTI project. This reduced access to key stakeholders (e.g. the original Commissioner of the CAC). Given the hierarchical bureaucratic context in Timor-Leste, in which following protocol is paramount, it proved difficult to interview mid-level staff (the direct beneficiaries) without first interviewing higher-level contacts and other gate-keepers. Recall and attribution bias: The fact that interviews took place three months after the end of the program increased the risk of recall bias—inaccurate or incomplete recollection of events—which was observed in some of the interviews. In the case of the FOTI program, there are various donor programs working with the same institutions and in some cases, the advisors had been subsidized by a previous USAID-funded program or other donors. The team attempted to mitigate this limitation by using techniques during interviews (e.g. listing names of advisors and types of activities, etc.) to facilitate interviewee recollection of their experiences with certain activities. Selection bias: Due to the fact that the program had ended, the team relied on the target institutions to provide access to direct program beneficiaries. In many cases, the team was not granted interviews with as many direct beneficiaries as desired. This could be perhaps because the beneficiaries had other priorities, although in certain cases, many of the beneficiaries were unavailable due to travel, which was the case for the CAC and members of the NP. Use of interpreter: Although the team had some Portuguese and Tetun language proficiency, most interviews were conducted through the use of an interpreter, which increased the risk of misunderstanding. To mitigate this risk, the team debriefed with the interpreter after each meeting to ensure that the team grasped the main ideas and sentiments of the conversations, though some details may have been lost in the process. 52 ANNEX III: DATA COLLECTION INSTRUMENTS Protocol for informants in management positions within beneficiary organizations # Evaluation question # Topic Protocol question Seeks to gather information regarding: 1 1A/B Program logic: relevance/logic of design How well did the project take into account the realities in Timor-Leste: 1) In terms of the political environment? 2) In terms of capacity? 3) Other factors? Assumptions, risks, external factors 2 1B Program logic: appropriateness of changes to design How appropriate were the changes in the program design for better addressing the needs of your organization? Stakeholder reaction to changes in design 3 2A/B Activities: occurrence of planned/new activities What forms of technical assistance did your organization receive? 1) Coaching/mentorship 2) Training 3) Other Incidence of planned activities 4 2A Activities: occurrence of planned activities Were there planned activities that were not completed? If so, why were they not completed? Incidence of planned activities 5 2C Activities: effectiveness Which of these activities were most helpful to your organization and why? Least helpful? Why? Institutional-level development achieved; what about specific activities worked well or not 6 2C Activities: effectiveness Can you provide specific examples about how has your office changed as a result of program activities? Institutional changes achieved 7 3A Outcomes: behavior change Are the skills from the trainings/mentoring being applied now that the program is over? Describe how some of the skills are being applied. Why (or why not) are they being applied? Contributions to behavior change 8 3A Outcomes: other What can your institution do that it wasn’t able to do before the FOTI program? Contributions to outcomes 9 4A Sustainability: activities Are the training curricula/manuals/tools/inter-agency protocols being used now that the program is over? How are they being used? Why (or why not) are they being used? Incidence of activities post￾program 10 4C Sustainability: replication What aspects of the FOTI program should (or should not) be replicated? Why? Recommendations for Ba Distrito 53 Another version: what are the aspects of FOTI that your organization would like to seek future funding for? OR How have the accomplishments supported by FOTI prepared for addressing corruption in a decentralized context? Decentralization Protocol for direct beneficiaries (non-management positions) # Evaluation question # Topic Protocol question Seeks to gather information regarding: 1 2A/B Activities: occurrence of planned/new activities What forms of technical assistance did your organization receive? 4) Coaching/mentorship 5) Training 6) Other Did you participate in these activities? If so, please describe them. Incidence of planned activities 2 2A Activities: effectiveness Which of these activities/products were most helpful to you and why? Least helpful? Why? Individual-level development achieved; what about specific activities worked well or not 3 2C Activities: effectiveness Which of these activities were most helpful to your organization and why? Least helpful? Why? Institutional-level development achieved; what about specific activities worked well or not 4 3A Outcomes: behavior change How has participating in these activities influenced you and your colleagues’ ability to perform your job? Describe how some of the skills acquired through the FOTI program are being applied. Changes in knowledge 5 3A Outcomes: behavior change How have the FOTI activities influenced your attitude/commitment to performing your job? Change in attitudes 6 4A Sustainability: activities Are the training curricula/manuals/tools/inter-agency protocols being used now that the program is over? How are they being used? Why (or why not) are they being used? Incidence of activities post￾program 7 4C Sustainability: replication What aspects of the FOTI program should (or should not) be replicated? Recommendations for Ba Distrito 54 Why? Another version: what are the aspects of FOTI that your organization would like to seek future funding for? OR How have the accomplishments supported by FOTI prepared for addressing corruption in a decentralized context? Decentralization Protocol for other stakeholders (e.g. other donors, other programs, etc.) # Evaluation question # Topic Protocol question Seeks to gather information regarding: 4 1A/B Program logic: relevance/logic of design Did the project match the political realities in Timor-Leste? How were your organization’s needs taken into account in the design of the project? Would you have done anything differently? Assumptions, risks, external factors 7 2A Activities: effectiveness With regard to individual development, which of these activities were most helpful to your organization and why? Least helpful? Why? Individual-level development achieved; what about specific activities worked well or not 8 2C Activities: effectiveness With regard to institutional development, which of these activities were most helpful to your organization and why? Least helpful? Why? Institutional-level development achieved; what about specific activities worked well or not 20 4B Sustainability: likelihood What are the barriers and enablers to sustaining the results of this program (for your institution)? Conditions for sustaining effects 21 4C Sustainability: replication What aspects of the FOTI program should (or should not) be replicated? Why? Another version: what are the aspects of FOTI that your organization would like to seek future funding for? OR How have the accomplishments supported by FOTI prepared for addressing corruption in a decentralized context? Recommendations for Ba Distrito Decentralization 55 ANNEX IV: SOURCES OF INFORMATION People Interviewed USAID/Timor-Leste James Coy, Contracting Officer’s Representative, FOTI Program Casals & Associates Eduardo Flores-Trejo, Chief of Party, FOTI Program Jamie Greenawalt, Procurement Advisor Procuradora Geral da República (Office of the Prosecutor General) José da Costa Ximenes, Prosecutor General Angelina Joanina Saldanha, District Prosecutor Remizia De Faria da Silva, District Prosecutor Matias Soares, District Prosecutor La’o Hamutuk (Timor-Leste Institute for Development Monitoring and Analysis) Charles Scheiner Centre of Studies for Peace and Development João Boavida, Executive Director Peace House staff (3) Counterpart International Carolyn Tanner, Chief of Party, Ba Distrito Program Nicolau Freitas De Sa, Baucau District Coordinator, Ba Distrito Program Anti-Corruption Commission, Timor-Leste Jose Antonio de Jesus das Neves, Acting Commissioner Jonas Floriano Guterres, National Adviser to the Office of the Commissioner Rosario Salsinha de Araujo, Director, Education and Research Ezequiel Cristovão da Silva, Research Officer, Education and Research Cirilio Haos Loca Abi, Research Officer, Education and Research Aniceto C.X. Filipe, Prevention Officer, Prevention Unit Feliciano Ximenes, Prevention Officer, Prevention Unit Maria da Graca de Vasconcelhos, Director, Investigation Unit Rosito Amaral, Investigator, Investigation Unit Office of the Inspector General, Timor-Leste Francisco de Carvalho, Inspector General Billy Tarillo, Senior Audit Advisor Maria Celeste Pimental Elsa Freitas Embassy of the United States, Timor-Leste Glenn Alexander, Resident Legal Advisor Parlamento Nacional de Timor-Leste (National Parliament of Timor-Leste) Vicente da Silva Guterres, President of the National Parliament Cristina Ferreira, Parliamentary Advisor Anildo da Cruz, Legal Advisor to the Parliamentary Committees Pedro Andrade, Legal Advisor to the Parliamentary Committees Sectretário de Estado para o Fortalecimento Institutional (Secretariat of State for Institutional Strengthening) Francisco Borlaku Soares, Secretary of State for Institutional Strengthening Henriqueta da Silva, Senior Management Advisor Jaquelina Guterres Correia, Advisor on Support to Institutional Development Global Organization of Parliamentarians Against Corruption, Timor-Leste Chapter Fransisco Miranda Branco, President of GOPAC and member of the Legislative Committee on Public 56 Finance Maria Ricardina, GOPAC Coordinator United Nations Development Program, Timor-Leste Farhan Sabih, Assistant Country Director, Head of Governance Unit Ministry of Education, Timor-Leste Dulce De Jesus Soares, Vice Minister for Preschool and Basic Education Court of Appeals, Timor-Leste Guilherme Maria da Silva, President of the Court of Appeals Judicial Training Center, Ministry of Justice, Timor-Leste Marcelina Tilman da Silva, Director Carlos Contas, Coordinator Media Journalists (3) Civil Society Organizations Staff of various organizations (3) Documents Reviewed I. Ba Distrito Documents o Ba Distrito Program Description o USAID-Timor-Leste Country Development Cooperation Strategy 2013-2018 II. Program Documents CAC o CAC Mini-Survey of Public, 2014 o Corruption Prevention Courses July-September 2013 o Corruption Prevention Courses October-September 2013 o Corruption Prevention Courses January-March 2014 o Corruption Perception Survey, 2011 o Frequency Snap Survey of Citizens Dili o Integrity Survey of Public Servants in Timor-Leste o Workshop 4 Pre-test and Post-test o Workshop 5 Pre-test and Post-test o Workshop 6 Pre-test and Post-test o Workshop 7 Pre-test and Post-test o Workshop 8 Pre-test and Post-test o Workshop 9 Pre-test and Post-test o Workshop 10 Pre-test and Post-test Casals Contracts o Mod 1 o Mod 2 o Mod 3 o Mod 4 o Task Order Civil Society o Baino and Binoi children’s storybook o Basic Course Pre-test and Post-test o CEPAD Anti-Corruption Manual o How long can the Petroleum Fund Carry Timor-Leste? 2013 57 o Intermediate Course Pre-test and Post-test o Presentation from La'o Hamutuk 2013 Journalists o Investigative Journalism Training Course 2012 o Media Training Pre-test and Post-test o Strengthen Investigative Reporting Skills to Promote Accountability and Transparency 2013 o Training and Coaching Activities on Investigative Journalism 2013 Monthly Reports o February 2011 o March 2011 o April 2011 o May 2011 o June 2011 o July 2011 o August 2011 o September 2011 o April 2012 OIG o Audit Training Needs Assessment Report o Internal Audit Charter o Internal Audit Handbook OPG o Financial and Corruption Investigations Pre-Test and Post-Test, February 2013 o Financial and Corruption Investigations Pre-Test and Post-Test, February 2014 o Asset Recovery Regulations and Mechanisms Pre-Test and Post-Test, February 2014 Procurement o Procurement Training Needs Assessment o Procurement Training Manual, 2013 Parliament o FOTI inputs to Law 18/11 on Money Laundering and Financial Terrorism o FOTI Manual on Parliament’s Financial Powers, 2012 Quarterly Reports o January-March 2011 o April-June 2011 o July-September 2011 o October-December 2011 o July-September 2012 o January-March 2013 o April- June 2013 o July-September 2013 o October-December 2013 o October-December 2014 o USAID-MCC Threshold Program for Anticorruption in Timor-Leste Final Report, April 2014 58 Work Plans o Year 1 Work Plan o Year 2 Work Plan o Year 3 Work Plan III. Anti-Corruption Background Documents o Best Practice Guides to Government Procurement o Corruption Perception Survey, 2011 o Democratic Republic of Timor-Leste Decree Law 14/2011 o Integrity Survey of Public Servants in Timor-Leste, 2012 o Justice Institutions Strengthening Project Fact Sheet o Justice Institutions Strengthening Project Final Report, March 2012 o Journal of the Republic Law No. 9/2011 o Key Institutions and Actors o Law on the Anti-Corruption Commission. Law No.8/ 2009 o Lessons Learned Fighting Corruption in MCC Threshold Countries: The USAID Experience, November 2009 o MCC-Timor-Leste Bilateral Agreement, September 2010. o MSI Corruption Assessment: Timor-Leste, September 2009. o Rule of Law in Timor-Leste, June 2007. o Strengthening Accountability and Transparency in Timor-Leste, January 2006. IV. FOTI Success Stories o Civic Education Campaign Motivates Partnership with Government to Engender Accountability o Curbing Corruption in Public Sector Procurement o Motivating a Proactive and More Effective Stance to Prevent Corruption o Nurturing the Capacity of Timor-Leste’s Anti-Corruption Commission to Examining and Addressing Causes and Consequences of Corruption o Strengthening Anti-Money Laundering Tools to Curb Corruption V. Other o FOTI program Initial Assessment, June 2011 o List of Manuals, Legislations and Products developed by FOTI in Investigations/Legislation o Timor-Leste FOTI Threshold Program Midterm Assessment Report, May 2013 o Mid-Term Assessment Follow Up Report: Timor-Leste Threshold Program for Anti-Corruption, August 2013 o Original Request for Proposal 59 ANNEX V: FOTI RESULTS FRAMEWORK Objective: To reduce corruption through an integrated approach that will engage build the capacity of a number of key actors within RDTL to constitute a coordinated and comprehensive system to prevent, detect, and punish the misuse of government funds. IR 1: Support the ACC Establish the ACC as the lead agency for anticorruption education, prevention, and investigation by ensuring that the ACC is able to successfully complete investigations of cases of alleged corruption and, if warranted, refer its findings to the (OPG). Activities  Assist and train the ACC and its staff in carrying out corruption prevention, education, and investigation activities, as well as the administration and management of cases.  Coordinate and collaborate with relevant threshold program actors, including the OPG and the audit, procurement, and civil society-related activities. Activities may include joint training seminars, an exchange of information, collaboration on corruption cases, and other interventions to promote a unified focus on reducing corruption IR 2: Support the OPG Build OPG’s investigative and prosecutorial capacity, particularly for the investigation and prosecution of corruption cases and economic crimes. Activities  Train and otherwise assist the OPG in developing a capacity to investigate and prosecute corruption cases, including training on key investigative concepts and prosecutorial techniques.  Coordinate and collaborate with relevant Program actors, including the ACC and the audit, procurement, and civil society groups. Activities may include joint training seminars, an exchange of information, collaboration on corruption cases, and other interventions to promote a single focus on reducing corruption.  Support the Government adoption of a working protocol or memorandum of understanding specifying investigative levels of authority and responsibilities with respect to economic crimes among the OPG, ACC, the National Police, the Banking Payment Authority IR 3: Support the Establishment of an External Audit Authority Strengthen the auditing capabilities of internal government audit entities and establish a functioning external audit authority to improve the Government’s ability to prevent and detect instances of corruption and, where necessary, assist OPG with criminal indictments. Activities  Facilitate independent audits of at least three large ministries and publicize the results.  Establish an independent external audit authority and provide institution-building support.  Train external auditors and inspectors general on modern audit techniques and assist in developing institutional capacity to carry out a large volume of audits.  Coordinate and collaborate with relevant Program actors. Activities may include sharing information on completed audits, assisting with investigations and prosecutions of economic and corruption￾related crimes, participating in joint training sessions, disseminating results to nongovernmental actors, including the media, and other interventions that increase synergies on fighting corruption  Support the Government in defining roles and levels of responsibility with respect to internal audit authority between the OIG and relevant inspector general offices within line ministries. IR 4: Improving Procurement Policies, Regulations and Practices Increase the knowledge and use of internationally-accepted procurement standards in relevant line ministries by having Government contracting authorities train procurement officers and establish an in- 60 service training system for contract officers on international and Government legal procurement standards. Support the establishment and operations of an independent procurement review commission empowered to adjudicate procurement complaints in accordance with its legal mandate. Activities  Coordinate and collaborate with relevant Program actors, including the ACC and the OPG, as well as across the audit, civil society and parliament related activities.  Provide training on internationally accepted procurement standards to staff with procurement responsibilities within the Ministry of Finance and other relevant ministries.  Develop a train-the-trainer program to increase the sustainability of efforts under this sub￾component.  Support the establishment of an independent procurement review commission and provide institutional support for this commission. IR 5: Enhancing Fiscal Oversight Skills of National Parliament Strengthen parliament’s role as a check on corruption and collusion through increased oversight of the government’s fiscal management practices, including using its legislative oversight authority to probe alleged corruption or misuse of funds, or follow up on recommendations of independent agencies. Activities  Train and assist parliament and staff on exercise of fiscal oversight authorities.  Provide technical expertise to draft new anti-corruption legislation, working through legislative committees and with advocacy groups.  Coordinate parliamentary oversight activities with other threshold program activities, including audit, procurement, and civil society activities, through joint trainings, information sharing and collaboration on legal reform initiatives. IR 6: Enhancing Watchdog Capacity and Public Advocacy of Non-State Actors  Increase the capacity and public advocacy skills of civil society organizations, media, academia, and private sector associations in order to improve their effectiveness in monitoring, advocating, and disseminating information on corruption-related issues, including through anti-corruption themed public advocacy campaigns and watchdog activities.  Support targeted and sustained campaigns and coalition building activities that will advocate for reformed policies or practices supported by the threshold program.  Provide training and support to selected organizations to carry out three corruption surveys and disseminate findings to the public.  Provide training and technical assistance to non-state actors including civil society organizations, the media, academia, the private sector, and others.  Training may include topics such as fact-based targeted advocacy campaigns, investigative journalism, public complaints handling, government budget procedures, public service report cards, data gathering, and other relevant topics.  Coordinate with all actors in the Program to advocate for reforms, disseminate information and results, and provide other interventions to support the Program’s coordinated approach to fighting corruption. 61 ANNEX VI: FOTI ORIGINAL COMPONENT ACTIVITIES Component A:  “…assist the ACC [USAID’s previous acronym for the Anti-Corruption Commission] and train its staff to carry out corruption prevention, education, and investigation activities, as well as manage the administration and management of cases. Activities would likely include specialized training in financial management, financial analysis, investigation skills, money laundering, developing education campaigns, as well as other training areas yet-to-be identified. To maximize resources and participation, training will be held in Timor Leste where possible. Attendance in offshore training such as those offered by the Malaysian Anti-corruption Academy may be considered on a case-to-case basis;” and  “…selected ACC staff may be sent to successful, more established corruption commissions to observe the actual investigation of corruption cases. The anti-corruption commissions of Indonesia and Macau are two institutions that can potentially accommodate interns from the ACC.” Component B:  “… help the OPG develop a capacity to investigate and prosecute corruption cases. This will likely include training on key investigative concepts and prosecutorial techniques that relate to corruption-related crimes, including a specific focus on evidence gathering that is in line with Timor Leste’s existing capabilities. With the goal of program integration, these trainings should be offered to all relevant stakeholders, where appropriate, including staff from the OPG, the ACC, the Bank Payments Authority, and the National Police, among other entities. Additional representatives from the Parliamentary sub-committee on anti-corruption, independent media and relevant civil society organizations should also be included, where relevant;”  “…offer training in more sophisticated investigative and prosecutorial techniques (such as trial advocacy, monitoring financial flows, electronic surveillance, and others) as the program evolves. These trainings will be offered to all relevant stakeholders, including participants in the other TPA components;”  “…support the RDTL’s adoption of a working protocol or memorandum of understanding specifying investigative levels of authority and responsibilities with respect to economic crimes among the OPG, ACC, the National Police, the Banking Payment Authority, the Office of the Inspector General, and other relevant stakeholders;” and  “…upgrade the technology within the OPG as it relates to electronic file management of economic crime cases.” Component C:  “…support the RDTL in establishing an independent external audit authority and provide institution-building support, and assist the RDTL in conducting independent audits of at least three large ministries and publicize the findings;”  “…train both external auditors and inspectors general on modern audit techniques and assist in developing internal institutional capacity within relevant government agencies to carry out large volume of audits;” and  “…support the RDTL in defining roles and levels of responsibility with respect to internal audit authority between the Office of the Inspector General and relevant inspector general offices 62 within line ministries68.” Component D:  “…develop and deliver a competency-based certification course on procurement for government procurement officers. If feasible, the Contractor shall partner with local educational/training institutions to facilitate the development of local capacity to deliver the certification course in the future, in order to create a sustainable procurement training system in Timor Leste. This course must also be responsive to existing procurement skills. Prior to administering this course to an estimated 300 MOF and line ministry personnel, skill levels must be evaluated and relevant findings incorporated into the final course product;” and  “…assist with the start-up of the Procurement Review Commission and provide training and technical assistance to the Commission in its first years of operation, with an eye toward developing a local capacity to oversee procurements, and resolve procurement disputes. The Contractor will assist the Commission in its outreach to business community69.” Component E:  “…train and assist the relevant parliamentary committees and staff on the exercise of fiscal oversight authorities. Specific training will be responsive to local needs and existing skills and include budget and expenditure issues, including oversight of the Timor-Leste Petroleum Fund. As the program and participant’s understanding evolves, the Contractor shall provide more sophisticated fiscal and budget training, with an eye toward sustainability;” and  “…provide technical expertise to drafting of new anti-corruption legislation working with legislative committees and advocacy groups.” Component F:  “…shall train and build capacity of (non-state) actors in fact-based, targeted and sustained advocacy campaigns and watchdog activities, investigative journalism, public complaints handling, government budgetary procedures, public service report cards and others as may be identified during the course of implementing the Threshold Program. Technical approaches should emphasize specialized training for civic/media groups in monitoring the RDTL’s use of the Petroleum Fund receipts and withdrawals;” and  “…support coalition building activities to strengthen the ability of non-state actors to coalesce around corruption issues of common interest and engage in public outreach and education activities.” 68 Task Order AID-486-TO-11-00001, 2011 69 Task Order AID-486-TO-11-00001, 2011 63 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523