I MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report FINAL EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF USAID CEGAH Monitoring Evaluation and Learning Support to USAID/Indonesia Submitted: October 13th, 2020 Above: Participants of “Gerakan Sadar LAPOR!” hold banners as they march on Jakarta’s thoroughfare Jl. Thamrin on Sunday, July 15, 2018. Photo Credit: Moses Sihombing/USAID CEGAH (Photo courtesy of USAID CEGAH) II MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report FINAL EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF USAID CEGAH Monitoring, Evaluation, and Learning (MEL) Support to USAID/Indonesia Mechanism Number: AID–486–I–14-00001 (IDIQ); Task Order 72049719F00001 Prepared for: Elizabeth Mendenhall (COR) United States Agency for International Development/Indonesia (USAID) American Embassy Jl. Medan Merdeka Sel. No.3-5, RT.11/RW.2 Gambir, Kecamatan Gambir, Kota Jakarta Pusat, Daerah Khusus Ibukota Jakarta 10110 Indonesia Prepared by: Social Impact, Inc. Contact: Valentine J Gandhi, Chief of Party 2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Adam Reisman, Ruth Suciadi, Alberta Christina and Daniel Sahanggamu on behalf of Social Impact, Inc. III MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ABSTRACT USAID commissioned a final Performance Evaluation of the USAID/Indonesia CEGAH (“Prevent”) Activity (2016-2021) implemented by Management Systems International through the USAID/Indonesia MEL Support Program led by Social Impact, Inc. CEGAH has administered 52 tasks designed to reduce corruption by bolstering the abilities of its justice sector, governing institutions and civil society to improve Indonesia’s “Community of Accountability.” The Evaluation Team (ET) conducted this evaluation remotely due to coronavirus restrictions, collecting data from May through July 2020 through a document review, 77 video-conference-based key informant interviews (KIIs) with 123 stakeholders, an online-administered mini survey sent to 126 beneficiaries, and “virtual” site visits in North Sumatra and East Java. A key area of inquiry for the ET was assessing which of CEGAH’s interventions were the most and least successful, and which external factors were affecting this process. The ET found that CEGAH’s most substantive contributions were through accessible oversight tools facilitating changes in policies and practices, such as the Supreme Court’s Direktori Putusan (Case Directory), being used for research to reduce sentencing disparities in corruption cases nationwide; and the SP4N LAPOR! national complaint handling system, which is changing how local governments deliver public services; conversely, CEGAH-supported efforts to address links between corruption and violent extremism resulted in research demonstrating limited connections, and civil service recruitment testing tools that raised concerns over potential misuse and human rights implications in their application by some agencies. External factors influencing these and other interventions included a new law restricting the enforcement capabilities of Indonesia’s primary anti￾corruption agency, key logistical, programming and budgetary restrictions caused by COVID-19, the varying quality of Internet coverage in remote areas, and key staffing rotations and departures within supported governing institutions, particularly at the local level. IV MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report TABLE OF CONTENTS Abstract................................................................................................................................III Table of Contents................................................................................................................IV List of Figures and Tables....................................................................................................V Acronyms List......................................................................................................................VI Executive Summary..............................................................................................................1 Evaluation Purpose ...............................................................................................................7 The Development Problem .................................................................................................7 USAID’s Response.................................................................................................................8 Summary Information ...................................................................................................................................... 8 Evaluation Questions ..........................................................................................................10 Evaluation Design and Methodology .................................................................................10 Data Collection...............................................................................................................................................10 Sampling ............................................................................................................................................................11 Ethical Considerations...................................................................................................................................15 Data Analysis....................................................................................................................................................16 Findings and Conclusions....................................................................................................18 EQ1: In what ways did the design and structure of CEGAH help or hinder the achievement of its results?.........................................................................................................................................18 EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process?..............23 EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program?...........................................................................................................38 Lessons Learned..................................................................................................................42 EQ4: What lessons learned in strategy and approach can and should be applied to future programming? .......................................................................................................................................42 Recommendations ..............................................................................................................47 Annexes................................................................................................................................51 Annex 1: Gantt Chart of Evaluation Activities........................................................................................51 Annex 2: Evaluation Team............................................................................................................................52 Annex 3: Bibliography....................................................................................................................................55 Annex 4: Data Collection Protocols..........................................................................................................56 Annex 5: Scope of Work............................................................................................................................104 Annex 6: List of CEGAH Required Tasks...............................................................................................115 Annex 7: Comments from USAID CEGAH on the Evaluations report and the responses from the evaluation team.................................................................................................................119 V MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report LIST OF FIGURES AND TABLES Figure 1. CEGAH’s support has strengthened my organization's ability to fight corruption more than support I received from other donors. ......................................................................................................................21 Figure 2. Marginalized/Vulnerable Groups were able to access/benefit from our CEGAH-supported activity to the same degree that men were...............................................................................................................22 Figure 3. To what extent have any external factors impacted the success of your CEGAH-supported work?..................................................................................................................................................................................36 Figure 4. Do you believe your activity will be able to continue once CEGAH’s support ends?..................38 Table 1. CEGAH Evaluation – Virtual Sub-National Site Visits & Activities......................................................12 Table 2. The Distribution of KIIs.................................................................................................................................14 Table 3. Ensuring Quality in Data Analysis................................................................................................................17 Table 4. ET Roles and Responsibilities.......................................................................................................................52 Table 5. Support Staff Roles and Responsibilities....................................................................................................54 VI MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ACRONYMS LIST ABMS Anti-Bribery Management System AGO Attorney General’s Office AJI Alliance of Independent Journalists AMELP Activity Monitoring, Evaluation, and Learning Plan APBD Regional Budget APBN National Budget ASN Civil Service B_Trust Bandung Advisory Trust BADIKLAT MA Supreme Court’s Judicial Training Center BADILUM Directorate General for General Courts Bansos Social Funds BAPPENAS National Development Planning Agency BAWAS MA Supervisory Body of the Supreme Court BIN State Intelligence Agency BKD Regional Officer Agencies BKN National Civil Service Agency BNPT National Agency for Combating Terrorism BPK Indonesian Audit Board BPKP Development and Finance Oversight Agency BSN National Standardization Body C4J Changes for Justice CDS Center for Detention Study CDCS Country Development Cooperation Strategy COB Close of Business COR Contracting Officer’s Representative COVID-19 Corona Virus Disease 2019 CPE Corruption Prevention and Enforcement CSO Civil Society Organization CVE Countering Violent Extremism DEC Development Experience Clearinghouse DFAT Department of Foreign Affairs Direktori Putusan Case Directory Diskominfo Communication and Information Office DRG Democratic Resilience and Governance FGD Focus Group Discussion E2J Educating and Equipping tomorrow’s Justice reformers ELSAKA Policy Advocacy and Study Agency ET Evaluation Team EQ Evaluation Question EQUI® Evaluation Quality, Use, and Impact FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FITRA Indonesia Forum for Budget Transparency GESI Gender Equality and Social Inclusion GOI Government of Indonesia HARMONI Towards Inclusion and Resilience Activity HQ Headquarters IBL Indonesia Business Link VII MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report IDEA Institute for Development and Economic Analysis ICW Indonesia Corruption Watch IG Inspectorate General INP Indonesian National Police IP Implementing Partner IR Intermediate Result IT Information Technology IRB Institutional Review Board J2SR Journey to Self-Reliance KAD Regional Advocacy Committee KASN Civil Service Commission Kemenag Ministry of Religious Affairs KemenATR/BPN Ministry of Land and Spatial Planning/National Land Agency Kemendagri Ministry of Home Affairs Kemendikbud Ministry of Education and Culture Kemenkes Ministry of Health Kemenkominfo Ministry of Communication and Information KemenkumHAM Ministry of Law and Human Rights Kemenaker Ministry of Manpower Kemenkopolhukam Coordinating Ministry of Politics, Law, and Human Rights KemenPAN-RB Ministry of Bureaucratic and Administrative Reform KII Key Informant Interview KIP Central Information Commission KPK Corruption Eradication Commission KPU General Election Commission KSP Executive Office of the President Lakpesdam-PBNU Institution of Human Resource Study and Development General Committee of Nahdlatul Ulama M&E Monitoring and Evaluation LAN State Administrative Body LeIP Institute for Study and Advocacy for Judicial Independence Lemhanas National Defense Agency LGTC LAPOR! Goes to Campus LKPP National Public Procurement Agency LMS Learning Management System MA Supreme Court MA Kepaniteraan Registrar of Supreme Court MaPPI FHUI The Faculty of Law, University of Indonesia / Indonesia Judicial Monitoring Society MCW Malang Corruption Watch MEL Monitoring, Evaluation, and Learning MoU Memorandum of Understanding MSI Management Systems International NGO Non-Governmental Organization OJK Financial Service Authority OPD Regional Apparatus Organization / Local Government Units ORI Indonesian Ombudsman PATTIRO Center for Regional Information Studies PDF Portable Document Format PE Performance Evaluation VIII MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Permenkes Ministerial Regulation of Health Permenkeu Ministerial Regulation of Finance PermenPAN-RB Ministerial Regulation of Administrative and Bureaucratic Reform PJKAKI Network Development and Cooperation Between Commissions and Institutions PKN-STAN State Finance Polytechnic Institute of STAN PN District Court PNS Civil Servant POLRI Indonesian Police Poltekkes State Polytechnic of Health PPATK The Indonesian Financial Transaction Reports and Analysis Center PR Public Relations PSBB Large-Scale of Social Restrictions PSHK Center for Law and Policy Studies PUKAT Anti-Corruption Study Center Pusdiklat BPK BPK Training Center Puskesmas Community Health Centers QLUE Jakarta “Smart City” Reporting Application for Citizens RPJMN National Mid-Term Development Plan RT Required Task SI Social Impact, Inc. Sesjampidsus Secretary of Deputy Attorney General for Special Crime SIAP Strengthening Integrity and Accountability Program SIJAPTI Civil Service Commission’s Information System for Senior Career Executives SIPINTER Civil Service Commission’s Independent Assessment on Merit Information System SIMPel Case Management Information System SIMSI Dispute Management Information System SIPP Public Service Information System SIPTL Follow-Up Monitoring Information System SKM Public Satisfaction Survey SNI ISO National Indonesian Standard International Organization for Standardization SOLUSI Anti-Corruption Information and Consultation Desk SOP Standard Operating Procedure SOW Scope of Work SP4N LAPOR! National Complaint Handling System SPI Internal Control Unit SPIP Governmental Internal Control System SPPT-TI Integrated Criminal Case Handling Database System – Information Technology STRANAS PK National Strategy on Corruption Prevention SUSTAIN Support for Justice Sector Reform in Indonesia TAF The Asia Foundation TePAT Tolerance and Pluralism Awareness and Attitudes Test TIPIKOR Special Court for Corruption Crimes TL Team Leader U-FE Utilization-Focused Evaluation UGM Gadjah Mada University UI Indonesia University UMJ Muhammadiyah University UNDP United Nations Development Programme UNPAD Padjadjaran University IX MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report UNTIRTA Sultan Ageng Tirtayasa University UNUSIA Nahdlatul Ulama University USAID United States Agency for International Development VE Violent Extremism YAPPIKA Foundation for Strengthening Participation, Initiative and Partnerships of the Indonesian Society 1 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EXECUTIVE SUMMARY PURPOSE The United States Agency for International Development (USAID) commissioned a final Performance Evaluation of the USAID/Indonesia CEGAH Activity through the USAID/Indonesia MEL Support Program led by Social Impact, Inc. The objective of this evaluation with regard to accountability was to determine the degree to which CEGAH achieved its stated goals, why or why not, and how. The objective of this evaluation with regard to learning was to capture lessons that may be applied to current and future programming. As an independent review of CEGAH’s performance, this evaluation highlights results, factors influencing those results, challenges faced by the Activity over the course of implementation, and how the Activity did or did not address these challenges. BACKGROUND Contract Number AID-497-C-16-00007 Contract Period May 31, 2016 to May 30, 2021 Total Estimated Cost $24.8 million Contracting Officer’s Representative (COR): Diah Januarti (former) Implementing Partner Management Systems International Chief of Party Gerard Mosquera / Juhani Grossmann (former) CEGAH aims to contribute to the achievement of the 2014-2020 USAID/Indonesia CDCS Intermediate Result (IR) 1.1: Community of Accountability Improved as well as all three of its sub-IRs, which align with the three principal components of the Activity: 1.1.1: Effectiveness of justice sector to prosecute and adjudicate corruption cases increased; 1.1.2: Key Government of Indonesia (GOI) corruption prevention institutions strengthened; and 1.1.3: Civil society initiatives on accountability increased. METHODOLOGY Given the unique limits on the final CEGAH performance evaluation imposed by the coronavirus pandemic, Social Impact, Inc’s. (SI’s) Evaluation Team (ET) designed this methodology with the following restrictions: the team leader (TL) did not travel to Indonesia for fieldwork and led the ET’s data collection efforts remotely. The ET’s in-country team members also conducted remote data collection or virtual site visits from their homes. SI used a mixed-methods design for this evaluation consisting of a document review, key informant interviews (KIIs), online mini survey, and “virtual” site visits to North Sumatra (Medan City, Binjai, Deli Serdang, Nias, Gunung Sitoli, and Pematang Siantar) and East Java (Malang City, Jember). The remote data collection was carried out from May through July 2020 using Microsoft Teams and other online videoconferencing/calling platforms. In all, the ET conducted 77 KIIs with 123 CEGAH stakeholders, including USAID staff, implementing partner MSI, CEGAH grantees and beneficiaries (including GOI representatives, judges and court staff, and civil society and media representatives) and external stakeholders. The ET administered the online survey through Survey Monkey, sending a link to 126 beneficiaries of several selected CEGAH grants via email or WhatsApp; it received 26 responses. 2 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report FINDINGS & CONCLUSIONS 1. In what ways did the design and structure of CEGAH help or hinder the achievement of its results? Many stakeholders saw the benefits of CEGAH’s design and structure: CEGAH’s multi￾dimensional, multi-faceted “supply and demand” approach necessitated a “breadth over depth” strategy, with a large number of shorter “required tasks” (RTs) – currently, 52 such tasks – spread across 19 government ministries and agencies and a number of CSOs (please see list of all of CEGAH’s required tasks in Annex 6) A significant number of grantee and beneficiary KIIs (21 out of 77) praised CEGAH’s approach for building/strengthening a network of supportive actors across the GOI and judiciary; for the flexibility to work with many stakeholders across ministries, the opportunity to build relationships between and within these ministries, agencies, commissions, offices and courts; and the ability to shift partners/focus if an intervention wasn’t producing the desired results. CEGAH’s network fostered productive inter-agency collaboration: CEGAH’s network has helped in fostering inter-agency collaboration through the good relations its staff have built with different stakeholders, particularly Indonesia’s Corruption Eradication Commission (KPK) and the Supreme Court. Generally, this has provided a high level of inter-agency connectivity among anti-corruption actors, a productive platform for establishing and nurturing these connections, and exploring potential new engagements as well, and more effective anti-corruption interventions in a number of cases. CEGAH seen as having too many tasks and too little in resources/time devoted to each task: In its interviews, the ET found that 14 beneficiaries, and implementing partner staff, felt that CEGAH’s design and structure included too many tradeoffs - i.e., too many tasks that were too short or not sufficiently funded. They told the ET that the relatively short duration of CEGAH interventions in particular limited their tasks' effectiveness. Some limitations due to differences in culture and capabilities among CEGAH beneficiaries: CEGAH’s inter-agency collaborations were sometimes challenged by differences between these organizations, including in their capabilities, willingness to engage with CEGAH or other partners under the interventions, or institutional cultures. 2. Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 2a: More Successful: Direktori Putusan. An area of CEGAH’s successful support to Indonesia’s Supreme Court is the Direktori Putusan (Case Directory) system, which collects and stores data from court rulings at multiple levels throughout Indonesia. This system has been beneficial for judges as a reference tool, and the public, CSOs, and journalists as a way to monitor court decisions and conduct research. CEGAH-supported updates have enabled the smarter and faster performance of the system. These capabilities, for example, are enabling analysis on whether there are sentencing disparities in corruption cases between courts and judges across Indonesia and will enable important implementation monitoring of the Supreme Court’s sentencing guidelines. SP4N LAPOR!. The ET found that SP4N LAPOR!, the CEGAH-supported national complaint handling system of Indonesia, improved in several areas, including increased integration of the system within its 33 pilot agencies and local governments, especially when coordinated with both local and national monitoring mechanisms established by decision-makers (e.g., mayors) and oversight agencies. Such monitoring 3 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report mechanisms are helping to ensure that the public service delivery units/local government agencies more promptly respond to citizen complaints in locations such as Medan City, which also made practical improvements in its delivery of sanitation and registry services after residents used its new WhatsApp mechanism to register complaints in these areas. Additionally, recent upgrades have given the system greater ease of use and new functionality, including improved scalability to handle higher complaint volumes, a new executive dashboard on the SP4N LAPOR! website, and an e-learning platform hosted by KemenPAN-RB to help local-level stakeholders better operationalize the system. Additional successful CEGAH interventions identified by the ET include capacity building in and application of data-driven journalism to produce high-quality investigative coverage, the effective certification of a pilot group of district courts in SNI ISO 37001 (an internationally-recognized anti-bribery management system), collaborative local government use of community satisfaction surveys to strengthen public health service delivery. 2b. Less Successful: CVE Interventions. The ET found that a majority of government and judiciary stakeholders it interviewed or surveyed demonstrated little interest in being part of CVE interventions. Additionally, the value of CEGAH-supported research at the nexus of CVE and corruption was seen as relatively limited. Finally, tools such as the Tolerance and Pluralism Awareness/Attitude Test (TePAT), while popular with some GOI institutions, showed a potential for misuse and human rights abuses in how it was applied to disqualify candidates for civil service jobs. 2c. External Factors: The ET identified several external factors that are affecting – or have the strong potential to affect – the effectiveness of CEGAH’s interventions. It determined that COVID-19, for example, is forcing many supported government and judiciary institutions to repurpose funding initially intended to support their anti-corruption work for the GOI’s pandemic response; in turn, this decrease in available resources among these organizations has forced cancellations of trainings and suspended development of new platforms, regulations, research, or other anti-corruption tools. However, the ET also found that CEGAH-built capacity in its supported government and judiciary institutions is still present regardless of the pandemic’s effects, and that CEGAH support has provided the ability to shift to or continue many beneficiary activities online. Separately, the ET found in both its KIIs and survey responses that the new KPK law (Corruption Eradication Commission Law – UU KPK) may decrease the agency’s effectiveness in several ways, such as the potential for the KPK’s new supervisory board to have a "chilling effect” on the agency pursuing enforcement actions as robustly as it has in the past. The new law may also shift KPK's focus more to prevention (versus enforcement) activities. Additionally, CEGAH interventions that rely heavily on utilization of online platforms (SP4N LAPOR!) have been affected by the varying quality of internet coverage in some parts of Indonesia. CEGAH interventions have also been affected by key staffing rotations and departures within the Indonesian government, particularly at the local level, which has resulted in changing levels of commitment from local governments to interventions or losing implementation time by having to retrain new staff, they added. 3. Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 4 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report The ET identified three criteria that will play a significant role in determining the ability of CEGAH activities to be sustainable after the activity ends: sufficient capabilities and resources to continue the work on their own, political will in the organization to continue or expand the activity, and evidence of the intervention’s effectiveness or usefulness within the organization or in the public space. A number of CEGAH-supported interventions meet these three criteria and appear well-positioned to continue this work after it ends, including the ongoing investigation and prosecution under the Corporate Criminal Liability regulation, the Direktori Putusan, and the Supreme Court’s corruption sentencing guidelines. However, certain unknowns, including some of the external factors outlined in this evaluation, may also impact the longer-term sustainability of these interventions. For example, while KPK is likely to attempt additional investigations of corporate corruption under the regulation on investigation and prosecution of Corporate Criminal Liability in the future, the new KPK law’s requirement that it must obtain the new supervisory board’s approval for warrants – e.g., to seal investigation sites, conduct raids, obtain wiretaps – may significantly limit how effectively KPK can actually investigate these cases. 4. What lessons learned in strategy and approach can and should be applied to future programming? Ensure strategic coordination with stakeholder counterparts to accelerate and implement interventions more smoothly. One issue raised by both grantees and beneficiaries was how to better address coordination issues that arose between stakeholders as part of CEGAH interventions. Some grantees, for example, thought that earlier planning between the KPK and AGO could have improved discussions and expectations and averted the dropping of RT 7 (“Support effective coordination between the KPK, AGO, and other law enforcement entities to strengthen corruption indictments”). In the future, several grantees suggested the use of earlier planning, MoUs that carefully delineate roles and responsibilities, and enhanced roles for the Supreme Court to motivate other organizations’ greater involvement. Expand and strengthen training opportunities. Training interventions should balance the needs, perspectives, and bureaucratic cultures of the participants. For example, an important secondary aim of one beneficiary training on financial investigation was to encourage more interaction and discussion among its interagency participants. The training had been designed with the aim of obtaining balanced representation from participants’ institutions. However, only two judges participated, while the rest of the participants were from two other organizations that dominated the training, making it a less productive interagency learning opportunity. “We need to [be] careful when trying to control the composition of participants, since we want their buy-in on the activities,” said one grantee. “We need to constantly foster the collaboration.” RECOMMENDATIONS Expand work with CSOs to build longer-term anti-corruption benefits. The ET recommends that USAID continue to invest in building a “community of accountability” in future anti-corruption activities through strong civil society participation, more room for and support to CSO initiatives, and opportunities to enable collaboration between CSOs and government. CEGAH initiatives like strengthening the use of data-driven journalism with journalism CSOs are good examples of strengthening the role of civil society while also fighting corruption. Conduct follow-up CEGAH interventions that enable tracking of both short- and long-term effects of supported anti-corruption activities. In the time remaining in the CEGAH activity, the ET recommends following up initial tasks with follow-on interventions to extend/expand the work of the intervention while also tracking the longer-term effects of CEGAH support. This could also provide some flexibility to conduct a smaller-scale pilot project to verify an intervention’s effectiveness. 5 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Build stronger learning culture and system into future anti-corruption programs. For future activities, the ET recommends that USAID incorporate a stronger learning culture and system into the program’s management within their design. This should include at least two things: first, frequent, and structured, learning sessions organized not only within the program’s management unit but also among the beneficiaries and grantees, either as stand-alone events or by incorporating smaller sessions into current CEGAH events. Second, the measurement and collection of data not only for accountability and demonstration of impact, but also for learning. Add greater flexibility and accountability to SP4N LAPOR! People must have multiple channels for complaining in case they can't access the Internet (which is a challenge in a country the size and diversity of Indonesia). The ET recommends greater promotion of alternate options to users for filing complaints (e.g., via SMS message, the manual complaint-filing system) and to local leaders for potential creation of “home-grown” solutions such as Medan City’s WhatsApp-based system. It also recommends advising KemenPAN-RB, KSP, and ORI officials to establish and integrate a joint, robust monitoring and supervision mechanism and supporting features into SP4N LAPOR! Create system of local incentives: The ET recommends that ministry/government agencies at the national or local levels establish an incentives system that could be supported with strong oversight. Specifically, this system could provide options to provide higher compensation to government officials (with the expectation of improved complaint response performance in return), penalties if responsible individuals didn’t respond to a complaint within the stipulated time, etc. This monitoring and oversight functionality in the system could be extended to the local governments as a set of tools they could use to support the monitoring and evaluation of SP4N LAPOR! implementation. Support alternative strategies to advance SP4N LAPOR! coverage and inclusion: Given how expansive and populous Indonesia is, the ET recommends that USAID consider alternatives to its current strategy, including suggesting modifications to KEMENPAN-RB’s current roadmap to address future coverage and inclusion needs (while the UNDP is formally responsible for this roadmap, the ET feels that offering innovative suggestions through interagency channels would be beneficial in this instance). In terms of coverage, this could include shifting focus away from remote, less developed locations and toward localities that have adequate telecommunication infrastructure (internet and phone signal) with greater digital or gadget literacy among the communities. In such communities, future efforts to socialize SP4N LAPOR! and integrate it with local complaint systems could be done with a lighter grantee “footprint,” including more communications with local leaders and socialization with communities online (which would work better in the current COVID-19 environment as well) and then by continuing one-time events such as SP4N LAPOR! Goes to Campus. In conjunction with its other recommendations on SP4N LAPOR!, the ET believes this will yield more “bang for the buck” in terms of increases in overall numbers of users, complaints, etc. At the same time, this strategy could be designed to more formally incorporate inclusion of marginalized or remote populations by designing pilots that mandate this kind of support for, for example, people with disabilities. Strengthen capacity of local SP4N LAPOR! oversight/implementation staffing in select locations. In conjunction with suggested changes above to the SP4N LAPOR! implementation strategy, the ET also recommends that CEGAH as well as future Mission-supported anti-corruption programming help to strengthen the capacity of government oversight representatives in local areas, which will enable more robust monitoring of the system’s utilization in the pilot localities. Such training could be conducted regionally or, alternatively, online, which would have the added benefit of being safer during the ongoing COVID-19 pandemic. 6 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Extend the duration of CEGAH/future anti-corruption interventions to enhance their effectiveness. Longer-term interventions can help to achieve better outcomes, such as policy making, and ensure that policy changes are approved and/or implemented within relevant institutions. They can also support human resource capacity strengthening; enlarge and strengthen joint coordination among institutions; and improve the ability of training participants. The opportunities to resolve coordination issues between or within stakeholders will also be a possibility. Enhance CEGAH’s monitoring and evaluation tools to better enable higher-level assessments of effectiveness. To help assess the effect of the Akademi Antikorupsi e-learning trainings, the ET recommends that CEGAH conduct a periodic survey of repeat users of the training – e.g., those who have earned a certificate of completion – on their learning, knowledge, and perception of corruption in Indonesia, as well as the degree to which users have been able to recall and apply the training in their studies and/or work. CEGAH could conduct the first iteration of the survey before the Activity ends (May 2021), then negotiate to have ICW continue to conduct it periodically (e.g., annually), which would enable the survey to identify trends among Academy participants. For a future anti-corruption activity, consider handing off specific CEGAH tasks: ● Local government components of SP4N LAPOR! – Providing support to an intervention as big and complex as SP4N LAPOR! and expecting tangible results requires dedicated resources, time and energy in working with various levels of government, national down to sub-national level, as well as civil society and the public as a whole. While CEGAH has made significant strides in socializing and integrating SP4N LAPOR! within its 27 pilot local governments and beyond, expanding this to other local governments in other regions of Indonesia will require a similarly significant effort on a major scale. This expansion might be more effective if it was implemented in conjunction with a local governance strengthening effort in areas in which USAID contractors have a presence and long-term relationships with local leaders. This could function as a cross-cutting partnership between a CEGAH-supported central government effort (in support of KemenPAN-RB, ORI, KSP) and an integrated parallel local governance-strengthening effort. ● Countering Violent Extremism – As CEGAH winds down, the ET recommends maintaining the relationships built between grantees, beneficiaries and USAID using CEGAH's strong network and leverage them within current CVE activities (e.g., HARMONI) as well as future ones; similarly, use CEGAH-supported research as baselines wherever possible for future studies of similar at-risk groups by stand-alone CVE programming. Identify more ways to maintain/expand online platforms/tools that have shown to be useful – or at least promising – among beneficiaries and through use by the public. They not only can reach more people throughout the country, an important consideration given Indonesia’s size and population, but can also continue to be used during the pandemic. 7 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EVALUATION PURPOSE The United States Agency for International Development (USAID) commissioned a final Performance Evaluation of CEGAH through the USAID/Indonesia MEL Support Program led by Social Impact, Inc. USAID’s Evaluation Policy establishes two goals for evaluations: accountability and learning. To support accountability, evaluations should measure project effectiveness, relevance, and efficiency; match metrics to meaningful outputs and outcomes; and compare commitments and targets through valid measurement and credible analysis. To support learning, evaluations should test underlying assumptions and strategies to refine design and improve future efforts. The objective of this evaluation with regard to accountability is to determine the degree to which CEGAH achieved its stated goals, why or why not, and how. The objective of this evaluation with regard to learning is to capture lessons that may be applied to current and future programming. The final evaluation provides USAID with an independent review of CEGAH’s performance. The evaluation highlights the achievement of expected results, factors influencing the achievement of these results, challenges faced by the activity over the course of implementation, and how the program did or did not address these challenges. The evaluation also highlights lessons learned and other insights for USAID to consider in the design of future programming. THE DEVELOPMENT PROBLEM Accountability in governance can strengthen democracy, prevent conflict, and spur economic growth. In Indonesia, rampant corruption and a lack of transparency hinder development and diminish public confidence in government.1 The United States supports Indonesia to deepen reform efforts, foster transparent governance that is responsive to citizens’ needs, and help a culture of accountability flourish. Despite progress in addressing corruption and broad public support for anti-corruption efforts, Indonesia still faces significant challenges. Transparency International ranks Indonesia 85th out of 180 countries in public sector corruption, and low scores on World Bank Good Governance indicators point to substantial accountability challenges. As a founding member of the Open Government Partnership, Indonesia has identified openness as the key to unlocking its own potential for public service improvement and economic growth. The foundation of the theory of change for USAID CEGAH, is as follows: A Community of Accountability will achieve effective and sustainable corruption prevention and enforcement (CPE) outcomes if the justice sectors, institutions of accountability, civil society, media, and the business community are mobilized in targeted and coordinated efforts to promote a sustainable anti-corruption agenda throughout Indonesia. Under USAID/Indonesia’s Country Development Cooperation Strategy (CDCS) for 2014 to 2020, CEGAH forms one element of the broader Democracy, Rights, and Governance (DRG) project designed to achieve progress towards Development Objective 1, Democratic Governance and Resilience Strengthened, and the Intermediate Result of Community of Accountability Improved. In addition to contributing to USAID’s current portfolio, CEGAH has built on previous USAID investments, including Changes for Justice (C4J), the Educating and Equipping Tomorrow’s Justice Reformers (E2J) activity, and the Strengthening Integrity and Accountability Program (SIAP 1 and 2). 1 Final Performance Evaluation of CEGAH Statement of Work 8 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report USAID/Indonesia is drafting a new CDCS (2020 – 2024) that is expected to be approved in the summer of 2020. This strategy will focus heavily on integrating the principles of the Agency’s Journey to Self￾Reliance (J2SR) Framework across its development objectives and throughout implementation modalities. Given Indonesia’s J2SR Country Roadmap, the Mission is focused on fostering and advancing Indonesian￾led development outcomes. Government Effectiveness, Transparency, and Accountability will continue to remain a key component. This performance evaluation will help inform the design of transparency and accountability activities under the new CDCS. USAID’S RESPONSE SUMMARY INFORMATION Contract Number AID-497-C-16-00007 Contract Period May 31, 2016 to May 30, 2021 Total Estimated Cost $24.8 million Contracting Officer’s Representative (COR): Diah Januarti (former) Implementing Partner Management Systems International Chief of Party Gerard Mosquera / Juhani Grossmann (former) Under the Democratic Governance Project, CEGAH aims to contribute to the achievement of the CDCS’s Intermediate Result (IR) 1.1: Community of Accountability Improved and all three of its lower￾level results: 1.1.1: Effectiveness of justice sector to prosecute and adjudicate corruption cases increased; 1.1.2: Key Government of Indonesia (GOI) corruption prevention institutions strengthened; and 1.1.3: Civil society initiatives on accountability increased. These are captured in CEGAH’s results framework (Activity Monitoring, Evaluation, and Learning Plan [AMELP], May 2019) below: CEGAH’s highest level result is “Community of Accountability Improved.” The Activity aims to reduce corruption by addressing its root causes and bolstering the ability of the GOI to implement effective countermeasures, thereby strengthening the link between prevention and prosecution and ultimately improving accountability. 9 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report The objective hinges on the achievement of the following intermediate results: a) Effectiveness of the justice sector to prosecute and adjudicate corruption cases is increased: Component 1 focuses on addressing corruption within Indonesia’s justice sector through capacity building and systemic reform, including strengthened Supreme Court ability to gather and analyze data on corruption cases; reduced sentencing discrepancies in corruption cases; improved anti-corruption education in tertiary institutions; and improved awareness of judges and attorneys on the linkages between corruption and countering violent extremism (CVE), complemented by improved capacity to analyze case data to improve sentencing. b) Key GOI corruption prevention institutions are strengthened: Component 2 focuses on strengthening the capacity of key accountability institutions within the GOI to increase government transparency and prevent corruption, including strengthened integrated actions to prevent corruption by GOI institutions; strengthened administrative reform efforts, such as strengthening internal controls systems and improving civil service recruitment and promotion processes; enhanced engagement by accountability agencies with the media and civil society on corruption prevention campaigns; and strengthened prevention of corruption in budget planning and execution by selected sub-national governments, by following up audit findings. c) Civil society and the media’s capacity to provide oversight and hold the government accountable are increased: Component 3 focuses on strengthening civil society organizations (CSOs) and the media in efforts to provide oversight and hold government accountable to its citizens, including increased government adoption of CSO input on transparency and accountability policies; enhanced CSO advocacy capacity at the national and/or sub-national levels, regarding the accountability sector as well as the areas of health care, education, and/or the environment; and enhance civil society capacity to research, analyze, and advocate on issues related to corruption and violent extremism. These results include a 2018 modification to CEGAH’s Statement of Work to add examination of the nexus between corruption and violent extremism, which is mainstreamed across the three components. CEGAH’s approach addresses the systems, processes, and inter-relationships necessary for ensuring effective public accountability. The activity currently works with more than 19 government ministries and agencies and dozens of civil society organizations. This includes: • Working with Indonesia’s Supreme Court and Attorney General to improve the transparency of the judiciary’s case management system, reduce sentencing disparities, and promote data exchange among law enforcement agencies. • Improving anti-corruption education in universities and continuing education programs within the legal profession, including support to Indonesia Corruption Watch (ICW) to develop a free e￾learning platform (Akademi Antikorupsi or Anti-Corruption Academy). • Supporting Indonesia’s Anti-Corruption Agency, the President’s Office, and the Planning Ministry in developing and implementing Indonesia’s National Anti-Corruption Prevention Strategy. • Training government institutions such as the Ombudsman on System Investigations and the Ministries of Health and Education in areas such as citizen complaint and reporting systems, corruption vulnerability assessment management, and performance auditing and monitoring. • Assisting with the development of the National Complaint Handling System (SP4N LAPOR!). • Engaging with universities, civil society organizations, think tanks, and the private sector to strengthen public demand for accountability and contribute to enhanced anti-corruption strategies through evidence-based advocacy initiatives and data-driven investigative journalism. 10 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report • Promoting public understanding of anti-corruption via the Anti-Corruption Film Festival, including filmmaking workshops, film screenings, and interactive talk shows. Though CEGAH has primarily focused on the national level, the activity also supports activities in selected sub-national locations (namely, North Maluku, North Sumatra, Banten, and East Java provinces) by emphasizing corruption prevention and enhanced accountability in strategic sectors such as health and education. Among its expected outcomes, CEGAH aims to achieve: 1. More efficient and effective anti-corruption courts that adjudicate corruption cases consistently fairly and transparently; 2. More capable, transparent and accountable government service delivery agencies; 3. More proficient and respected CSO networks providing greater oversight of government conduct and performance; and 4. Better oversight capacity on the part of the government to monitor and prevent corruption and violent extremism. EVALUATION QUESTIONS This evaluation will address the following Evaluation Questions (EQ), drafted by USAID/Indonesia in the initial Scope of Work (SOW): 1. In what ways did the design and structure of CEGAH help or hinder the achievement of its results? 2. Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 3. Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 4. What lessons learned in strategy and approach can and should be applied to future programming? EVALUATION DESIGN AND METHODOLOGY Given the unique limitations of this evaluation imposed by the coronavirus pandemic, Social Impact, Inc’s. (SI’s) Evaluation Team (ET) designed this methodology with the following restrictions: the team leader (TL) did not travel to Indonesia for fieldwork and led the ET’s data collection efforts remotely. The ET’s in-country team members also conducted remote data collection or virtual site visits from their homes. The remote data collection was carried out during June 2020 using online videoconferencing and communications platforms such as Microsoft Teams, Zoom, and Skype Call. SI’s Jakarta-based “Monitoring, Evaluation, and Learning (MEL) Support to USAID/Indonesia” Program supported logistics for the ET, such as obtaining lists of respondents and official letters. DATA COLLECTION SI used a mixed-methods design for this evaluation, which collected both qualitative and quantitative data to address the EQs. The design consisted of the following data collection methods: document review, site visits, key informant interviews (KIIs), and a mini survey. 11 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report DOCUMENT REVIEW The ET reviewed the documents produced by and relevant to CEGAH in order to better understand the Activity’s design and implementation, extracted findings relevant to the EQs, and informed development of the evaluation’s data collection protocols so that instruments appropriately supplemented or cross￾checked against information in the background documents. The ET reviewed a range of Activity documents included in the SOW, including CEGAH’s annual and quarterly reports; annual work plans; project snapshots; and AMELP. Additionally, the ET reviewed relevant USAID documents (e.g., the USAID/Indonesia 2014-2020 CDCS) as well as pertinent GOI strategies, laws, and regulations. The ET also utilized relevant third-party research from reputable organizations (e.g., Transparency International). KEY INFORMANT INTERVIEWS KIIs were conducted remotely by the ET. The KIIs were conducted on an individual basis or in small groups (to maximize efficiency) with key target groups including CEGAH beneficiaries (e.g., GOI representatives, judges and court staff, law school and university students, civil society, and media representatives) as well as USAID, CEGAH implementing partner staff, and external stakeholders (e.g., non-USAID donors, think tanks, business association representatives, and individual businesses). The ET anticipated KIIs would last approximately 1-1.5 hours each. When also factoring in time for interview note cleaning and validation, the ET conducted approximately four to six KIIs per day, which were done by two sub-teams composed of one of the ET’s evaluation specialists and either the team’s research assistant or the SI MEL Support to USAID/Indonesia’s in-country program manager/M&E specialist. This structure was augmented by the TL joining and leading nearly all KIIs with English-speaking respondents, and also joining KIIs with Bahasa-speaking stakeholders that were considered high priority (e.g., those in senior roles in their organizations), despite an 11-hour time difference for the TL. The ET developed semi-structured data collection protocols that delineated appropriate questions for each of the target groups, established the highest possible range of comparability among interviewed stakeholders, and ensured that questions and responses were consistently recorded. ONLINE SURVEY The ET conducted a small-scale/mini online survey, using SurveyMonkey, of CEGAH beneficiaries and grantees covering EQs 1 through 3. Conducting this survey benefitted the ET’s data collection in several ways. First, it enabled the ET to solicit input from a larger audience than would be possible with KII data collection alone. Second, it produced quantitative data that could be used to triangulate qualitative data obtained through KIIs. The ET utilized SurveyMonkey to administer the survey, which consisted of approximately 15 closed-ended questions and five open-ended questions designed to be completed in approximately 10-15 minutes. To improve chances for a better survey response rate, CEGAH staff e￾mailed or texted the survey link to beneficiaries and grantees. SAMPLING ACTIVITY SELECTION Given the number (52) and diversity of CEGAH’s “required tasks” (RTs) and grantees/beneficiaries implementing those, the ET considered several factors that guided its focus on a narrower but still representative set of CEGAH tasks. These factors align with the criteria applied by the ET in sampling for KII and survey participants and virtual site visits. 12 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report • The ET ensured that it focused on RTs that were representative of all three CEGAH components. • The ET placed less emphasis on specific tasks that CEGAH chose to end relatively early in the life of the Activity, such as the 11 tasks “frozen” during year two2 (in order to allow “more funds to flow toward more promising engagements”3), Most of these were designated lower priority activities by CEGAH for reasons with which the ET concurs – e.g., because they funded tools that were being used minimally and were not accompanied by policy changes that would encourage public use of these tools, because little progress had been made in getting supported organizations to change their standards, because these tasks were being consolidated with others, or due to a lack of funding to continue these tasks. • Similarly, the ET focused less on grants for “self-standing local programs” that CEGAH began phasing out after year two (in lieu of national programs with regional/local support components) because of the significant resources involved with maintaining them.4 SITE SELECTION Because CEGAH has focused its work primarily at the national level, the majority of the interviewees and the ET’s KIIs were in Jakarta. However, as CEGAH also implements a number of its 52 required tasks (RTs) in sub-national locations, the ET also conducted “virtual” site visits to North Sumatra (Medan City, Binjai, Deli Serdang, Nias, Gunung Sitoli, and Pematang Siantar) and East Java (Malang City, Jember). These sites were purposively selected based on the following: • RTs under all three CEGAH components/results have been implemented there (see table 1 below for examples of RTs under each component). • They have relatively high levels of CEGAH support in these regions as assessed by levels of financial support and/or the ongoing presence or number of interventions. • Inclusive representation of different geographic regions of Indonesia. TABLE 1. CEGAH EVALUATION – VIRTUAL SUB-NATIONAL SITE VISITS & ACTIVITIES Sub-National Sites Component 1 Activities Component 2 Activities Component 3 Activities North Sumatra Training of courts in financial investigation and asset recovery SP4N LAPOR! CSO Engagement on SP4N LAPOR! Support to CSOs (e.g., ELSAKA) on health care accountability East Java Training of courts in financial investigation and asset recovery Strengthening Auditor & School supervisors on CVE Training on Public Services and SIPP Effectiveness of Regional Advocacy Forum (KAD) Adoption of Anti-corruption Academy in Poltekkes Malang Monitoring efforts in East Java (Malang, Jember) – e.g., focus on 2 In the case of EQ2, for example, because these 11 did not figure prominently among those activities grantee and beneficiary interviewees saw as the most or least successful CEGAH interventions, and did not sufficiently meet the criteria the ET established for determining success (see last bullet), the team excluded these RTs from consideration under this question. 3 CEGAH Year 2 Workplan 4 CEGAH Year 2 Workplan 13 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Sub-National Sites Component 1 Activities Component 2 Activities Component 3 Activities Training on updated SIPTL platform for ministries & agencies subject to BPK audits public procurement & Integrated Criminal Justice System (SPPT-TI) Support to CSO (IDEA) in Jember on updated & improved Public Satisfaction Survey (to measure public satisfaction with health care services) KEY INFORMANT INTERVIEWS To construct its sampling frame for KIIs, the ET primarily drew from the contact list of 72 beneficiaries and 94 grantees provided by CEGAH. The ET used purposive sampling to identify grantee and beneficiary candidates from these lists, with the following criteria: • The KII selections were representative of CEGAH in terms of its components, clusters, and geographic scope; • The KII selections strived for balanced gender representation within its sample to the maximum extent possible to account for any differences in the benefits of CEGAH’s interventions for women versus men; • The KII selections considered the key informants’ relative positions of authority within their organizations and the degree to which they were beneficiaries of CEGAH support. The ET also utilized snowball sampling to reach additional potential respondents. The ET also interviewed USAID and CEGAH (e.g., MSI, The Asia Foundation) staff, and identified relevant external stakeholders (e.g., non-USAID donors, think tanks, business association reps, and individual businesses) that could provide objective but knowledgeable perspectives on CEGAH’s interventions. The table below includes the number of KIIs conducted by the ET. The ET conducted its KIIs using a dual sub-team approach due to the breadth of interventions under CEGAH, which enabled the ET to engage more of the Activity’s stakeholders. 14 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report TABLE 2. THE DISTRIBUTION OF KIIS KIIs Jakarta North Sumatera5 East Java6 Total USAID 2 2 CEGAH Implementing Partners (MSI) 8 8 CEGAH’s Partner Grantees 12 12 GOI Representatives 37 7 2 46 Judiciary (Judges & Court Officials) 12 2 0 14 Civil Society Organization 6 3 9 CEGAH’s Other Grantees 24 1 25 Media 5 5 External Stakeholders 2 2 Total 102 15 6 123 SURVEY Survey respondents were purposefully selected from the list of beneficiaries and grantees provided by the CEGAH team; specifically, the ET sent the survey to all RT beneficiaries on the list, as well as additional beneficiaries of several selected grants. These grants were selected because they were representative of CEGAH’s three primary components, were implemented in different regions of the country, consisted of beneficiaries who have participated in CEGAH-supported trainings relatively recently (so were more likely to remember the experience and be more aware of CEGAH). A total of 126 respondents received the link to the survey via email or WhatsApp, and the ET received 26 responses. LIMITATIONS • Travel Restrictions – Given limitations mandated by the COVID-19 situation and Large-Scale of Social Restrictions (PSBB) rules in Indonesia during the data collection period (June 2020), the TL was not able to travel to Indonesia and needed to lead data collection efforts remotely. Additionally, the Indonesian team members were not able to travel within the country. Thus, the ET conducted video call-based rather than in-person interviews, and “virtual” site-visits (by video￾call using Microsoft Teams, Zoom, and Skype Call for phone-calls) to CEGAH Activity locations, such as Jakarta, North Sumatra, East Java, and other areas where respondents worked, including Banten, Bandung, Jogjakarta, Aceh, Lombok, Sulawesi, and Kalimantan. In all except a handful of cases, internet connections were sufficient to allow the ET to conduct its KIIs using the 5 Medan City, Pematang Siantar, Deli Serdang, Gunung Sitoli, Binjai, Nias 6 Malang City, Jember 15 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report aforementioned platforms; during the few times such connections were poor, the ET was able to complete the interview by mobile phone. Relatedly, the ET also was unable to conduct Focus Group Discussions (FGDs) because of Large-Scale Social Restrictions in June and the impracticalities of doing these remotely by videoconference. • Recall Bias – The ET was cognizant that respondent answers might be subject to recall bias related to past events or experiences. Indeed, when the ET triangulated its findings with data from the project documents, KIIs and survey, generally, more respondents tended to discuss more recent and/or bigger projects. The ET mitigated against this by asking follow-up probing questions where appropriate about other, less recent examples of their organizations’ work with CEGAH since the activity launched in 2016. Additionally, both the KIIs and survey allowed the ET to address this by ensuring a larger number of informants/respondents, thus mitigating individual challenges with recall. Primary data collection was cross-checked with CEGAH’s desk review documents, which given the timeframe over which they were developed, should be subject to less recall bias. • Selection Bias – In the course of data collection, the ET encountered selection bias, especially with regard to select KII respondents (e.g., the ET needed to select grantees and beneficiaries that were within specific geographic locations, while maintaining a gender balance to the extent possible). The ET attempted to mitigate selection bias to some extent by selecting KII respondents that cover as wide a range of selection criteria as possible (e.g., in terms of different CEGAH cluster types, activity objectives, etc.) and sent the survey to all CEGAH grantees and beneficiaries engaged since the launch of the Activity since 2016. • Response Bias – The PE may have evoked response/desirability biases whereby grantees or beneficiaries changed their answers due to explicit or implicit expectations. Grantees, for example, might believe that negative evaluation findings could affect their ability to obtain USAID funding in the future. The ET mitigated the risk of response bias, by utilizing data triangulation to note discrepancies in responses through different data sources (desk review, survey) and preparing follow-up questions for key informants that encourage more specific responses. The ET also mitigated this risk by clearly noting, as part of the informed consent procedures, that they were independent of CEGAH or USAID, and that their ability to maintain or in the future secure funding from the USG would in no way be impacted by their responses to the ET’s questions. The ET also ensured confidentiality of respondents, and only presented aggregated responses and data in the report. ETHICAL CONSIDERATIONS In line with its rigorous, proven approach on performance evaluations, SI’s Institutional Review Board (IRB) reviewed and approved all consent scripts and protocols used in this whole evaluation. Per IRB requirements, the ET has taken all necessary steps to ensure that respondents’ rights were protected, risks mitigated, and confidentiality of personally identifiable information of all respondents involved in this evaluation maintained. Evaluators informed KII participants through an informed consent “script” read to them prior to commencing the interviews that their comments would remain confidential and that their responses would be aggregated with others before inclusion in a report. 16 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report DATA ANALYSIS DATA ANALYSIS METHODS The ET employed a number of qualitative and quantitative data analysis methods to identify key findings from the collected data, as well as to draw conclusions and to make recommendations. The type of analyses depended on the specific data being assessed (e.g., content analysis for qualitative KII data). Analytical methods included the following: 1. Trend Analysis – Trend analysis enabled the ET to examine different CEGAH indicators over time to identify patterns of convergence (or divergence) of Activity outputs and outcomes toward the Activity’s objectives. 2. Gap Analysis – Gap analyses examined which aspects of the Activity, if any, fell short of anticipated performance, and the likely factors behind these gaps. 3. Comparative Analysis – The ET also took comparisons of CEGAH Activity results across the different data collection methods employed, as well as across different CEGAH stakeholder groups, objectives, or sectors to assess either convergence or divergence in perspectives. 4. Content Analysis – Content analysis entailed the ET’s intensive review of KII data to identify and highlight notable examples of CEGAH’s successes (or lack of successes) that supported (or inhibited) improvements in Indonesia’s “Community of Accountability.” Additionally, the ET held internal working sessions to identify and discuss emerging findings and themes, and to categorize conclusions and recommendations by EQ. The ET then used the emergent themes to generate a coding structure and systematically coded all qualitative data summary notes using a Microsoft Excel-based tally sheet. For coded KII data, the ET ensured that its results were disaggregated and analyzed through comparisons of the data by CEGAH intervention, objective, location, and gender. For its quantitative survey data, the ET ensured that survey results were analyzed through SurveyMonkey’s internal analytical tools to generate cross-tabulations, allowing disaggregation and comparisons of the data by CEGAH intervention, objective, location, and gender. DATA QUALITY Table 3 below demonstrates the methods the ET utilized during the CEGAH evaluation to validate the quality of its data and to ensure the quality of its analysis. These criteria are drawn from USAID’s Data Quality Standards. 17 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report TABLE 3. ENSURING QUALITY IN DATA ANALYSIS Data Quality Criteria Potential Issues Solution Validity Data don’t represent intended results Triangulation in KII protocol & survey question design & in data results; piloting of KII protocols and survey; internal team working sessions during fieldwork to verify accuracy/clarity of questions/methods. Reliability Inconsistent use of data collection and analysis methods Notes validation between team members; spot checks of question consistency across KIIs; internal team working sessions to verify consistent use of questions/methods. Integrity Risk of data transcription errors or manipulation KII notes validation between team members; safe storage of survey results (through SurveyMonkey, Google Forms, etc.); collective team identification of emerging themes/findings. Precision Insufficient detail of data to support client decision making Participatory process and agreement with client of evaluation design and scope, depth and type of data; mixed-methods design to ensure depth and breadth of data across data collection methods. Utility Data not timely or sufficiently practical to inform client decision making Data utilization plan with target dates; actionable recommendations GENDER AND SOCIAL INCLUSION ANALYSIS The ET worked with SI’s Gender and Social Inclusion Specialist to ensure that the evaluation design incorporated gender-sensitive approaches for data collection instruments, interviewing strategies, and outreach methods. The ET ensured gender diversity among interviewees and explored how women were included in activities and what role gender played in CEGAH’s processes and outcomes. All data collection methodologies (both KIIs and surveys) included gender-responsive questions, relating to EQ2 in particular and the gender aspects of the other EQs as appropriate. KII and survey questions were written to allow for analysis of any unintended consequences with respect to women and other marginalized groups. The ET ensured to the best of its ability that its sampling approach achieved balance between women and men, and included steps to facilitate this parity, such as adjusted the timing and location of KIIs, and allowed children to accompany the virtual interviewee, if needed. Women also made up most of the ET, which allowed for easier building of rapport during interviews with female respondents. Data were disaggregated by respondent gender (and location). Additionally, the ET explored whether CEGAH had a Gender Equality and Social Inclusion (GESI) plan, and the extent to which it was implemented. 18 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report FINDINGS AND CONCLUSIONS EQ1: IN WHAT WAYS DID THE DESIGN AND STRUCTURE OF CEGAH HELP OR HINDER THE ACHIEVEMENT OF ITS RESULTS? Introduction: USAID/Indonesia designed CEGAH to unite several of its previous anti-corruption activities, including Change for Justice (C4J) and the Strengthening Integrity and Accountability Program (SIAP) 1 and 2, in an approach one prominent CEGAH stakeholder called a “melting pot.” At the same time, the SOW for the CEGAH Activity noted that “given the project’s small budget compared to Indonesia’s enormous geographic size and population, this five-year effort will result in modest improvements in Indonesia’s ability to prevent corruption, be more transparent, reform its bureaucracy, protect citizen rights.....” This section describes how CEGAH stakeholders—implementer, grantees, beneficiaries, and USAID—believe this design helped or hindered CEGAH achieve its intended results. Specifically, this section covers stakeholders’ views on whether CEGAH’s design, which encompasses many smaller, shorter tasks, made it more or less flexible, engaged, burdensome, measurable, and effective. Many stakeholders saw benefits to CEGAH’s design. CEGAH’s multi-dimensional, multi-faceted “supply and demand” approach entailed a “breadth over depth” strategy, with many shorter “required tasks” (RTs)—currently, 52 such tasks—spread across 19 government ministries and agencies and several CSOs. Many (21) grantees and beneficiaries praised CEGAH’s approach for building and strengthening a network of supportive actors across ministries, agencies, commissions, offices, and courts, flexibly working with many stakeholders across ministries; and shifting focus or partners when interventions did not produce desired results. As one long-time CEGAH grantee said, “they have many RTs, so they can reach out to a lot of organizations in Indonesia . . . CEGAH reached not only the Supreme Court, Corruption Eradication Commission (KPK), Attorney General’s Office (AGO), but [also] bar associations, lot of CSOs, lot of ministries.” CEGAH’s network fostered productive interagency collaboration through the relations its staff built with stakeholders, particularly the KPK and the Supreme Court. By strengthening existing connections among anti-corruption actors and encouraging new engagements, CEGAH contributed to productive interagency collaboration, in many cases leading to more effective anti-corruption interventions. “Collaboration with CEGAH is not only talking about the project,” said one GOI beneficiary, ”[they] also have good investments in relationships in comparison to other agencies.” An example of the network’s ability to support results was CEGAH’s promotion of the Supreme Court guidelines on investigating and prosecuting Corporate Criminal Liability, a concept that until then had proven challenging to enact in the Indonesian justice system and which the Court enacted as a regulation in late 2016.7 CEGAH’s promotion of the new guidelines included a series of seminars that brought together the KPK and the Indonesian Chamber of Commerce (KADIN) to raise awareness within the private sector. “There was concern within the private sector about this new regulation,” said one government beneficiary. “In fact, [it] provides legal certainty for corporations and law enforcement.” Such coordination has provided a platform for ongoing private-public sector conversations on the new regulation, as well as clarity to enable better corporate compliance, another government beneficiary told the ET. Another GOI beneficiary highlighted a second example: addressing the long-running difficulty of coordinating efforts among Indonesia’s law enforcement agencies. CEGAH served as a catalyst between these organizations by bringing them together for joint trainings, such as on financial investigation and asset recovery, beginning in 2018. These trainings led to exchanges of ideas and allowed for balanced 7 The regulation (PerMA No. 13/2016) stated that if a business is found to have materially benefitted from a crime, acquiesced to the commission of a crime, or failed to take clear steps toward the prevention of a crime, it may face fines, the seizure of its assets, and the imprisonment of its personnel. CEGAH, FY 2017 Annual Report. 19 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report curricula that did not disproportionately benefit any one agency, which was an issue in the past. “What is done by CEGAH should be our duty as law enforcers, but we cannot coordinate,” a GOI beneficiary stated. According to another GOI beneficiary, “we never did a training with other law enforcers. The AGO has its own education program, BPK has its own education program. Because of CEGAH, the joint training happened.” As a third example, CEGAH helped several NGOs connect to and collaborate with GOI executing agencies—specifically, the Ministry of Bureaucratic and Administrative Reform (KemenPAN-RB), the Indonesian Ombudsman (ORI), and the Office of Presidential Staff (KSP)—to implement SP4N LAPOR! and the SP4N LAPOR! competition to measure public service improvements in local government agencies. One NGO noted that many GOI agencies have large internal structures, with some units not always coordinating well with other units. “CEGAH introduced me to all of them,” this judiciary beneficiary said. CEGAH also introduced NGOs to contacts in different agencies and facilitated agency approvals of changes to levels of effort and scopes of work. “They help us to communicate with them and get their approval and ensure that it is in line with the requirement of beneficiaries,” said a grantee. This network also made it easier for CEGAH to add a set of RTs focusing on a new area: studying and addressing the link between corruption and violent extremism, as one CEGAH staff member explained to the ET. CEGAH was flexible enough to absorb CVE into its design, since studying the corruption-CVE nexus required working in governance and government business processes, and the CEGAH network provided a ready group of organizations and actors that could become potential partners on these tasks, or at least consulted, or “pitched to,” he added. Stakeholders saw CEGAH as having too many tasks and too little resources and time devoted to each task. The SOW for CEGAH noted up front the tradeoffs of its “wide-angle” approach: “Given the project’s small budget compared to Indonesia’s enormous geographic size and population, this five-year effort will result in modest improvements in Indonesia’s ability to prevent corruption, be more transparent, reform its bureaucracy, protect citizen rights . . .” The ET found that 14 interviewed beneficiaries and implementing partner staff felt that CEGAH’s design and structure made too many tradeoffs, with too many tasks that were too short or underfunded. The relatively short duration of CEGAH interventions in particular limited tasks’ effectiveness. “The weaknesses of USAID CEGAH: the scope of program is still limited,” said one government beneficiary, “we need a program with wider scope and longer duration.” Another asserted that training modules could not be developed “optimally due to the strict time frame set by USAID/CEGAH,” and that they generally found this program to be “less structured and incidental, rather than a whole working program conducted for a year.” For example, CEGAH supported the formulation of guidelines/standard operating procedures (SOPs) on risk management for planners and auditors in the Inspectorate General of a GOI ministry, according to two beneficiaries. These guidelines made the process clearer, but IG auditors have not been able to implement them fully in their work for two main reasons: (1) They have not yet been able to integrate these guidelines or procedures for implementing them into the ministry’s regulations, and (2) although CEGAH organized trainings of trainers within the ministry to disseminate knowledge on these guidelines to their colleagues, these master trainers have not organized the needed trainings because there has been no movement on this from the ministry’s office of the secretary general, which has authority for risk management at the implementation level, including these trainings, within the ministry. “We wanted to take over, but in its [regulations], it says SecGen’s responsibility, so, we stop there,” said a ministerial beneficiary, adding that they have asked for CEGAH/USAID assistance to help push things forward on both issues, but at the time of this evaluation, had not yet heard back. CEGAH’s approach included follow-up grants on some tasks to enable additional “next steps” or additional progress. For example, when issues were identified with the design of Indonesia Corruption Watch’s 20 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report (ICW) online Anti-Corruption Academy training system, a follow-up grant enabled implementation of consultants’ recommendations to revise the Academy’s learning management system, improve user￾friendliness, enhance security, and refine its grading system.8 All 14 grantees and beneficiaries found this to be a good follow-up design. Ongoing communication and coordination by CEGAH was generally viewed as good. Respondents in more than half of the ET’s grantee and beneficiary KIIs (37 of 69) felt that CEGAH’s design helped staff maintain good communication and coordination with them throughout task implementation. However, nine respondents presented some examples of what they viewed as a lack of coordination among specific stakeholders on CEGAH interventions, including among grantees working with the same beneficiaries, between grantees and the IP or USAID, among beneficiaries, and even within the different divisions of a single beneficiary. Relatedly, some RTs appeared to produce similar if not redundant outputs, such as ICW’s e-learning Akademi Antikorupsi (Anti-Corruption Academy) and Indonesia Judicial Monitoring Society (MaPPI)’s work with Gadjah Mada University (UGM), Padjadjaran University (UNPAD), and Indonesia University (UI) to adopt anti-corruption modules into tertiary education curricula. While the original design for ICW’s Akademi focused on targeting activists, it evolved to the point where both sought to expand anti￾corruption education into university curricula up through FY 2018. This can in part be attributed to KPK requesting a change in focus for the MaPPI intervention (developing modules not only for existing university classes but also developing expanded versions for a standalone university course and for a professional education program9). Ultimately, however, CEGAH recognized the similarities of these efforts and elected to move forward only with ICW’s Akademi Antikorupsi after FY 2018, after MaPPI completed its deliverables for KPK, in order “to create economies of scale.” CEGAH encountered some limitations due to differences in culture and capabilities among CEGAH beneficiaries. CEGAH’s interagency collaborations were sometimes challenged by differences between organizations, including in their capabilities, willingness to engage with CEGAH or other partners, and institutional cultures, according to interviews with multiple grantees. Three noted their belief, for example, that these differences were not always sufficiently factored into the CEGAH planning process, with one stating “If you look at the program document, it seems like everyone is equal—same reform maturity, same understanding awareness, same influence on law enforcement. In reality, each institution is different.” After implementing an intervention, she noted that, among a few beneficiaries, “we didn’t see any kind of significant impact or at least significant willingness to conduct reform or consider us as a strategic and critical partner.” Differences in institutional culture could also make these partnerships difficult to implement, according to three grantees interviewed by the ET. Bigger ministries, for example, could be more hierarchical, bureaucratic, and rigid, while smaller organizations could be more flexible and could make decisions more quickly. For example, one of the grantees coordinated an intervention among three major but different GOI organizations, and found two of the partners easier to work with than the third. “The flexibility was varied among these program managers,” said the grantee: “One was really strict, while the other was quite flexible.” For example, CEGAH’s RT 7 focused on conducting an interagency discussion forum with KPK, AGO, and the Supreme Court on the different interpretations of the Anti-Corruption Law to reduce disparities in indictments and sentencing. However, relations among the agencies on this proved difficult, as KPK and AGO in particular tend to be institutional competitors and engage in turf wars over similar corruption work “There’s institutional ego and a complicated relationship between KPK and AGO—they compete on 8 CEGAH, FY 2019 Annual Report. 9 CEGAH, FY 2018 Annual Report. 21 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report corruption cases,” said one grantee. He added, “Our achievement was getting them to sit in the same room together.” But beyond that, the organizations made little headway in reconciling competing interpretations of and best approaches toward reducing indictments and sentencing disparities.” He added that they tried to “persuade them to have better coordination with each other,” but ultimately came up short. As a result of these ongoing difficulties, CEGAH had the grantee drop this collaboration. A separate example of these capacity and cultural differences among Indonesia’s public institutions creating challenges for CEGAH’s collaborative interventions was its support to seven district courts that recently obtained their SNI ISO 37001 certification (anti-bribery management systems; see also under EQ2a). According to CEGAH’s grantee and judiciary beneficiary, these “pilot” courts were spread throughout the country and were all different in terms of their cultures and the size and capabilities of their staff, which created challenges. Jakarta’s district court, for example, was more mature, had more personnel than the six others, and had certain expectations for the pace and structure of the training as the largest district court in Indonesia’s capital, while the district court of Ternate was smaller, very remote, and had technological issues that made it more challenging to communicate. Finally, a CEGAH grantee stated that understanding the differences in culture between all of their supported organizations is a necessity, but also a “very hard and difficult task. I don’t think CEGAH can do it well.” Instead, he added, the grantees themselves must understand the differences between the organizations they are supporting. “If we want to work deeply with an agency, then we have to have deep amount of knowledge,” she added. Some grantees see CEGAH’s administrative requirements as burdensome, but others do not. A total of 15 grantees and beneficiaries saw CEGAH's administrative requirements and work culture as rigid and burdensome, such as requiring too many reports or overly involved branding and formatting rules. "For reporting, we have to produce a lot of reports – sometimes it is overwhelming,” said one CEGAH grantee. “Sometimes I felt I already wrote it in other report, but I have to write again in a different report. We can simplify the reporting responsibility.” This same grantee had also worked with both DFAT and GIZ, which only required interim and final reports. Several noted, for example, that their CEGAH grants included milestones, each of which required a separate progress report, as well as activities and outputs within these milestones, which required separate reports as well. A separate grantee noted, “CEGAH is strict. When we have challenge or maybe there’s development or progress from our beneficiaries, sometimes it’s very hard to adjust because you know the project is very rigid.” Conversely, a total of 14 grantees and beneficiaries found CEGAH’s administrative requirements and work culture to be relatively flexible and simple in comparison to their work with other donors, stating that CEGAH staff was willing to modify strategies and activities when necessary. Four grantees, for example, appreciated CEGAH's utilization of Fixed Amount Awards, which they stated significantly reduced stakeholders’ financial reporting burden. Others found adherence to milestones a clearer, more straightforward process, and found CEGAH to be flexible when such milestones needed to be shifted. “CEGAH made it easy for us, delaying the time to submit milestones or adding milestones,” said one grantee. For example, when the grantee could not conduct an in-person training due to the COVID-19 pandemic, CEGAH permitted them to conduct it as an online Zoom meeting. When they could not finalize operating procedures because of changes in regional or local government leadership, CEGAH let them change the deadline or deliverable milestone. 0% 15% 65% 19% Strongly disagree Disagree Agree Strongly agree FIGURE 1. CEGAH’S SUPPORT HAS STRENGTHENED MY ORGANIZATION'S ABILITY TO FIGHT CORRUPTION MORE THAN SUPPORT I RECEIVED FROM OTHER DONORS. 22 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report More than 85 percent of GOI/judiciary beneficiary survey respondents (22 out of 26) stated that CEGAH’s support has strengthened their organization’s ability to fight corruption more than support they received from other donors (Figure 1). Challenges measuring CEGAH’s impact. CEGAH’s structure makes it difficult to assess its overall effectiveness. With such a wide range of tasks, sectors, and actors, it is challenging to develop overarching, higher-level indicators that can measure what all this adds up to. CEGAH uses several outcome-level indicators that cannot be attributed directly to CEGAH support, such as the World Justice Project Rule of Law Index. One grantee noted, “We have difficulties measuring the impact for all this approach that CEGAH has. Whether BPK has improved their internal accountability, whether KPK has improved their internal investigation process, we cannot measure that.” Additionally, individual interventions tend to emphasize output-level indicators that tell CEGAH less about the higher-level impact, such as information on numbers of training participants rather than outcomes based on participants’ applications of trainings. Similarly, an indicator such as “number of complaints in NCHS (National Complaint Handling System) processed within legal timeframe by assisted GOI,” does not reveal how many complaints were addressed satisfactorily or what was achieved once complaints were addressed. Additionally, this emphasis on the number of complaints filed and processed within the NCHS (i.e., SP4N LAPOR!) incentivizes grantees to implement activities that maximize the number of people who download the SP4N LAPOR! mobile app and then file a complaint (whether or not what they enter is an actual complaint or random text, as one grantee stated). This has resulted in the use of “gimmicks” to encourage app downloading and complaint filing. “This is the only way we can claim and get reimbursement,” the grantee stated. One additional issue with CEGAH’s indicators is that they often exceed targets, in some cases by significant margins. This is usually an indication that targets were set too low. USAID does not require a specific range within which an indicator must fall, but for a performance plan indicator, for example, it does require a “deviation narrative” if the result is 10 percent higher or lower than the target.10 Difficulties setting appropriate targets tend to be particularly challenging for broad, diverse activities like CEGAH. Relatedly, four grantees stated that the short duration of CEGAH’s tasks also made it more difficult to determine the effectiveness of specific interventions. For a CEGAH pilot-level intervention like the certification of the seven district-level courts in the SNI ISO 37001 “Anti-Bribery Management System” standard, for example, CEGAH could measure the relatively short-term achievement of court certification. But it could not measure the actual impact of the certification on the courts’ cases and day-to-day business, which would have been possible through a longer-term intervention (or a follow-up task under CEGAH) that could track metrics related to the actual application of SNI ISO 37001 practices as part cases, trials, and court operations, such as how quickly cases are resolved or whether employment due diligence and screening are performed. “It is difficult to assess the effectiveness of the program because we can only measure the output level,” one grantee told the ET. 10 USAID, Program Cycle Monitoring Toolkit: Performance Indicator Targets, November 2017, https://usaidlearninglab.org/sites/default/files/resource/files/cleared_-_mt_-_performance_indicator_targets_r.pdf. FIGURE 2. MARGINALIZED/VULNERABLE GROUPS WERE ABLE TO ACCESS/BENEFIT FROM OUR CEGAH￾SUPPORTED ACTIVITY TO THE SAME DEGREE THAT MEN WERE 23 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Consideration of women or marginalized groups in CEGAH’S design and structure. From the KIIs, 25 grantees and beneficiaries stated that they felt CEGAH did as much as possible to include women or marginalized groups in the design and structure of its activities. An additional 18 felt that their CEGAH￾supported anti-corruption interventions did not involve a gender focus, so they did not see this as relevant to CEGAH’s role. But 22 stated that their own grantee or beneficiary organizations’ policies, formally or informally, already provided equal opportunities to all. In the survey administered to beneficiaries, the ET found that almost 100 percent of respondents agreed that women were able to access or benefit from CEGAH-supported activities to the same degree that men were. A smaller group (six) stated that it was difficult to include more women and marginalized groups11 because of gender imbalances within the GOI and judiciary institutions, which employ more men than women at key levels. When asked if marginalized groups were able to access or benefit from CEGAH-supported activities 8 percent respondents from GOI/judiciary didn’t agree, but more than 90 percent agreed (Figure 2). CONCLUSIONS CEGAH focused on establishing, maintaining, and expanding ties with GOI and judiciary entities. This yielded positive results, including the flexibility to engage with a range of stakeholders across government ministries, foster interagency collaboration, and communicate with grantees and beneficiaries. At the same time, CEGAH has engaged major Indonesian CSOs such as ICW, Alliance of Independent Journalists (AJI), Indonesian Institute for Independent Judiciary (LEIP), Center for the Study of Law and Policy (PSHK), and others as partners to train and build ties with government agencies, work with local governments to improve accountability, monitor elections and selection of officials, and create learning tools for students and professionals. EQ2: AMONG CEGAH’S INTERVENTIONS, WHICH WERE THE MOST SUCCESSFUL, WHICH WERE THE LEAST SUCCESSFUL, AND WHY? WHAT WERE THE EXTERNAL FACTORS THAT AFFECTED THIS PROCESS? Given the number (52) and diversity of CEGAH’s “required tasks” (RTs) and associated interventions and grantees/beneficiaries implementing those, the ET took several steps in determining its most and least successful interventions: • The ET first narrowed its sample somewhat by excluding specific tasks that CEGAH chose to end relatively early in the life of the Activity for reasons related to efficiency or performance, such as the 11 tasks “frozen” during year two (in order to allow “more funds to flow toward more promising engagements”).12 Please see Methodology’s “Activity Selection” sub-section above for a more detailed explanation of this. • The ET then applied a three-part criterion to determine the most and least successful interventions under CEGAH, examining the degree to which: 1) interventions demonstrate their effectiveness by moving beyond output-level targets to deliver verifiable changes or benefits, either within the beneficiary organization itself or broader society; 2) they demonstrate or show the potential to be sustainable after CEGAH ends; and 3) they result in tools, platforms or trainings that are accessible in terms of their use by or potential to benefit wide parts of the Indonesian public. 11 Marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious groups, and immigrants. 12 CEGAH Year 2 Workplan 24 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report • Additionally, the ET’s selection process also factored in the results of its KIIs with grantees and beneficiaries when asked which interventions they felt were the most and least successful, using both the response “counts” for specific interventions from the ET’s internal tally sheet as well as their qualitative responses justifying their preferences for particular tasks. The ET then triangulated between this “scoring” of these interventions within the tally sheet and the application of the above three-part criteria to determine the most and least successful interventions below. The results of the ET’s analysis reveal that the most and least successful interventions discussed below cover multiple RTs. For example, SP4N LAPOR! received support from CEGAH under component 2 through RT 18 (for upgrading the technology and accessibility of the system, training local government staff in its use, and facilitating its implementation in 33 pilot government agencies), and component 3 through three other RTs13 (to support CSOs/grantees that work with local government and civil society representatives to promote and aid the local adoption and utilization of SP4N LAPOR!). Additionally, CVE activities discussed below cover six RTs.14 Also, interventions such as CEGAH’s support to the revitalization of the Direktori Putusan yielded cross-over benefits to other RTs – e.g., activities related to the revitalization of the system were conducted principally under RT 1, but four beneficiaries spoke of the system’s benefits to RT 6, supporting both the court and AGO to develop and implement common sentencing guidelines for corruption cases. EQ2A: MOST SUCCESSFUL INTERVENTIONS The following are the most successful CEGAH interventions, based on the ET’s findings and criteria: Direktori Putusan. A key area of CEGAH’s support to Indonesia’s Supreme Court under sub-IR 1.1.1 (component 1 - Effectiveness of justice sector to prosecute and adjudicate corruption cases increased) has been the Direktori Putusan (Case Directory) system, which collects and stores data from court rulings at multiple levels throughout Indonesia. This system has been beneficial for both judges as a reference tool, and the public, CSOs, and journalists as a way to monitor court decisions and conduct research. However, prior to CEGAH’s support, the system had not been able to deliver its full intended benefits because of several problems, including slow performance, a poor search engine, bad data input including issues like empty files, low capacity to accommodate large court files, and increasing system requirements. “Before CEGAH’s support, there were 3 million court decisions with lack of server support, a less reliable search engine, and there were some problems such as empty files,” said one judiciary beneficiary. Recognizing the important role of Direktori Putusan in improving the effectiveness of the justice sector in Indonesia, CEGAH provided support to improve the system and address these problems, upgrading Direktori Putusan to version 3, supported by a data warehouse infrastructure that enables smarter and faster performance. One of the grantees explained the strength of the current version of Direktori Putusan and how users have benefited from the improvement: “Version 3 is already able to accommodate 5 million court decisions with sub-second performance, meaning under one second or just a bit above one second speed. The strength is the search engine, it is a smart search engine that could give recommendation based on your search. This is what we highlight to the public because this is what the public usually use more. The information from Registrar Unit in Supreme Court is that the responses to the upgraded system are positive because it really facilitates searching of information.” Four key judiciary beneficiaries noted the benefits of the improved search engine, especially its speed and ease of use to support decision making. These capabilities, for example, are enabling analysis on whether there are sentencing disparities in corruption cases among courts and judges across Indonesia and will 13 RT 34, 35, 39 14 RT 44, 48, 49, 50, 51, 52 25 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report enable important implementation monitoring of the Supreme Court’s sentencing guidelines (see discussion under EQ3). “How is the trend of 400 District Courts (Pengadilan Negeri) in Indonesia when they’re trying to charge same cases, whether the disparity range is far or not. From this research, we will be able to give new policies,” said one judiciary beneficiary. Furthermore, the system is available not only to the courts but also to the general public, which can also use it to obtain information or conduct research. A significant increase in data traffic was recorded by CEGAH’s grantee. “The most successful are public services, so people can access them easily. We have made a report to CEGAH of the increase in traffic of people using the Direktori Putusan,” said one CEGAH grantee. Aside from faster and smarter system capabilities, Direktori Putusan is currently the only system that is ready to be scaled-up, if needed, to enable transfer of court documents and files electronically, according to one grantee. The importance of this capability was also mentioned by four judiciary beneficiaries, who recognized that while COVID-19 is ravaging the country, courts would still have to proceed and perform their essential functions. Because COVID-19 has put many restrictions on travel to deliver hard copies of decisions and other key documents that must be sent immediately, the Indonesian judiciary is planning to optimize the system to enable transfer of court documents electronically. One beneficiary acknowledged that, “the only system that is currently ready to accommodate this plan is Direktori Putusan.” Direktori Putusan directly supported sub-IR 1.1.1/component 1 and helped address corruption not just by serving as an accessible resource for gathering and analyzing data on corruption cases, including data that is assisting in reducing sentencing discrepancies in corruption cases, but also serving as a widely used tool by the public, CSOs and media to promote transparency and accountability of the judiciary and support CEGAH’s broader IR (1.1) to improve the community of accountability in Indonesia. While the evaluation confirmed the strengths and benefits of the newly revitalized Direktori Putusan, it also found ways in which it could be improved to achieve its potential. For example, grantees highlighted the importance of quality data input to the system to produce quality outputs. This would require the Supreme Court to adjust work flows as well as tasks and functions of staff in both the Supreme Court and lower courts. This also would entail development of new SOPs, provision of trainings, and implementation of changes in practices across the judiciary. Without such measures, the newly revitalized Direktori Putusan and underlying data warehouse will not be able to fully realize its potential. SP4N LAPOR! Under CEGAH’S sub-IR 1.2 (component 2 - Key Government of Indonesia (GOI) corruption prevention institutions strengthened), the activity provided comprehensive support to improve SP4N LAPOR! the national complaint handling system, and its implementation at both national and sub-national levels. Through CEGAH’s support, SP4N LAPOR! has achieved solid progress on several fronts. It should also be noted that the system’s overall effectiveness is as yet unclear, as an overall assessment has not been done, and integration and adoption at the local level has not always been consistent. At the national level, CEGAH provided support to improve the national complaint handling system as well as provide the necessary tools to support operationalization of SP4N LAPOR!. Upgrades from version 2.0 to the latest version, 3.1, included troubleshooting, improved user experience, new features, improved scalability to handle higher complaint volumes, a new executive dashboard on the SP4N LAPOR! website, and an e-learning platform hosted by KemenPAN-RB to help stakeholders at the local level operationalize SP4N LAPOR!. The executive dashboard is one the most important features in SP4N LAPOR!, letting decisionmakers at the national and local levels monitor management of complaints and improve public service delivery. Understanding the nature and content of public complaints is a crucial step to devising appropriate policies and programs for government agencies. As reported by CEGAH, KemenPAN-RB has begun to identify that many of the complaints reported through the system were in fact requests for information. Hence, KemenPAN-RB took the initiative to create and improve SIPP (the National Public 26 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Service Information System) to be more proactive in providing information on the types, locations, and costs of available public services.15 CEGAH also included an introduction to SIPP alongside the public awareness-raising campaign for SP4N LAPOR!. Ten interviewees from beneficiary organizations, CSOs, and grantees at the national and local levels stated that CEGAH’s support to SP4N LAPOR! provided effective assistance in key areas to the 33 pilot agencies, including in implementing the system in their respective areas and/or increasing system use. Interviewed grantees and local governments claimed that before CEGAH’s support, local governments had registered and set up accounts on SP4N LAPOR!, but the accounts remained inactive and unused by local governments. “The successful story was how 33 districts/cities that were connected but not active eventually become active on SP4N LAPOR!. This has led to increased number of complaints/reports coming through the system,” said one CEGAH grantee. CEGAH supported pilot agencies in pilot areas to introduce the latest version of the system, establish necessary workflows, create complaint handling units, SOPs, and local regulatory frameworks, and train SP4N LAPOR! administrators and local government working units. CEGAH also has helped to build awareness of the importance of implementing the SP4N LAPOR!, not only in the pilot locations, but also in other regions. This has proven an important step in CEGAH’s support to increase the number of government agencies integrated into the NCHS, a key metric of this intervention.16 For example, one grantee pointed out that their work in Banjarmasin, South Kalimantan (one of the pilot areas), has sparked the interest and willingness of other local government leaders near Banjarmasin to start operationalizing SP4N LAPOR! themselves. Similar to the experience in South Kalimantan, support in North Sumatra initially targeted only five districts/cities in the province, but the Provincial Secretary decided to include all 33 districts/cities to raise their awareness and understanding of how to implement SP4N LAPOR! in their own areas. CEGAH reported that its grantee has successfully helped 366 government agencies to connect with SP4N LAPOR!, improving their capacity to manage the system and establish complaint handling units.17 Despite the expansion, the evaluation found that some regions progressed further than others in their use of SP4N LAPOR!. Medan City, for example, made several practical improvements in how it delivers public services after city residents used the system to register complaints. For example, the Office for Sanitation and Landscaping in Medan City is now addressing problems with accumulated garbage on public roads. And its Civil Registry Office has sped up the process of issuing registry documents and delivering this service closer to the community. One CEGAH grantee pointed out that Medan City’s progress with SP4N LAPOR! has been great particularly because the local government has the adequate budget, capacity, and integrity. Medan City is also among the few pilot locations that received additional support to integrate their local complaint handling system, called Medan Rumah Kita, into the national system (SP4N LAPOR!) to streamline online complaint mechanisms. In its interviews with grantees and local governments, the ET found that better performance of SP4N LAPOR! was tied to monitoring mechanisms from decisionmakers and oversight agencies, including locally created systems and ombudsman representatives at both the national and local levels. Such monitoring mechanisms help ensure that the public service delivery units and local government agencies promptly respond to citizen complaints. Medan City, for example, devised its own monitoring mechanism through the creation of a chat group (in WhatsApp) consisting of each government unit’s liaison officer or administrators, as well as the Mayor of Medan City. This mechanism has proven to be quite effective in boosting the performance of government units in responding to complaints, because the local government 15 CEGAH, FY 2018 Annual Report. 16 Indicator 12, USAID CEGAH Activity, Monitoring, Evaluation and Learning Plan, May 2019. 17 CEGAH, FY 2020 Q1 Report. 27 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report units feel that the Mayor is informed of every conversation, and hence continuously monitors their responses. An interview with one of CEGAH’s grantees affirmed this conclusion. Not only had Medan City put in place a monitoring mechanism, he stated, but it had also used performance in responding to public complaints in SP4N LAPOR! as a consideration in its regional budget allocation process: “Medan city is quite successful . . . They have made the complaints from citizens for evaluation of performance of local government units and input for regional budgeting process (APBD),” said the grantee. Through campaign activities like “SP4N LAPOR! Goes to Campus” (LGTC) and technological advances such as improvements to the app and the launch of Android/iOS mobile apps to facilitate access and enhance user experiences, CEGAH also strengthened the promotion of SP4N LAPOR! to the wider public. These activities were relatively successful, as the number of new users downloading the app to their phone has continued to increase. CEGAH’s 2019 annual report noted that as of September 30, 2019, more than 9,000 new users have downloaded the app.18 As of July 12, 2020, the Google Play store had recorded approximately 50,000 app downloads. In addition to this, one of the key institutions responsible for overseeing SP4N LAPOR! recognized that the system is more visible now, as indicated by the increased number of complaints coming through the system: “For SP4N LAPOR! REPORT it is more visible . . . We encourage regional governments to be more transparent. This is more visible, because now complaints are higher than in 2016 or 2017.” In general, users and administrators view the system positively, as shown in a recent survey of SP4N LAPOR! stakeholders. Among respondents, 72 percent felt satisfied using SP4N LAPOR!.19 The awareness and visibility of SP4N LAPOR! as a national complaint handling system is increasing, particularly among local governments, even though there is more work to be done to socialize the system to the wider public in Indonesia. While CEGAH has successfully increased the visibility and implementation of SP4N LAPOR!, especially in the pilot areas, the ET also identified several issues in the use, implementation, and management of the system. For example, the utilization rate of the system is still considered low in comparison to Indonesia’s population, a point echoed by three government and CSO beneficiaries. According to the SP4N LAPOR! website, as of July 13, 2020, there were 801,257 registered users and 1,389,891 complaints. Two government representatives noted that the number of users and submitted complaints in SP4N LAPOR! is not proportionate to the total population. “The number of users in SP4N LAPOR! is currently under 1 percent of total population, or about 1 million,” said one government beneficiary. Additionally, a SP4N LAPOR! administrator at the local level shared that the number of complaints submitted to the system is not very high: “In the Nias Islands, the reports or complaints are not much. Through the report in 2019, there were around 300 reports. Until now, there has been no increase in complaints.” The ET also found that, aside from the low utilization rate, the report closure rate of SP4N LAPOR! is also low. Specifically, the majority of reports are either in pending status or have taken longer than 60 days to close (based on data from October–December 2019).20 These relatively low utilization and report closure rates indicate challenges in the wider implementation of SP4N LAPOR! as a national complaint handling system. The ET found that rapid staff turnover among the personnel in charge of administering the system within each local government unit was a significant obstacle to SP4N LAPOR!, according to four local government representatives. In general, the role of the SP4N LAPOR! administrator is assigned to staff who do not have enough authority as decisionmakers to sufficiently enforce implementation of the system and who are also subject to any decisions made by supervisors. As one interviewee at the local GOI level noted, “as administrators, we are not policy makers, we need to be considered as subordinates.” When staff is 18 CEGAH, FY 2019 Annual Report. 19 Polling Center: Public Satisfaction Survey on SP4N-LAPOR!, November 2019. 20 CEGAH, FY 2020 Q2 Report. 28 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report transferred to another government unit, the role is taken over by a new person. The double-burden of operator/administrators and frequent staff turnover affect complaint administration, as trainings have to be redone, and there is not enough time for skills to accumulate. In addition, SP4N LAPOR! administrators stated that their additional duties are not balanced with proper compensation to incentivize the extra work. The ET also found that differences in resources, capabilities, and characteristics of each district/city add to the complexity of increasing local capacity to implement SP4N LAPOR!, according to local government beneficiaries. In terms of financial resources, for example, Medan City is better positioned to support SP4N LAPOR! as compared to other districts and cities in North Sumatera that received the same support from CEGAH, as one local government beneficiary stated. However, not all districts or cities have sufficient budget for this, including the governments of Nias, Gunung Sitoli, and Deli Serdang, according to interviews. The lack of budget means that the district/city will have difficulty disseminating information on SP4N LAPOR! to their communities, because they could not organize events to promote the platform. In areas with challenging geographical conditions like Nias, promoting SP4N LAPOR! to communities would also require additional resources to allow local government representatives to regularly visit remote or less accessible sub-districts and villages. In addition to this issue, areas like Nias also have less reliable internet connectivity and network quality, which in turn disrupts access to and use of an online system like SP4N LAPOR!. This also creates programming inefficiencies without complementary support in other areas - e.g., strengthening Internet capabilities in such localities. Overall, the ET found that SP4N LAPOR! achieved significant progress on several fronts under CEGAH’s direction, particularly in key metrics, such as the number of users and complaints and the favorability of the system in the view of users and administrators. Additionally, evidence above demonstrates that CEGAH’s support of SP4N LAPOR! is driving progress against its sub-indicator 1.1.2 (component 2) because the system is beginning to produce changes in how government agencies in specific pilot districts are delivering services – and becoming more accountable and transparent in the process – to their citizens. SP4N LAPOR! is also complementing other component 2 efforts, such as external controls established at both the national and local levels, including locally created systems and ombudsman representatives; this comprehensive approach also increasingly drives progress toward CEGAH’s broader goal of building a culture of accountability within these pilot districts. However, the ET notes that for SP4N LAPOR! to serve as the national complaint management system for a country the size and population of Indonesia, support for the system must move beyond its current activities – e.g., demonstrating implementation progress in select pilot districts and raising awareness among stakeholders and potential users. To become the national system in reality would require a longer timeframe and more dedicated resources, given the complexities and external factors involved in implementing the system. Such factors, including differences in capacity, resources, the commitment of each local government, and the availability of the required infrastructure for accessing the system all need to be factored in for SP4N LAPOR! to be successful. Building a strong oversight mechanism that could provide quicker responses to complaints and information on how satisfied the public is with the resolution of those complaints could help improve the system’s effectiveness while increasing awareness and interest. Data-driven journalism. In 2017–2018, CEGAH collaborated with a journalism-strengthening CSO in Indonesia to promote data-driven journalism at the national and local levels. This initiative aimed to strengthen the capacity of media and civil society to exercise effective oversight by using data and turning it into high-quality investigative coverage. In the first year, they trained 80 journalists (including 20 women) from various media outlets (broadcast, print and online media outlet) in national and sub-national level, while the second-year activity focused on developing an e-learning platform on data-driven journalism. 29 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Through the platform, jurnalismedata.id, a total of 638 participants accessed the online course on data￾driven journalism. The evaluation found that even though this intervention was relatively small and ended in 2018, it yielded successes in promoting data-driven reporting among local journalists and demonstrated the important role of media and civil society in exercising oversight functions through the support of open government data. In general, informants from media told the ET that this training on data-driven journalism was beneficial and that they could apply the new knowledge in their work and respective media outlets. Besides improving their ability to produce stories that identified potential incidents of corruption and prompted public responses or investigative action (see below), another benefit of applying data-driven journalism skills in their media coverage was an enhanced ability to produce good data visualizations that could facilitate better understanding by the public of the issues being covered, as expressed by one of the training beneficiaries. The training also inspired a training participant to establish a local media outlet, called Zona Utara (zonautara.com), which focuses on applying data-driven journalism in covering and presenting investigative news. Zona Utara has already shown that even with limited funding, it could attract viewers with its data-driven approach to journalism. Based on the CSO’s records, Zona Utara has many viewers, even compared to other sources such as the Tribune, said a journalist. To promote and build the capacity of local journalists to better conduct data-driven journalism, CEGAH and its grantee used a unique approach combining training activities with a fellowship program in which the 15 strongest proposal submitters from the training received financial support to produce their proposed investigative articles. Training participants felt this was a success, and the ET found that some participants successfully wrote investigative articles revealing potential incidents of fraud and corruption. Some of these articles caught the attention of relevant agencies and resulted in further investigation of the case they had covered. A participating journalist spoke of reporting to uncover fraud regarding a locally administered levy. “We revealed this case,” the journalist stated. “The Ombudsman came in and saw the results of our writing. They conducted further investigation. Now, the local government does not use cash, but barcode payments to collect the levy.” Other respondents shared similar stories. One journalist wrote an investigative article regarding misuse of scholarship funds at DPRA (Aceh Provincial House of Representative), which prompted investigation by the Aceh Regional Police. Another journalist produced investigative articles exposing the practice of antimony mining in Kapuas Hulu, which led to the case finally being investigated by the Customs office, as it involved illegal exports to another country. While not all of this reporting led to further investigations or resolution of these allegations by the relevant law enforcement or authorized agencies, these examples demonstrate that open government data, combined with training in and support to conduct data-driven journalism, are important in strengthening media oversight capabilities. Discussion with the media and CEGAH’s grantee, however, revealed that the biggest challenge in data-driven journalism is the availability of quality government data. According to interviewed journalists, such data was either not available from key government institutions, not updated, or available but difficult to find, access, and process. Data availability is even more difficult at the local level, as media respondents said they had even less success accessing detailed local government data. Anti-bribery Management System (SNI ISO 37001). Beginning in 2019, CEGAH provided technical assistance to seven district courts in their pursuit to become certified as SNI ISO 37001-compliant organizations. SNI ISO 37001 is an internationally recognized standard that demonstrates that the certified organization has a viable anti-bribery management system, including an anti-bribery policy, financial and 30 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report commercial controls, a person to oversee anti-bribery compliance, and requirements to conduct training, risk assessments, and due diligence on individuals with whom it engages.21 CEGAH’s support to seven district courts, in cooperation with BAWAS MA (Supervisory Body of the Supreme Court), resulted in all of them receiving SNI ISO 37001 certification. As reported by CEGAH, beyond the certification itself, the seven district courts have made tangible changes. The courts have established policies and implemented adjustments in their court procedures to ensure that they minimize the risk of bribery. These included conducting bribery risk assessments, separating public and private court areas, and publishing clear information on court schedules, services, and costs.22 As a member of one of the pilot courts stated, “We make various SOPs . . . Every two hours, gifts from outside parties in any form, including promises and receipts are announced regularly by our court. . . . There are different public and private areas and others, we show it to the people who enter our court . . .” A representative from one of the judiciary organizations stated that achieving the SNI ISO 37001 certification would enable the courts to create a “trademark” related to the supervision and guidance necessary to implement an effective anti-bribery management system. The representative further explained, “Without the help of CEGAH for ISO 37001, the cost would be very large and too heavy to be borne by each government unit or Supreme Court as a whole.” They noted that the success and certification of the seven courts in implementing SNI ISO 37001 has raised the interest of other government units in implementing the system. The implementation of this anti-bribery management system is also in line with the GOI’s national “Integrity Zones” program (ZI menuju WBK/WBBM), which encourages and provides a process for government institutions and units to establish and maintain corruption-free environments and clean bureaucracies within their institutions.23 SNI ISO 37001 also offers added value to the seven courts, and potentially other courts and governing institutions in the future, by strengthening their knowledge of how to better implement an Indonesia￾based anti-bribery management system (such as the “Integrity Zones” program mentioned above), including having clear and detailed directions on how to implement such system. A representative of one of the pilot courts found SNI ISO 37001’s requirement of a bribery risk assessment, which they had never done before, to be beneficial: “The difference with our system with this ISO is: Identification of potential bribery opportunities, an indication of our authority. We are directed to identify potential bribery opportunities in each line of authority held by each position or executor in the District Court. From identification, we introduced how we had to find solutions to minimize or eliminate, by building a system.” Implementation of SNI ISO 37001 could also help the Supreme Court develop supervision and guidance to strengthen the integrity of other courts in the judiciary. Collaborative local government use of community satisfaction surveys to strengthen public health service delivery. CEGAH facilitated collaboration on public health service delivery among CSOs and local governments in North Sumatra and East Java through community satisfaction surveys (Survey Kepuasan Masyarakat/SKM). In both locations, CEGAH-supported CSO grantees worked with municipal governments and health offices, securing agreements to update, collaborate on, and administer the surveys. CEGAH incentivized local governments to enter these agreements through CSO grantee support and survey administration, which could reveal key data on the current quality of public health service delivery in those areas as well as gaps in delivery, in turn helping governments provide better services. In Medan, the grantee worked with the municipal government and public health centers to regularly administer the survey as well as hold regular forums between the health department and community 21 “ISO 37001 Anti-Bribery Management Systems,” ISO, 2016, https://www.iso.org/iso-37001-anti-bribery￾management.html. 22 CEGAH, FY 2019 Annual Report. 23 Zona Integritas Menuju WBK/WBBM (Integrity Zone towards Free Corruption Area and Clean Bureaucracy) is stipulated under the regulation of Ministry of Bureaucratic and Administrative Reform, PermenPAN-RB, No. 60, 2012. 31 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report representatives to gather input to improve services. In Jember, the grantee worked with the local government and the University of Jember in East Java to establish the FORASKER forum, which helped secure an agreement to conduct the SKM. These interventions supported CEGAH’s Component 3 in strengthening CSOs’ abilities to hold local government accountable by increasing these municipalities’ adoption of CSO input toward more transparent practices in the delivery of public services. According to KIIs in both locations, this CSO support resulted in more effective feedback from the strengthened, more frequently administered SKMs, which in turn led to improvements in community health facilities. In Jember, a local GOI beneficiary stated, “So far, the results of the survey can be used properly. For example, there is a separate location for infection control, separated between sick and healthy patients and PKM [Community Health Centers] more child friendly.” In Medan, an interviewee noted that as a result of the survey, “service officers are more friendly [toward people] with disabilities. Previously, there was a reluctance to seek treatment at the Puskesmas (Public Health Center), given the difficulty in accessing the place. But the people at the Puskesmas have started to make good changes.” Similarly, following improvements in service quality, the government in Jember incorporated the SKM into its budget planning process for regional development. CONCLUSIONS The Direktori Putusan, SP4N LAPOR!, data-driven journalism, and anti-bribery management system (SNI ISO 37001) demonstrated promising results in the fight against corruption. These interventions were successful because they contributed in practical ways to CEGAH’s core aims. Direktori Putusan is increasing the effectiveness of the justice sector by serving as a transparent, accessible, and increasingly sophisticated research tool used not only by the courts but more and more by the public. SNI ISO 37001 is strengthening the justice sector by establishing new anti-bribery practices that are aiding the pilot courts in dispensing justice in speedier and more impartial ways and are serving as a model for other courts to follow in the future. SP4N LAPOR! is rapidly expanding and systematizing the pilot agencies’ capacity to implement the system and strengthening the ability of Indonesian citizens to hold their governing institutions accountable and obtain the public services they are entitled to. CEGAH is also enabling civil society organizations and the media to provide oversight through its data-driven journalism. As noted above, SP4N LAPOR!’s success is tempered by the fact that there is still considerable work to be done to ensure its effective implementation and performance as a national complaint handling system. Specifically, efforts to raise awareness and promote SP4N LAPOR! to all local government and agencies across Indonesia and the wider public must continue. Additionally, support to the Indonesian government at both the national and local levels is needed to establish a robust monitoring and oversight mechanism as SP4N LAPOR! is implemented more widely. EQ2B: LEAST SUCCESSFUL INTERVENTIONS Several factors, including complexity, barriers to sustainability, and the possibility of misuse are underlying reasons why some interventions have been less successful. Countering Violent Extremism interventions. The ET found that CVE interventions under CEGAH yielded a mixed record in terms of their effectiveness. Overall, most CEGAH government and judiciary partners did not view these as an ideal “fit.” Eleven respondents from GOI beneficiary organizations stated that they did not identify a strong link between anti-corruption and CVE and that CVE was not part of their mandate: “In the context of extremism and radicalism, I don't think anti-corruption meets with the extremism issues. That's more to BNPT (Badan Nasional Penanggulangan Terorisme/National Counterterrorism Agency),” said one of the respondents. This view was supported by the evaluation survey of CEGAH beneficiaries: when asked to choose between 14 CEGAH activity components that had been the most important in 32 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report contributing to their organization’s ability to fight corruption (they could select up to five of these components), none of the 26 respondents selected “the ability to understand and/or identify radicalism within your organization or sector.” This view was not universally shared, as five interviewed government and judiciary beneficiaries welcomed such programming to strengthen their capacity to identify and prevent radicalization in their organizations. In its CVE-related research initiatives, CEGAH studied sentencing trends in terrorism trials, the potential leakage of government funds to violent extremist actors, procurement of food in prisons, the use of corruption as violent extremist propaganda, radicalization at state universities, and the impact of discriminatory bylaws. CEGAH identified some notable trends in these studies (e.g., Muhammadiyah University’s survey of 600 students found that 11.5 percent of students surveyed supported the establishment of an Islamic caliphate in Indonesia24), but there were fewer conclusive findings than expected around the relationship between corruption and CVE.25 In the study on the potential leakage of government funds to violent extremist actors, for example, a key finding was that most violent extremist actors, with respect to their financing sources, tended to stay far from government and instead were more likely to obtain funds from sympathetic businesspeople and international sources, “Linkages between CVE and corruption were not quite as strong as we had hoped,” said one implementing partner staff member in reference to that study, However, an implementing partner informant noted that the aforementioned studies and others, such as a perceptions survey, would be useful in informing the design of later CVE programs. In terms of CEGAH’s direct engagement with GOI and judiciary entities on CVE, an example of a less successful intervention was the development and implementation of the Tolerance and Pluralism Awareness/Attitude Test (TePAT) tool. CEGAH’s grantee developed the TePAT to identify people at risk of holding intolerant and extremist views as part of the civil servant recruitment process. The test was adopted and used to screen more than 33,000 applicants for civil servant positions at six ministries and agencies in 2019.26 Both CEGAH’s implementing partner and grantee viewed the tool as successful, based on its widespread application in 2019 and the interest demonstrated by agencies in addition to these six (such as the Ministry of Finance). However, evidence of misuse of the tool emerged soon after, according to a key informant familiar with the process. Specifically, while TePAT’s results were primarily being used to gather baseline data on and prepare mentoring plans to help guide new recruits, some agencies were using it to disqualify candidates without considering other tools first (such as observation, interviews, or mentoring) or the human rights implications. CEGAH had recognized that this might become an issue for this task, and took steps to attempt to mitigate the threat of TePAT being misused, including employing human rights experts and consulting external CVE specialists to provide input in the development of this task. However, when evidence emerged that it was being used as the sole reason to disqualify candidates, CEGAH/USAID eventually withdrew their support for TePAT. “We saw that the government was not interested in employing the other tools to take into account the safeguards and pay attention to human rights,” the informant said. “I felt good about the use for mentoring, but I don’t feel good about its use to disqualify people.” Relatedly, a grantee noted “that there was a concern from USAID that people who are not pro￾government . . . will be removed.” 24 CEGAH, FY 2019 Annual Report. 25 The ET notes that this relative lack of identified linkages between corruption and CVE runs contrary to some USAID studies and other respected studies (e.g., “Consequences of Inequality in Indonesia: Extremism, Corruption and Economic Costs,” July 2017, and “Corruption and Local Democratization in Indonesia: The Role of Islamic Parties,” March 2011, among others). 26 Ministry of Law and Human Rights (KemenkumHAM), Ministry of Religious Affairs (Kemenag), National Defense Agency (Lemhanas), National Standardization Agency (BSN), Ministry of Villages, Disadvantaged Regions, and Transmigration (KemendesPDTT), and Ministry of Manpower (Kemenaker). 33 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Despite having to withdraw its support to this tool, USAID CEGAH reported that at least seven ministries have expressed interest in continuing to use TePAT for their civil servant recruitment process.27 While TePAT’s grantee considered this a successful intervention in terms of the level of adoption and interest in using the tool, the ET found this to be a questionable conclusion, based on the human rights concerns raised and the fact that the effectiveness of the tool had not as yet been tested. Finally, two key informants acknowledged that conducting CVE interventions under CEGAH was at least in part influenced by timing. "The goal was to bridge the gap between political pressure to have a CVE program and not having a mechanism available at the time," said one of the informants. Later standalone CVE ventures such as MSI’s Harmoni activity could then build on CEGAH’s CVE studies and tools.28 “The CEGAH team did a good job in identifying relevant issues within their scope or within the CVE scope to focus on and inform later CVE programming,” the informant stated. SOLUSI. CEGAH created SOLUSI as a consultation channel to provide the private sector with free technical assistance to enhance their business integrity practices and minimize risks of corruption and fraud, with a focus on infrastructure and the health sector. Despite showing promising potential in its trial phase (86.7 percent of participants agreed that it would help private sector firms increase their compliance), this consultation channel no longer receives support, nor is it being used by its target, the private sector. There is very limited information regarding SOLUSI, as CEGAH only reported about SOLUSI in its annual report in 2017 and provided no more information in subsequent reports. However, interviewees explained several reasons behind SOLUSI’s lack of success. First, key government institutions did not commit to supporting it. Second, small and medium enterprise (SME) actors, who are the main target of SOLUSI, tend to perceive corruption as an acceptable practice. “There was a platform called SOLUSI, but it did not work because lack of resources and required large costs. There was also no support from the KPK to run SOLUSI...we disseminated it to the business sector, but the issue of anti-corruption is not generally discussed by many people. Many felt apathetic and felt that it's okay to do bribery as long as they don't get caught. So that, SOLUSI was not widely used by UKM/SME people due to the unavailability of UKM/SME people to commit,” said one grantee. This lack of KPK support for this intervention was partly a question of resources (human, financial and time); SOLUSI, for example, was seen as costly, and KPK might not have been willing to expend the necessary resources to fund it, one interviewee noted. Additionally, KPK appeared to be more interested in different approaches to engaging with the private sector, such as through the Regulation and Guideline on Corporate Criminal Liability (see next paragraph), and through state-owned enterprises (please see “Changes in Government Leadership, Personnel, and Policies” under EQ2c below). SOLUSI did align with other CEGAH-supported initiatives, such as the KPK and Supreme Court enactment of the Regulation and Guideline on Corporate Criminal Liability in 2016, which for the first time permitted the judiciary to hold Indonesian corporations accountable for corrupt practices. The expectation was that the regulation/guideline would strengthen the environment for related private sector-focused interventions like SOLUSI. However, the target of SOLUSI was SMEs, with the rationale that they have limited resources for compliance and therefore require more assistance. Unfortunately, while SOLUSI initially showed potential, it ran only for about a year after its trial phase. Grantees and beneficiary interviewees from relevant organizations have acknowledged that initiatives such as the Corporate Criminal Liability Regulation/Guideline as the preferred and more effective programming vehicle by the KPK, Supreme Court, and other beneficiaries (see findings under EQ1 – e.g., under “CEGAH’s network fostered productive interagency collaboration…” – and EQ3). 27 CEGAH, FY 2020 Q2 Report. 28 MSI, “Bolstering Tolerance and Resilience in Indonesia,” accessed July 2020, https://msiworldwide.com/projects/bolstering-tolerance-and-resilience-indonesia. 34 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report CONCLUSIONS CVE interventions were among the less successful interventions undertaken by CEGAH because activities such as TePAT showed the potential for misuse and human rights abuses when applied unilaterally instead of in tandem with a range of CVE strategies, and because CVE interventions did not attract sufficient buy￾in from a majority of CEGAH’s government and judiciary partners, who did not see it as part of their mandate. There was also little interest from the private sector in the SOLUSI activity, which did not successfully work in parallel with the Supreme Court’s Regulation and Guideline on Corporate Criminal Liability. EQ2C: WHAT WERE THE EXTERNAL FACTORS THAT AFFECTED THIS PROCESS? Effects of COVID-19. This evaluation was conducted only several months after the pandemic hit Indonesia (and the rest of the world), but COVID-19 has already created major logistical and budgetary issues in Indonesia for CEGAH activities, both at the regional and local levels during that time. The key consequences have been restrictions on travel and public gatherings, which have limited or forced the cancellation of planned CEGAH activities, including trainings, workshops, press conferences, in-person audits or exams, and other meetings. One grantee stated, “COVID-19 does not allow us to do face-to-face meetings. We must do virtual meetings as much as possible; however, if you want to reach the higher level, like The Supreme Court, it would be appropriate if we do it with non-virtual, face-to-face.” In addition, COVID-19 has affected supported institutions’ budgets and their anti-corruption work. GOI/judiciary institutions have been forced to “repurpose” part of their budget to the Ministry of Finance for the national COVID-19 response. A government beneficiary, for example, was required to divert IDR 600 billion (USD $41 million) to the Ministry of Finance and reallocate IDR 700 million ($48,000 USD) from its remaining budget to handle COVID-19 within its own department (representatives from three other ministries said they had seen their budgets similarly diverted, to the detriment of their anti￾corruption activities). Additionally, this has resulted in reductions or cancellations of trainings. For example, a representative of one agency’s training unit stated that they will not be able to conduct any training activities this year. A judiciary beneficiary said that 96 judges rather, than the usual 120, would be participating in the online version of the Supreme Court’s anti-corruption certification training this year, because of repurposing. Relatedly, a GOI respondent stated that the pandemic-related reallocation of their operational budget had negatively impacted the quality of the bandwidth used for the agency’s e￾learning program. The pandemic has also forced the delay and cancellation of trainings with beneficiaries that must be done in person, including trainings on using the Direktori Putusan and on aspects of improving delivery of public health services in local areas (e.g., monitoring compliance with minimum service standards), according to two CEGAH grantees. One grantee stated, “We must delay this offline course because it cannot be done by online platform.” The CSO informant also noted that “we usually have to do these [public health services] in person because not all people have smart phones.” CEGAH has been proactive in responding to the COVID-19 situation as well as the new COVID-19 law, which several CEGAH stakeholders, CEGAH staff, grantees, and GOI beneficiaries see as providing opportunities for corruption. Some interviewees, for example, expressed concerns about the new COVID-19 Law (“The Law on State Financial Policy and Financial System Stability for the Handling of the COVID-19 Pandemic”) over provisions that allow the government to freely reallocate funds from various sources, including education endowment funding, public hospital funding, and other social safety net funds, 35 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report in response to COVID-19 and related threats to the economy.29 Prominent examples of CEGAH’s responses to COVID-19 include the following: • A grantee created a feature within the SP4N LAPOR! system to allow quicker responses to complaints on COVID-19, which automatically routes the complaint to the appropriate agencies for immediate follow-up. However, KemenPAN-RB rejected this request to add “social aid” to SP4N LAPOR!, since they predicted many complaints about social aid distribution, according to a grantee. • CEGAH provided support to ICW to conduct monitoring of COVID-19-related procurement and distribution of social safety net policies and practices in key areas, conducted a legal review of these policies/practices, and identified recommendations on how to do such procurement legally during the pandemic. • The IG of the Ministry of Health (MOH) has received specialized technical training on emergency procurement corruption risks, rapid assessment of these risks, and dissemination of the new regulation on emergency procurement control. The aim of these new programs is to ensure that the auditors of the IG of Kemenkes and SPIs throughout Indonesia are prepared for the upcoming emergency procurement of medical supplies.30 During the pandemic, CEGAH beneficiaries demonstrated increased usage of CEGAH’s online and e￾learning platforms, tools, and modules, as well as interest in maintaining and potentially expanding the use of these resources. Noting the current pandemic-related restrictions on in-person gatherings and travel, CEGAH staff have pointed out that it has been advocating for and promoting the usage of online and e￾learning modules, platforms, and tools well before COVID-19, such as the Akademi Antikorupsi (Anti￾Corruption Academy) modules and the Direktori Putusan. The ET agrees that these and other interventions demonstrate CEGAH’s push for greater e-learning and online capabilities for its beneficiaries, although it notes that the achievements of each of these interventions must be assessed individually (as discussed in the relevant sections under EQ1 and EQ3 of this report). There has been an increased use of SP4N LAPOR! for questions and complaints about social aid, as mentioned by a GOI beneficiary, “if there is social assistance that is not in accordance with expectations, or there are other issues related to pandemic, it can be delivered to the government through SP4N LAPOR!” CEGAH support has enabled utilization of online monitoring processes using a risk-based approach, the Remote Joint Audit through Financial Intelligence Unit (PPATK). As mentioned by one GOI beneficiary, “Even though remote audits are not as effective as physical audits, during this pandemic condition it needs to be done this way. Virtual data audits, virtual training, and verification are conducted through online platforms such as Zoom.” Beneficiaries are more enthusiastic about online/e-learning activities (such as ICW's Anti￾corruption Academy), as illustrated by its increasing number of users (see EQ3 below). According to a judiciary informant, a partner is currently working with the Supreme Court to develop an online hearing system for criminal cases. There has also been an increased use of Direktori Putusan to transfer court documents, as noted under EQ2 above. In the context of COVID-19, beneficiaries are seeing new opportunities to promote and utilize Direktori Putusan across Indonesia. COVID-19 makes it difficult for courts and relevant stakeholders to send and share court documents, as the postal service is not functioning. Thus, Direktori Putusan can help with file transfers. According to three interviewees from the judiciary and grantees, Direktori Putusan version 3 is currently the only system ready to accommodate this. “One of the features in the Direktori 29 Also see: Asian Forum for Human Rights and Development, “Indonesia: Law on COVID-19 must not undermine the country’s commitments to anti-corruption and democracy,” May 15, 2020, https://www.forum-asia.org/?p=31733. 30 CEGAH, FY 2020 Q2 Report. 36 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Putusan, in addition to transparency instruments, is for data communication to send legal appeal data from 800 courts to the Supreme Court,” said a judiciary informant. Local Internet Capacity. Another external factor affecting CEGAH’s implementation, as noted by eight beneficiaries/grantees, is poor internet coverage in some areas of Indonesia. Four local stakeholders in North Sumatra and other areas, for example, stated that poor Internet was an obstacle to utilization of SP4N LAPOR! Relatedly, local budgets have proven insufficient to providing the quality and reach of Internet access required in each region, as noted by a local GOI beneficiary: “Our complaint is the internet network, so people who use lapor.go.id find it very limited and difficult. If it is not supported by a good network, it will be disrupted. Our local budget is limited to provide internet access in each region.” Internet capacity is also an issue for judges in remote locations trying to participate in Supreme Court trainings via Zoom, as mentioned by one judiciary informant, “There are still many courts or districts that are weak in terms of signals to be able to communicate (e.g. North Maluku, East Nusa Tenggara, Papua), sometimes it’s stalled when they talk, and communication is interrupted, so it must be delayed.” The quality of the electrical power supply in several of these areas is inconsistent as well, he added. Changes in Government Leadership, Personnel, and Policies. According to the ET’s online survey (Figure 3), 60 percent of respondents stated that new laws (including the KPK Statute) impacted the success of their CEGAH-supported work either “significantly” (36 percent) or “moderately” (24 percent). FIGURE 3. TO WHAT EXTENT HAVE ANY EXTERNAL FACTORS IMPACTED THE SUCCESS OF YOUR CEGAH-SUPPORTED WORK? The consensus view among 27 interviewees was that KPK will be significantly less effective than it was before because of the new KPK Law. The new statute, the Corruption Eradication Commission Law (UU KPK), created a new supervisory council (Dewan Pengawas) to oversee the KPK’s policy-making and operational activities. For example, the council’s approval is now required for warrants, a key tool in the Agency’s enforcement work that enables them to seal investigation sites, conduct raids, and obtain wiretaps. UU KPK also now requires that all employees be civil servants. As one implementing partner interviewee noted, “The new law of KPK was definitely the major external factor that affected CEGAH implementation. The new law changed the supervisory boards, where internally they are still struggling under it.” According to the grantee, the politicians and other decisionmakers in this country might suddenly change 8% 8% 8% 13% 10% 32% 29% 46% 38% 20% 24% 38% 29% 29% 60% 36% 25% 17% 21% 10% New laws, such as the KPK Statute Recent political developments Recent economic developments Recent health developments Other Not at all To a small extent Moderately Significantly 37 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report the policy affecting the anticorruption efforts being implemented thus far. For example, while the KPK has used the Regulation and Guideline on Corporate Criminal Liability to prosecute six cases against corporations accused of participating in bribery or corrupt practices, the authority of KPK’s new supervisory council under the 2019 law may have a "chilling effect” on the agency pursuing these and other enforcement actions as robustly in the future, according to a GOI beneficiary. KPK's role and focus may therefore shift more to prevention (versus enforcement). While this change will likely not have the same direct effect on CEGAH programming as other changes under UU KPK, it may shift the sectoral emphasis of the support KPK requests from CEGAH, such as potentially less focus on building the investigative and prosecution skills of KPK staff, one GOI beneficiary told the ET. “The previous Chairman had the motivation and interest to develop our investigation and prosecution practices,” the beneficiary said. “Therefore, we tried to address the big corporation cases, money laundering, and other big cases. The focus now has changed.” Additionally, UU KPK now requires that CEGAH obtain prior approval from KPK's Director of Prevention for every activity proposal in addition to the Secretary General of KPK’s program implementation unit, with whom they have frequently engaged and enjoyed ”easier communications” in the past, according to two interviewees. They added that this has led to the rejection of several proposed CEGAH interventions that they believe would have been approved had this second layer of permission not been required. ”[S]ome activities that we offer them, which are seen as critical interventions by the [KPK] technical staff and have already been approved by the Secretary General, were disapproved by the Director of Prevention,” said one of the interviewees. This has included rejections of several potential activities related to COVID-19 that the program implementation unit voiced support for, including guidance CEGAH was developing on preventing corruption during a crisis such as the pandemic (”once we talked with the Deputy of Prevention, he mentioned that this is not the mandate of the KPK...so it just stopped right there,” the interviewee stated). This also resulted in the rejection of a proposed intervention with the Ministry of State-Owned Enterprises, which would have enabled them to make progress toward greater engagement with the private sector. Additionally, CEGAH has had to contend with changes in leadership or key staff within the Indonesian government, particularly at the local level, according to 10 grantees and beneficiaries. This has been principally due to staff rotating into new positions or leaving their current positions outright, and can result in grantees facing changing levels of commitment from local governments or losing implementation time by having to retrain new staff, they added. This affected the pace and efficiency of implementation of CEGAH interventions to strengthen local government accountability in the delivery of public health services in Medan, Jember and nearby localities. “One of the obstacles we faced was the commitment of local governments that often changes due to the change of people,” one grantee told the ET. “We had done the training but apparently there was a change in positions. The impact was that we needed to form a team again, and this process is long because it must be approved by the regional/district head.” CONCLUSIONS As a result of COVID-19, Fewer GOI resources are available to conduct anti-corruption activities, and delayed and cancelled trainings are creating at least a short-term freeze on many current and a few new interventions. However, CEGAH-built capacity in its supported government and judiciary institutions is still present, and CEGAH support has enabled shifting or continuing many beneficiary activities online. As to the new COVID-19 law, CEGAH has attempted to take several preemptive steps that will seek to publicize issues with the new law and expand capabilities of online platforms and beneficiary staff to address COVID-19 related issues. The preemptive steps taken by CEGAH team include the additional feature on COVID-19 related complaints in SP4N LAPOR!, commissioning a study of the new law and 38 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report associated regulations on the Financial Policy and System Stability during the pandemic response, and specialized training for the IG of MOH on emergency procurement control and corruption risk rapid assessment. The new KPK law’s supervisory board’s authority under the 2019 law may have a “chilling effect” on the agency pursuing enforcement actions as robustly as it has in the past. KPK's focus may shift more to prevention (versus enforcement) activities. While this change will likely not have the same direct effect on CEGAH programming as other changes under UU KPK, it may shift the sectoral emphasis of the support KPK requests from CEGAH, such as potentially less focus on building the investigative and prosecution skills of KPK staff. EQ3: WHICH INTERVENTIONS AND THEIR OUTCOMES HAVE THE GREATEST POTENTIAL FOR SUSTAINABILITY AFTER THE END OF THE PROGRAM? More than 75 percent of GOI/judiciary beneficiary survey respondents (20 out of 26) stated that they believed their work would continue after CEGAH ends (Figure 4), as did 38 grantees and beneficiaries interviewed by the ET. Of this second group, 30 added that they believed this work would continue regardless of whether they received additional funding, a point that was echoed by several USAID and IP staff as well. They noted that CEGAH’s value added was in its guidance and expertise, its network of government and CSO contacts, and its ability to develop and launch trainings, events and new/improved online tools that GOI/judiciary beneficiaries could then take over and fund themselves. “Our money is not a significant factor in their decisions, and they’re not really dependent on our funds,” said one interviewee from the implementing partner. “These are their programs.” Additionally, CEGAH staff noted that they attempt to tailor the design of their tools at the outset, such as e-learning systems, based on the ability of each supported GOI/judiciary institution to sustain them from both a budget and human resource perspective. However, while the ET found that aligning the design of an intervention with beneficiary capabilities/resources is indeed a key criteria for sustainability, it also identified two other criteria that appear to be present to some degree in those CEGAH interventions that appear to have the greatest potential for sustainability: political will within the organization to continue the program, and evidence of its effectiveness (or at least the promise of that effectiveness – e.g., utility within the organization). Some key examples of interventions identified by the ET as meeting the three criteria mentioned above included the following: • Supreme Court’s anti-corruption training/certification: Judges are required to attend this training, which the Supreme Court has been conducting annually since 2006 and been budgeting for since 2009. But the training was not standardized, so the speakers, content and format were different each year. Beginning in 2018, CEGAH helped the Court’s Education and Training Center create 11 standardized modules for the training, which they then put online, “so that before coming to the training venue, they will know and understand the materials that will be discussed in the training,” said one judiciary informant. This has improved the efficiency and quality of the training, since participants arrive prepared to discuss the material, and instructors can progress to discussions of practical applications of the law. The existence of these online, standardized modules has become even more significant during the pandemic, as they helped the Court to conduct the training virtually via Zoom, the FIGURE 4. DO YOU BELIEVE YOUR ACTIVITY WILL BE ABLE TO CONTINUE ONCE CEGAH’S SUPPORT ENDS? 77% 4% 19% Yes No Don't Know 39 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report beneficiary said. A CEGAH-designed assessment for participants in the training will be administered in the near future, but in the meantime, the training remains an ongoing core element of Court learning in the eyes of its leadership, three judiciary grantees and beneficiaries told the ET. • Supreme Court draft prosecution and sentencing guidelines: The Supreme Court also appears on track to finalize and adopt its draft prosecution and sentencing guidelines for corruption cases through SC regulation as early as July 2020. CEGAH has supported development of these guidelines, the impetus for which was the wide range of inconsistent sentences handed out to those convicted of corruption crimes in courts across Indonesia. While they have not yet been implemented, the Court strongly supports the guidelines, and with CEGAH has held approximately 20 meetings over an extended period with other institutions (including KPK, AGO, and the police) to build support and overcome disagreements. “The fact that they agreed to develop these sentencing guidelines is a success,” said one grantee. In the eyes of the Court, the importance of implementing and sustaining these guidelines beyond the end of CEGAH is not just in ensuring that they are applied fairly and uniformly, one GOI beneficiary stated, but also in serving as a successful first step in the Court generating guidelines to standardize sentencing for other areas of law. “In terms of cases that occur in Indonesia in the criminal field, 60% of those that come into the Supreme Court involve narcotics,” the judiciary said. “Second is child protection, then corruption. The community wants fair, consistent sentencing in the same types of cases, so in my opinion, it’s more urgent.” CEGAH’s additional assistance is therefore needed to support both implementation of the corruption sentencing guidelines as well as those in other areas. • CEGAH-supported online/e-learning platforms and tools: Several CEGAH online/e-learning tools are generating greater interest – and potential for sustainability – among stakeholders due not only to their efficiency – e.g., the ability to reach anyone with an connection in a country the size and geographic spread of Indonesia – but also to the ability to use (or continue using) these during the pandemic. o Indonesia Corruption Watch’s (ICW) Akademi Antikorupsi e-learning modules. ICW’s system started as a set of online courses for activists that distilled the NGO’s knowledge in conducting oversight, investigations, and advocacy, then broadened to different sub-groups, particularly students. CEGAH acknowledges that there were some initial usability issues after its launch (April 2018) with the "hybrid nature of the system" (e.g., number of essays) that resulted in low completion rates 31 Consultant recommendations appear to have largely addressed these issues. ICW now has more than 4,100 active users, has signed MOUs with 11 tertiary education institutions to integrate and implement Akademi materials into their curricula, and several universities, including the State Finance Polytechnic Institute of STAN (PKN-STAN) and the Malang Politeknik Health Institute, made it a required portion of students’ coursework even before adopting a formal agreement with ICW.32 These MOUs with the universities will also “reduce the effort ICW needs to manage the administrative arrangements with students,” said one grantee. “Because it will be managed by the university – and it is still free.” Separately, ICW launched an anti-corruption school in March 2020 for all members and secretariat officials of the General Election Commission (KPU) of DKI Jakarta, which will require completion of the Akademi modules and will include four offline classes as well. Additionally, ICW also obtained commitment from the West Java Provincial Election Supervisory Agency (BAWASLU) to use the e-learning platform to train BAWASLU staff on anti-corruption.33 It also conducted a coaching class for three staff of 31 CEGAH FY 2019 Annual Report. 32 CEGAH FY 2020 Q2 Report. 33 CEGAH Q1 FY 2020 Quarterly Report. 40 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Dompet Dhuafa, an Islamic philanthropy organization in February 2020, to train them as facilitators in a Dompet Dhuafa “Philantegrity” event, which will use the Anti-Corruption Academy as its tool to train others.34 The ongoing and expanding interest in the Akademi modules among multiple audiences makes it highly likely that they will continue beyond CCE. o Direktori Putusan: As indicated above under sub-section EQ2a, Direktori Putusan has proven a popular research tool with both the judiciary and the Indonesian public, and its functionality (e.g., search engine, scalability) has only grown stronger in its evolution from version 2.0 to 3.0. It is funded annually through the state/APBN budget and managed by the MA Kepaniteraan (Registrar). While the Supreme Court will continue funding it, and has shown significant interest in further expanding its utilization/functionality in the future, according to several interviewed beneficiaries, there are several issues that could impact its sustainability if not addressed robustly: First, the quality of data entered into the system needs to be monitored. Both operators & judges need to be trained to ensure accuracy in data input, including the introduction of standard and new processes, one judiciary beneficiary told the ET. For example, many findings entered into the system have been found to have been misclassified, such as decency crimes filed as crimes against state security. “This indicates that there is no quality control,” the judiciary informant added. “Awareness of court leaders about quality control must be increased.” Second, the current budget for maintaining the Direktori Putusan is limited, especially due to recent budget repurposing requirements in response to COVID-19, while the system’s data traffic is growing exponentially. “There are six million cases in a year in Indonesia,” he said. “10 MB for a case is no longer sufficient; we may need 100 MB per case. We need a server that can manage big data.” Current budget levels may affect the procurement of sufficient servers for Direktori Putusan in the future, said one judiciary informant. • The guidelines on investigation and prosecution of Corporate Criminal Liability: Several GOI and judiciary beneficiaries stated that these guidelines, enacted as an internal regulation by the Supreme Court in late 2016, are likely to continue after CEGAH ends. As noted under EQ1, the regulation for the first time enabled the investigation and punishment of corporate involvement in bribery and corruption cases and has since been used to help KPK investigate and prosecute six high￾profile cases, one beneficiary told the ET. It has also been used by the AGO (in at least five cases) and the Indonesian National Police (INP) in their investigations as well. Additionally, the regulation also forced companies to improve their capacity in compliance and risk management, which enabled them to mount legal defenses against corporate corruption charges but also made them more vigilant regarding corruption within their companies. “This regulation will be sustained because it has already been enacted and has been brought up publicly by both NGOs and academics to the [KPK] Chairman about the corporate cases – they force the KPK to still conduct investigation on that,” said one GOI beneficiary. • SP4N LAPOR!: GOI beneficiaries at both the national and local levels spoke of the importance of sustaining Indonesia’s National Complaint Handling System, and of their “internalization” of SP4N LAPOR! as it has been implemented within its 33 pilot ministries and local governments “SP4N LAPOR! funding will be continued because this is provided for on the roadmap with funding from the state budget,” said one GOI beneficiary. KemenPAN-RB, which oversees SP4N LAPOR! together with ORI and KSP, began gradually assuming control of SP4N LAPOR! and now maintains its staffing and systems itself, according to one interviewee, adding that KemenPAN-RB has also taken on building up the capacity of newly appointed SP4N LAPOR! administrators among the 27 designated pilot local governments by paying for all of the costs associated with their training – e.g., flights/accommodations – except for the training itself, which CEGAH provides. As interest in implementing SP4N LAPOR! locally has 34 CEGAH Q2 FY 2020 Quarterly Report 41 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report grown (see above sub-section on SP4N LAPOR! Under EQ2a), several local government beneficiaries from the pilot areas have also indicated that they have or were ready to provide funding to sustain the program through their budgets. • Alternatives possible to sustain benefits of pilot district courts’ SNI ISO 37001 Certification: From its interviews, four interviewees told the ET that it will be difficult for the seven district courts that recently earned the SNI ISO 37001 anti-bribery management system certification to replicate this achievement (or for other courts to certify for the first time) due to its high cost. These four all voiced this issue, which poses a significant barrier for promoting the certification to other courts or agencies or even maintaining the certification at the pilot courts once it expires. However, some alternate solutions have been discussed among CEGAH and relevant grantees that may enable the courts to sustain the benefits of the SNI ISO 37001 certification, and for other courts to potentially gain from it as well. One would be to adopt the anti-bribery quality management system into a national mechanism to be applied incrementally, an idea that has been discussed with the country’s National Certification Body (BSN). As noted by one grantee, similar quality management system approaches have been adopted by the Supreme Court and could be applied in this situation to strengthen the possibility that the standards achieved under the SNI ISO 37001 certification would be sustained. Similar discussions have taken place regarding potentially aligning the anti-bribery management system’s standards with those employed in the KemenPAN-RB’s Integrity Zones, which could also help to reduce the administrative burden to maintaining or meeting these standards within the current or other district courts. The grantee also pointed out that CEGAH would need to coordinate these interventions with BADILUM (Directorate General for General Courts) in the Supreme Court (i.e., not just BAWAS MA, the Supreme Court’s supervisory body), to ensure implementation of this standard and practices to the general courts in the future, since BADILUM is responsible for assisting general courts. CONCLUSIONS Key factors in the ability of CEGAH activities to be sustainable after this activity ends are aligning the design of an intervention with beneficiary capabilities/resources, as are an organization’s political will to continue the program, and some evidence of its effectiveness or usefulness within the organization. Beyond these, however, are certain unknowns, including some of the external factors outlined above, that may also impact the longer-term sustainability of these interventions. For example, while the aforementioned tools and services are likely to continue after CEGAH ends, the new KPK Law, UU KPK, has introduced a degree of uncertainty within these processes. While KPK is likely to attempt investigations of corporate corruption under the guidelines/regulation on investigation and prosecution of Corporate Criminal Liability in the future, the statute’s new requirement that KPK must obtain the new supervisory council’s approval for warrants to conduct key tasks in their enforcement work may significantly limit how effectively KPK can actually investigate these cases. Separately, while the Supreme Court is keen to see its sentencing guidelines succeed, as they hold the potential to be a first-of-their-kind, country-wide standard for setting punishments for corruption crimes, the real test of their sustainability will be assessing how closely and uniformly they are followed by the courts once they are officially approved as a regulation. Likewise, the Direktori Putusan is a popular, widely-used research tool by both the courts and the public alike, but in the future, the sustainability of this flagship system will be tied to the ability to secure the government funding necessary to serve the system’s exponentially growing data storage and server needs. 42 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report LESSONS LEARNED EQ4: WHAT LESSONS LEARNED IN STRATEGY AND APPROACH CAN AND SHOULD BE APPLIED TO FUTURE PROGRAMMING? Ensure strategic coordination with stakeholder counterparts to accelerate and implement interventions more smoothly. One issue raised by both grantees and beneficiaries was how to better address coordination issues that arose between stakeholders as part of CEGAH interventions.35 Some grantees, for example, thought that earlier planning between the KPK and AGO could have improved expectations and prevented the dropping of RT 7 (“Support effective coordination between the KPK, AGO, and other law enforcement entities to strengthen corruption indictments”). “Maybe at the beginning we were supposed to have deep discussion between KPK and AGO,” one grantee told the ET with respect to the difficult “institutional ego” encountered among some staff. One suggestion would be a methodical, detailed, and precisely worded MoU carefully delineating the respective roles of the KPK (as investigators), the AGO (as prosecutors), and the Supreme Court. This could be hammered out during the planning phase and serve as a tool the program implementer can refer to when issues arise during task implementation. Similarly, several grantees also suggested that more strategic engagement with stronger GOI/judiciary partners could not only help CEGAH achieve better results but could also incentivize other partners to involve themselves more proactively in CEGAH-supported activities. “We need to focus on institution that will achieve much better results—then, other agencies will come along and participate,” said one implementing partner staff member. For example, CEGAH could engage more in future work with the Supreme Court, which the grantee considered a strong CEGAH partner; at the same time, “the court can also drag prosecutors to come along and get involved with them,” said an implementing partner KI. The Supreme Court could drive change, for example on its development of criminal proceedings during the pandemic, which could motivate the AGO to participate and make changes within their office as well, the beneficiary concluded. Such a process would be worth exploring up to a point, another grantee noted, adding that leaving specific GOI agencies out of CEGAH-supported activities comes with risks: “In terms of coordination, we have to choose which activities can bring them together in the same program . . . It’s easier just to work with MA [Supreme Court], but . . . if you don’t involve AGO, KPK, it will create problems in the end.” In general, grantees stated that a significant factor in effectively working with government or judiciary beneficiaries was establishing good communication with its leaders in the initial stages of the intervention. Several grantees suggested adopting more prominent and top-down relationships with GOI and judicial entities, such as prosecutors within the AGO. They stated that this communication could be improved by having access to top leadership to enhance implementation. One beneficiary noted that one intervention had been discussed with mid-level points of contact at the organization but rejected once it was taken up at the senior-management level. Another grantee stated, “My suggestion for future program implementation is strengthening the relationship between USAID CEGAH and government bodies up to the top management level/Directorate General to facilitate the coordination process in the program implementation.” The evaluation survey showed that 16 grantees and beneficiaries at both the national and local levels stated that CEGAH could improve by strengthening links with stakeholder counterparts; separately, six stated that CEGAH could better support coordination of law enforcement anti-corruption efforts. Expand and strengthen training opportunities. According to nine grantees and beneficiaries who administered or participated in CEGAH trainings, some CEGAH trainings to build GOI and judiciary 35 See EQ1 findings. 43 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report beneficiaries’ capacity did not consider differences in levels of knowledge between participants. One key informant from the media said, “If I want to give a training, I really need to understand, there's participants who are still at the entry level and that they have to take another training to be able to understand.” To reach the same level of understanding of the training materials among the participants, the ET recommends that CEGAH more specifically tailor trainings based on participants’ competency and knowledge. Among these nine training administrators and participants, three suggested that future trainings, particularly online trainings, could be more productive if trainees were provided with soft copies of training materials. The soft-copy materials can be downloaded in PDF format, so the participants can use them for further learning. This has, for example, proven especially useful in the Supreme Court’s anti-corruption certification training, which provides its CEGAH-developed modules electronically to participants ahead of the training. On the other hand, some participants also pointed out that putting certain materials online would not be beneficial if they were too long or too difficult to understand, noting that a book guide to a recent training of trainers course had these issues. A separate GOI training participant was critical of not being able to distribute materials from the training more widely within his organization after the training to help build the skills of other colleagues because of intellectual property rights concerns. While there would be additional costs associated with sharing the materials more widely, the participant thought this would be worthwhile to improve colleagues’ capacity in specific technical areas and to have them all learn from the same set of high-quality documents. For future activities, the beneficiary suggested that CEGAH and USAID provide GOI and judiciary organizations the option to directly purchase materials through their own budgets. Training interventions should also balance the needs, perspectives, and bureaucratic cultures of the participants. For example, an important secondary aim of one beneficiary training on financial investigation was to encourage more interaction and discussion among its interagency participants. The training had been designed with the aim of obtaining balanced representation from participants’ institutions. Even though the training aimed to balance participation, this was not successful: there were only two judges, while the rest of the participants were from the AGO’s office and KPK. As a result, AGO and KPK dominated the training, making it a less productive interagency learning opportunity, according to a grantee familiar with the training. “Last workshop was very AGO focused... we tried to ensure a balance of participants, but it was influenced by political interests,” the grantee said. “We need to [be] careful when trying to control the composition of participants, since we want their buy-in on the activities. We need to constantly foster the collaboration.” Longer interventions are more likely to achieve desired impacts. The short timeframes of CEGAH’s interventions could prevent long-term, sustainable benefits, according to 13 grantees and beneficiaries who raised concerns about intervention durations. These grantees and beneficiaries discussed the need for longer-term CEGAH interventions to maximize their effectiveness. Shorter-term CEGAH interventions, for example, could only target a limited number of beneficiaries and training participants, according to almost half of this group of 13 beneficiaries. They believe that if CEGAH had given them more time, these interventions could achieve higher-level results and benefit greater numbers of beneficiaries. One grantee remarked, “My only concern is on the duration of the program; I think it needs to be extended. This is because sometimes within a given period of 3–4 months, we have only reached the output, not achieved the outcome of the program yet.” Similarly, a separate grantee added, “This is about continuity of our project. If it is only 1 (one) year, sometimes we don’t have enough time to bring impact to government.” CEGAH staff largely saw the Activity’s built-in flexibility as a sufficient way to continue support where appropriate for grantees and beneficiaries, such as its ability to offer follow-up 44 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report grants on some tasks to enable additional “next steps” or additional progress (please see EQ1, "Stakeholders saw CEGAH as having too many tasks and too little resources and time devoted to each task" subsection). Other beneficiaries noted that longer-term ventures could provide time for grantees or CEGAH staff to complete key intervention steps to better realize intended impacts. For example, CEGAH helped develop SOPs on risk management for staff within the Inspectorate General of a GOI Ministry, according to two GOI beneficiaries (see EQ1). However, these staff members could not apply the SOPs fully in their work because they had not been written or adopted into the relevant ministerial regulation along with technical instructions, a task they had asked CEGAH for assistance on but did not hear back before their engagement with CEGAH on this task ended. “We asked for help to revise the regulation, but there has been no follow-up,” said one GOI beneficiary: “In 2020, there’s no collaboration yet.” Another grantee provided a separate example of what they saw as an intervention that was too short: their “mystery shopper” activity. Because of a short seven-month time frame, the grantee could only build the capacity of a few judges and registrars, who still needed time to absorb the technical knowledge to use the mystery shopper technique. The grantee said, “If you want to promote a good supervision mechanism in [the] judiciary, you need to spend more time, not only 7 months.” Limited effectiveness of specific interventions if they cannot be expanded. Eleven grantees and beneficiaries stated that the benefits of specific interventions would be undermined, if not negated, if they were not expanded to additional audiences or areas. In the case of SP4N LAPOR!, for example, several noted that the value of this intervention was tied to its expansion to additional locations and institutions (one beneficiary noted that they were required to report on this information to their grantee as a measure of their success).36 It is important to connect the system to service providers outside local towns or cities, because many complaints relate to such outside services as the Attorney General’s Office, police, the State Electricity Company, public hospitals, courts, and schools. If connected, the complaints could be transferred to the relevant institution, as one local government beneficiary noted. Other activities that could facilitate expanding this intervention are establishing a CSO and stakeholders’ forum and dedicated work units in each new locality who would be responsible for managing SP4N LAPOR!. In the auditing sector, two GOI beneficiaries suggested that supplemental trainings on auditing plans were needed, alongside new risk management training materials. “If USAID CEGAH wants to increase the capacity and capability of the three-line defense, it will be very necessary, in my opinion,” one GOI beneficiary suggested: “That includes risk management training to prevent fraud and corruption.”37 Journalists also suggested additional activities, noting that they needed to continue to improve their abilities to research and identify key data to be able to unearth important news. “The benefits are increasing our ability for in-depth journalism and data-focused techniques. Benefit in the office is the knowledge about packaging the investigation...we also provide knowledge to other colleagues. In terms of the reader, they interact through our social media and they are interested in investigations, especially if the investigation is closely related to current issues,” one journalist said. In this context, a key constraining factor was the CEGAH model itself, which emphasized by necessity a "breadth over depth" approach - e.g., less funding and shorter duration per task - which in turn limited 36 For example, the indicator “number of government agencies integrated into NCHS with support from CEGAH” (Indicator 12), USAID CEGAH Activity, Monitoring, Evaluation and Learning Plan, May 2019. 37 The Inspectorate General of Ministry of Health has a program to conduct training on risk management for all levels. All those levels are called the ”three-line defense,” which consists of: risk owners (first line), risk-supervisors (second line), and internal auditors (third line). 45 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report the extent to which CEGAH could expand these activities and still cover all of the tasks mandated by USAID under CEGAH (see ET’s response under EQ1 for more details on the effectiveness of the CEGAH model). Improving SP4N LAPOR! audience targeting and public use. An important aspect of continuing to popularize SP4N LAPOR! in new communities is linking awareness not just to new geographic areas but also targeted groups, providing specific information relevant to their needs with respect to SP4N LAPOR!. As one GOI informant said, “The number of users in LAPOR program is not proportionate with the total population (currently under 1% of total population; about 1 million). It may be necessary to increase intensity of socialization program, or there may be some areas that are not reachable by technology . . . the socialization of SP4N LAPOR! program into the community must be further improved including the increasing of intensity and the appropriate targeted group for socialization. Therefore, SP4N LAPOR! would be closer to the community.” As part of this expanded targeting, CEGAH should use its experience providing tailored training to CSO/grantees focused on gender equality and social inclusion issues to expand this support to new gender-focused partners. In 2017, for example, CEGAH held trainings in Jakarta, Banten, Jember, Medan, Ternate, and Malang, attended by CEGAH grantees and GoI partners that focused on increasing awareness on gender equality and empowering women to play a role in preventing corruption, while at the same time introducing SP4N LAPOR! TO them as a reporting tool for corruption cases. Internet access in specific areas of the country is another challenge for SP4N LAPOR!, according to five grantees and beneficiaries. One of the local GOI beneficiaries from one of these areas said, “Our complaint is the internet network, so people, who use lapor.go.id find it very limited and difficult. If it is not supported by a good network, it will be disrupted. Our local budget is limited to provide internet access in each region.” Similarly, future promotion of SP4N LAPOR! also needs to consider an area’s geographic situation, which can significantly affect a local community’s ability to follow the dissemination activity, given access and connectivity limitations. As explained by one local government interviewee, “If we conduct dissemination in each district, it really requires energy and cost, because of its mountainous geographical location.” Strengthen capacity and incentives for SP4N LAPOR! administrators. Three key informants described how administrators needed specific skill and knowledge qualifications for the system to work effectively among government entities at various levels. “LAPOR is a shared responsibility to improve government performance, and there must be same understanding,” one local GOI beneficiary stated. “They must understand how to use the application and distribute complaints to the relevant Regional Apparatus Organization (OPD/Organisasi Perangkat Daerah). Sometimes the understanding from the head of the OPD or regional apparatus is not same as ours, so when they distribute it to the administrator or his staff, their understanding is not good.” An incentive system can encourage stronger performance in responding to complaints from SP4N LAPOR!. The GOI and local governments need to develop and manage an effective incentive structure. Besides the SP4N LAPOR! competition (the CEGAH-supported contest sponsored by KemenPAN-RB, KSP, and ORI to measure the improvement of public services in local government agencies), there have been no other incentives for system administrators, according to one local GOI informant: “We found that there’s no reward or additional incentives for the administrators of SP4N LAPOR!” Reiterating a point discussed above under EQ1, this GOI informant noted that there is no dedicated local government staff member responsible for maintaining SP4N LAPOR!, so a designated official has to maintain SP4N LAPOR! on top of their other duties. “Because the administrators also have other main tasks of their role in the governments, meanwhile their salary is the same with their colleagues that are not the SP4N LAPOR! administrators. If there is a reward, there needs to be a punishment for the government that does not implement SP4N LAPOR! properly,” this local GOI key informant stated. Another local government informant thought that the central government could play a more robust role in the future in monitoring the performance of local government by managing an incentive mechanism: “Maybe it would be better if the central government provides 46 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report rewards and punishments to regions that are already running this application program. The use of SP4N LAPOR! is related to the performance of local government, and maybe it can be stated in the rules.” The ET provides details in its “Recommendations” section on how relevant GOI institutions could implement this approach (see ”Create system of local incentives”). 47 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report RECOMMENDATIONS ● Expand work with CSOs to build longer-term anti-corruption benefits. Civil society is part of the overall system that links government and society together and continues to play an important role in the development discourse and dialogue. The ET therefore recommends that USAID continue to invest in building a “community of accountability” in future anti-corruption activities, including strong civil society participation, more room for and support to CSO initiatives, and opportunities to enable collaboration between CSOs and government. CEGAH initiatives like strengthening the use of data-driven journalism with journalism CSOs are good examples of strengthening the role of civil society while also fighting corruption. Working with grassroot CSOs has also shown positive results, as it allows CEGAH to tap into their experience and network with other CSOs and with local governments. For example, the collaboration between CSOs and local government in Jember and Medan under CEGAH strengthened the role of the community satisfaction survey (Survey Kepuasan Masyarakat/SKM) of public health services, which in turn led to improvements in public health services and also became a recommended step in the budget planning process for regional development. This is an example of a good practice that could be continued. ● Conduct follow-up CEGAH interventions that enable tracking of both short- and long￾term effects of supported anti-corruption activities. In the time remaining in the CEGAH activity, the ET recommends following up initial tasks with follow-on interventions to extend/expand the work of the intervention while also tracking the longer-term effects of CEGAH support. This could also provide some flexibility to conduct a smaller-scale pilot project to verify an intervention’s effectiveness. A good example of this would be CEGAH’s support to seven pilot district courts in obtaining their SNI ISO 37001 "Anti-Bribery Management System" certification; under this recommendation, this intervention would have been conducted using a smaller number of courts than the seven that were certified, before administering a follow-up project after to assess the impact of the certification, such as how quickly cases are assigned to judges in these courts.. ● Build stronger learning culture and system into future anti-corruption programs. For future activities, the ET recommends that USAID incorporate a stronger learning culture and system into the program’s management within their design. This should include at least two elements: first, frequent, and structured learning sessions organized not only within the program’s management unit but also among the beneficiaries and grantees, either as stand-alone events or by incorporating smaller sessions into current CEGAH events, such as grantee coordination meetings, presentations of project or research findings, and trainings. The focus of these learning activities should be to reflect on interventions and how they are contributing to the program’s expected goals, as well as what will be needed to contribute to the goals based on the changing conditions. By facilitating this conversation among stakeholders, CEGAH stands a better chance of making improvements to programming based on learning among its stakeholders. Second, it should also include the measurement and collection of data not only for purposes of accountability and demonstration of impact, but also to further learning. This should include moving beyond focusing only on output-based indicators and reporting, which places the emphasis only on meeting targets and ensuring that the targets are achieved and reporting it. Such learning sessions could focus on changing conditions, influencing factors and what could be done to adapt to these to ensure the success of the program. ● Add greater flexibility and accountability to SP4N LAPOR! People must have multiple channels for complaining in case they can't access the Internet (which is a challenge in a country the size and diversity of Indonesia). In localities where this situation exists, the ET recommends greater promotion of alternate options to users for filing complaints (e.g., sending complaints via SMS message, the manual complaint-filing system) and to local leaders for potential creation of “home-grown” 48 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report solutions such as Medan City’s WhatsApp-based system, which can queue up messages offline and send them once a phone is connected to the Internet. The ET also recommends advising KemenPAN￾RB, KSP, ORI officials to establish and integrate a joint, robust monitoring and supervision mechanism and supporting features into SP4N LAPOR! As identified in this evaluation, better performance of SP4N LAPOR! is tied to the presence of monitoring mechanisms by decision makers and oversight agencies. Some good practices could be drawn from the implementation of complaint-handling systems at the local level as examples. Again, Medan City’s government provides an example, as it has used the performance of local government units in responding to the complaints/reports in SP4N LAPOR! as part of consideration in the regional budget allocation. Another example could be taken from the implementation of QLUE, which is a complaint-handling system initiated by Jakarta Province. QLUE has had considerable success in the past when it is supported with a very strong oversight mechanism from the Governor. This system was integrated into the Jakarta Command Center as part of Jakarta’s Smart City design, and tracked municipal public service delivery, including number of complaints, and how many days per rate of response to those complaints, analysis of reports/complaints from QLUE is also used to inform budget allocations in a process similar to Medan City. These examples also demonstrate the need to build similar oversight mechanisms and feedback loops to policies in order to boost the performance of SP4N LAPOR! as a national complaint handling system. ● Create system of local incentives: From its lessons learned, the ET recommends that USAID work in the future with ministry/government agencies at the national or local levels establish an incentives system that could be supported with strong oversight. Specifically, this system could provide options to enable higher compensation to government officials (with the expectation of improved complaint response performance in return), penalties if responsible individuals didn’t respond to a complaint within the stipulated time, etc. This monitoring and oversight functionality in the system could be extended to the local governments as a set of tools they could use to support the monitoring and evaluation of LAPOR! implementation. Currently, the SP4N LAPOR! competition provides limited incentive to develop innovative tools/processes for implementing the system locally. As part of the approach to implement this system, USAID could at the national level, USAID could provide technical support to KemenPAN-RB, ORI and KSP in developing policies, roadmaps & guidelines for sub￾national government for LAPOR! concerning the aforementioned incentive and oversight systems. USAID should also consider capacity building for oversight agencies such as ORI at the national and subnational levels, particularly in optimizing the use of the data and information from SP4N LAPOR! for monitoring and oversight purposes. ● Support alternative strategies to advance SP4N LAPOR! coverage and inclusion: Given how expansive and populous Indonesia is, the ET recommends that USAID consider alternatives to its current strategy, including suggesting modifications to KEMENPAN-RB’s current roadmap to address future coverage and inclusion needs (while the UNDP is formally responsible for this roadmap, the ET feels that offering innovative suggestions through interagency channels would be beneficial in this instance). In terms of coverage, this could include shifting focus away from remote, less developed locations and toward localities that have adequate telecommunication infrastructure (internet and phone signal) with greater digital or gadget literacy among the communities. This also includes assessing whether the main issue to SP4N LAPOR! utilization among the communities is the lack of awareness about SP4N LAPOR! or that the communities have a different preferred method of conveying complaints/reports. Working in localities where the communities are more familiar with and more likely to use a system like SP4N LAPOR! would enable the program to direct its efforts more toward strengthening the management aspect of SP4N LAPOR! In such communities, future efforts to socialize SP4N LAPOR! and integrate it with local complaint systems could be done with a lighter grantee “footprint,” including more communications with local leaders and socialization with 49 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report communities online (which would work better in the current COVID-19 environment as well) and then by continuing one-time events such as LAPOR Goes To Campus. In conjunction with its other recommendations on SP4N LAPOR!, the ET believes this will yield more “bang for the buck” in terms of increases in overall numbers of users, complaints, etc. At the same time, this strategy could be designed to more formally incorporate inclusion of marginalized populations by designing pilots that mandate this kind of support for, for example, people with disabilities. While SP4N LAPOR! currently provides different channels for complaint submission, such as text message, twitter, and websites. However, the system could be improved through technological upgrades and offline channels to facilitate access for people with disabilities. More broadly, it is also important to integrate with or allow parallel usage of other complaint systems, such as those outlined in the previous recommendation (see “Add greater flexibility and accountability to SP4N LAPOR!"). Recognizing that not all people have access to SP4N LAPOR! or the technology to access it (which some communities have responded to, as noted above, by making complaints capable of being submitted by text if they do not possess smartphones), or that some communities have different or preferred ways of submitting complaints, is also important to ensure inclusion. ● Strengthen capacity of local SP4N LAPOR! oversight/implementation staffing in select locations. In conjunction with suggested changes above to the SP4N LAPOR! implementation strategy, the ET also recommends that CEGAH support strengthening the capacity of government oversight representatives in local areas, which will enable more robust monitoring of the system’s utilization in the pilot localities. Such training could be conducted regionally or, alternatively, online, which would have the added benefit of being safer during the ongoing COVID-19 pandemic. Specifically, the ET recommends building the capacity of local ombudsmen as the main oversight bodies in pilot localities, and also inviting civil society participation in such trainings to help ensure that SP4N LAPOR! is implemented by the local government with optimal use by the communities. Additionally, the ET recommends extending access to the recently developed ‘executive dashboard’ for SP4N LAPOR! to local ombudsmen and head of regions (governor, regent, or mayor) or other sub-national oversight agencies, to enable them to monitor SP4N LAPOR! more closely. This would include training/ building their capacity on how to read the dashboard, analyze the information, and use it as feedback for policy. ● Extend the duration of CEGAH/future anti-corruption interventions to enhance their effectiveness. Longer-term interventions can help to achieve better outcomes, such as policy making and ensure that policy changes are approved and/or implemented within relevant institutions. It can also support human resource capacity strengthening, enlarge, strengthen joint coordination among institutions, and improve the abilities of training participants. The opportunities to resolve coordination issues between or within stakeholders would also be possible, with more time permitting sufficient additional CEGAH-mediated engagements between conflicting parties or resolution of interagency bureaucratic logjams. ● Enhance CEGAH’s monitoring and evaluation tools to better enable higher-level assessments of effectiveness. To help assess the effect of the Akademi Antikorupsi e-learning trainings, the ET recommends that CEGAH conduct a periodic survey of repeat users of the training – e.g., those who have earned a certificate of completion – on their learning, knowledge and perception of corruption in Indonesia, as well as the degree to which users have been able to recall and apply the training in their studies and/or work. CEGAH could conduct the survey its first year, then negotiate to have ICW continue to conduct it periodically (e.g., annually), which would enable the survey to identify trends among Academy participants. 50 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ● For a future anti-corruption activity, consider handing off/sharing specific CEGAH tasks: o Local government components of SP4N LAPOR! – Providing support to an intervention as big and complex as SP4N LAPOR! and expecting tangible results requires dedicated resources, time and energy in working with various levels of government, national down to sub-national level, as well as civil society and the public as a whole. From the experience of CEGAH’s grantees working with various local governments, it also demonstrates the challenge of trying to account for different levels of capacity, resources and commitment of each local government entity. While CEGAH has made significant strides in socializing and integrating SP4N LAPOR! within its 27 pilot local governments and beyond, expanding this to other local governments in other regions of Indonesia will require a similarly significant effort on a major scale. This expansion might be more effective if it was implemented in conjunction with a local governance strengthening effort in areas in which USAID contractors have a presence and long-term relationships with local leaders. This could function as a cross-cutting partnership between a CEGAH-supported central government effort (in support of KEMENPAN-RB, ORI, KSP) and an integrated parallel local governance-strengthening effort. o Countering Violent Extremism – As CEGAH winds down, the ET recommends maintaining the relationships built between grantees, beneficiaries and USAID using CEGAH's strong network and leverage them within current CVE activities (e.g., HARMONI) as well as future ones; similarly, use CEGAH-supported research as baselines wherever possible for future studies of similar at-risk groups by stand-alone CVE programming. ● Identify more ways to maintain/expand online platforms/tools that have shown to be useful – or at least promising – among beneficiaries and through use by the public. Tools such as the Akademi Antikorupsi modules, for example, have not only proven that they can be tailored for specific audiences and can reach more people throughout the country, an important consideration given Indonesia’s size and population, but can also continue to be used during the pandemic. 51 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEXES ANNEX 1: GANTT CHART OF EVALUATION ACTIVITIES Activities Mar￾20 Apr -20 May￾20 Jun￾20 Jul￾20 Aug -20 Recruitment of Team Leader and team members * Document Review by Team Leader/SI and Evaluation Team * Preparation of the work plan and evaluation design * * USAID reviews and approves work plan * Instrument Design and finalization of work plan based on USAID feedback * Submission of Instrument for USAID input and feedback, and the revised Work Plan * Final Work Plan approved by USAID * Online in-briefing * Finalization of data collection instrument * Data collection * Data analysis and findings * Online Debriefing – rough findings based on field notes and observation during field work * Report writing * Submission of Draft Report to USAID * USAID review of draft report * Incorporation of USAID comments and submission of final report * Submission of Final Evaluation Report to USAID * 52 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 2: EVALUATION TEAM TEAM ROLES AND RESPONSIBILITIES TABLE 4. ET ROLES AND RESPONSIBILITIES ROLES RESPONSIBILITIES Team Leader (TL) Mr. Adam Reisman The TL will be responsible for direct liaison with USAID, managing evaluation team personnel and assignment responsibilities, and producing high-quality evaluation deliverables, including the evaluation methodology and work plan, evaluation presentations, and the draft and final reports. The TL also will lead development of the survey and KII protocols, as well as survey implementation and analysis, and KII qualitative coding efforts. The TL will coordinate with, and be supported by SI’s evaluation specialists and research assistants. Evaluation Specialist – Bureaucracy Reform Ruth Alicia The Evaluation/Bureaucracy Reform Specialist will work under the supervision of the TL and will be responsible for contributing to all aspects of the evaluation, including supporting the development of the evaluation's methodology, data collection tools and management, data analysis, and writing the draft and final reports. The ES/BR also will support survey and KII data analysis, and coding of KII qualitative data. Evaluation Specialist – Anti-corruption Ratnayu Sitaresmi The Evaluation/Anti-corruption Specialist will work under the supervision of the TL and will be responsible for contributing to all aspects of the evaluation, including supporting the development of the evaluation's methodology, data collection and management, data analysis, and writing the draft and final reports. The ES/Anti-Corruption also will support survey and KII data analysis, and coding of KII qualitative data. Research Assistant Alberta Christina The Research Assistant will support the TL and ET in its data collection and evaluation management efforts, including tracking deliverables and deadlines, assisting with data entry, cleaning, coding and analysis as needed. During fieldwork, the research assistant also will be responsible for coordinating meeting appointments, for KIIs, site-visits, and other data collection activities. TEAM COMPOSITION Mr. Adam Reisman, Team Leader: Mr. Reisman has more than 16 years of experience evaluating, monitoring, and developing democracy & governance, conflict-mitigation, and capacity-building programs in developing and fragile states. He has led or served in senior technical roles on more than 20 performance, whole-of-project and process evaluations for USAID and the Department of State, including evaluations of justice-sector strengthening and anti-corruption activities. His background also includes corruption-focused research for USAID on the linkages between corruption and conflict, as well as experience evaluating justice-sector strengthening and anti-corruption activities; and designing indicators on corruption, accountability, transparency and rule of law for activity-level monitoring and evaluation (M&E) plans. Currently, Mr. Reisman serves as a Senior M&E Technical Specialist with SI’s Strategy, Performance and Learning Team, where he provides technical leadership and expertise in the development of performance management tools and performance evaluations of USAID projects both overseas and in Washington, DC. Mr. Reisman holds a Master of Art in International Affairs from the George Washington University. 53 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Ruth Alicia, Evaluation Specialist – Bureaucracy Reform: Ms. Alicia holds a Master’s degree in Public Sector Innovation and e-Governance. She has more than 10 years’ experience in reporting, monitoring, and evaluation for good governance programs in national and international organizations, including United Nations Development Programme (UNDP), Millennium Challenge Account-Indonesia (MCA-I), and Partnership for Governance Reform. She is well-equipped to comprehend, develop, analyze M&E framework and tools with a strong focus on learning. Alberta Christina – Research Assistant: Ms. Christina holds a Bachelor’s degree in Anthropology and has more than one year of experience conducting ethnography research focusing on human rights, elections, and power-relationships in local areas. She has also done project-based research with both national and international non-governmental organizations, including the Zoological Society of London, PILI (Indonesian Environmental Information Centre) Green Network, and the Traditional Territory Registration Board (Badan Registrasi Wilayah Adat), to evaluate and monitor the program in several local areas. EVALUATION MANAGEMENT In addition to the ET members described above, the evaluation will be supported by in-country support staff and SI-headquarters (HQ) staff. In-Country Support Staff. Three in-country SI staff from the MEL Support project, through which this study was commissioned will ensure quality and compliance of deliverables, adherence to deadlines, and budget management. Valentine Gandhi, the MEL Support Chief of Party acted as liaison between USAID Program Office and the Evaluation Team, as well as reviewed the deliverables. The Project Manager, Daniel Sahanggamu, is based at SI’s Indonesia office and will be responsible for coordination between the ET and SI-HQ to ensure quality assurance for the evaluation process. The Finance and Administration Specialist, Fina Hastuti, is also based at SI-Indonesia and will provide administrative and logistical support. Valentine Gandhi, the MEL Support Chief of Party acted as liaison between USAID Program Office and the Evaluation Team, as well as reviewed the deliverables. SI-HQ Support Staff. Three SI management personnel with distinct roles and responsibilities are critical to the successful implementation of this evaluation. The Project Director, Kari Nelson, is based at SI-HQ and will be responsible for quality assurance review of the evaluation deliverables. The Project Assistant, Nora Chamma and Alexandra McMullin, are based at SI-HQ and will provide coordination support for onboarding and training of all consultants and deliverables development. The Gender and Social Inclusion Specialist, Kathleen Sciarini, will ensure gender integration throughout the evaluation activities and deliverables. More details regarding in-country and HQ support staff roles and responsibilities are included in the table below. 54 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report TABLE 5. SUPPORT STAFF ROLES AND RESPONSIBILITIES Location Roles Responsibilities In￾Country Chief of Party Valentine Gandhi ● Responsible for liaison with PRO office COR as well as overall management of the study team. Work closely with the Program Manager to facilitate the study process. Support the Team Leader to liaise with CEGAH partners and USAID. Responsible for contractual and financial management of the evaluation. In￾Country Project Manager Daniel Sahanggamu ● Responsible for adherence to deadlines, requirements of the SOW, and budget management. ● Coordinates meetings and interactions among SI-HQ, the SI￾Indonesia MEL Support team, and within the evaluation team. ● Ensures quality and compliance of deliverables with contractual requirements. ● Manages the budget by tracking level of effort usage and other spending. In￾Country Finance and Administration Specialist Fina Hastuti ● Provides administrative and logistical support—arranges travel and lodging, processes expense reports, etc. ● Supports evaluation team members with invoicing, procurement, and other financial management. HQ Project Director Kari Nelson ● Conducts quality assurance review on all deliverables. ● Provides high-level guidance on methodology, major deliverables, and any problems. HQ Project Assistant Nora Chamma / Alexandra McMullin ● Supports contractual, onboarding process, and training arrangement for the evaluation team members, and setting up project site in SharePoint. ● Supports deliverable development, including formatting, proofreading, and copyediting. 55 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 3: BIBLIOGRAPHY Project Documents 1. Annual work plans 2. Quarterly reports 3. Annual reports 4. Monitoring, evaluation, and learning plans 5. Other project-produced products (list to be provided): a. Common Sentencing Guideline b. Prosecution for anti-corruption guideline by 2010 (Perja 039/2010) Other Documents 6. USAID Country Development Cooperation Strategy 2014-2018 7. Indonesia’s Medium-Term National Development Plan 2015-2019 8. Corruption Eradication Commission (Komisi Pemberantasan Korupsi/KPK) Long-term Strategic Plan 9. National Strategy on Preventing Corruption (STRANAS-PK) 10. Law No 25/ 2009 on Public Services; Presidential Regulation No. 76 /2013; KEMENPAN Regulation No. 24/2014; KEMENPAN Regulation No. 3/2015; KEMENPAN Decree No. 310/2019 -- on Complaint Handling System 11. Law No. 25/2004 on National Development Strategic Planning; Law No. 14/2008 on Information Disclosure Law No. 37/2008 on Ombudsman Republic of Indonesia (ORI); ORI Regulations No. 2/2009 on Procedures for Receiving, Inspecting and Completing Report; ORI Regulations No. 030/2018 on Organizational Structure and Working Procedures 12. Law No 5/2014 on Civil State Apparatus (ASN); Presidential regulations No. 42/2004 for Civil Service and Civil Code of Ethics; Gov Regulation No. 53/2019 on Discipline on Civil Servants 13. Indonesian Civil Service Commission (KASN) Regulation No. 9/2016 on Evaluating the quality of open selection and positions of senior leaders and civil servants; KASN Reg No. 2/2017; KASN reg No. 3/2018; KASN Reg No 4/ 2017 14. Supreme Court Regulation No. 13/2016 on guidelines for the prosecution of corporate entities; Supreme Court Regulations on the Integrity of the Public Services in the Court; Joint Ministerial Decree on Corruption Prevention Action 2019/2018 -- Supreme Court 15. External Documents: i. Transparency International: Indonesia - Overview of corruption and anti￾corruption 56 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 4: DATA COLLECTION PROTOCOLS KII PROTOCOLS KII PROTOCOL FOR USAID GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer to not answer 57 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc.) ● Other ___________________________ (please specify) QUESTIONS EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. In what ways did CEGAH’s design and structure help it achieve its objectives in terms of: a) the number of activities (probe: were there too many activities/too few)? b) the size of its activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) the mix of its activities (probe: what are examples of this within CEGAH – e.g., anti-corruption education at tertiary institutions, LAPOR! / taken together, did activities sufficiently focus on your beneficiaries’ needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your grantees or beneficiary organizations in terms of: a) the number of activities; b) the size ($) of its activities; and c) the mix of its activities 3. Was CEGAH’s design and structure sufficiently flexible and adaptive to enable it to modify its activity approach (if/when needed) during implementation? - Possible probes: - Ability to revisit/modify programmatic priorities: indicators, districts, actors (e.g., CVE in year 2; private sector engagement) - Internal management, standard operating procedures at National, provincial and district level, including activities implemented by sub-grantees - Processes for reviewing/learning from/acting on, challenges, needs, etc. (outcome, intervention, scope, scale) 58 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 4. Was CEGAH effective in coordinating efforts between stakeholders – e.g., between grantees, between CSOs & media? If yes, what were examples of this – Probe: Among specific actors? Specific activity types? Redundancy between activities? If not, why not? 5. Did CEGAH’s activities collectively produce stronger outcomes than they would have if implemented on their own? If yes, did CEGAH’s coordination (in #4 above) play a role in this? 6. How effectively did CEGAH measure its own progress – in terms of indicators used, data collection methods? To what extent did CEGAH use monitoring information to make course corrections and mitigate challenges? 7. To what extent were women/marginalized groups considered in CEGAH’s design and structure? How about marginalized groups (marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, immigrants)? Probe: - Were there any specific guidelines, strategies and/or tools provided by CEGAH for its partners and counterparts to facilitate the inclusion of women/marginalized groups in the implementation and monitoring of CEGAH? - Were partners able to use these guidelines, strategies, and/or tools in their implementation and/or monitoring of CEGAH’s activities? Why or why not? 8. Did the role of grantees in the structure and design of CEGAH help or hinder achievement of its results? Did grantees face any obstacles in their roles in CEGAH? If so, what were examples of this? Were they able to overcome them? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 9. In your view, which interventions were the most and the least successful. Why? Probe for example if needed. 10. In what ways did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc)? Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 11. Were women able to access/benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups? (Probe: marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 12. To what extent did CEGAH prioritize improving the transparency and accountability of local governments? - Probing Questions: - To what extent did they prioritize the adoption or integration of complaint handling system (LAPOR SP4N) or other local government initiatives to develop local complaint handling system? - To what extent have CSOs been involved in decision making and improving local government accountability? - How effective is the Regional Advocacy Forum in improving local government accountability and transparency? 59 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 13. What CEGAH initiatives do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 14. To what degree have CEGAH-supported entities – e.g., courts (case management system), law schools/universities, the Anti-Corruption Agency, civil society organizations, journalists – been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? - Probe: Is the GOI now funding any of CEGAH’s components that were previously supported through the activity? If not, to what extent has CEGAH given beneficiaries the capacity to secure such funding on their own? 15. What role is USAID playing in promoting the sustainability of CEGAH initiatives? Probe: co-creation on additional/future program planning? coordination with provincial and national representatives? 16. What other factors, if any, have or have not increased the chances for long-term sustainability of the activities? - Probe: For example, commitment of the organization, uptake of skills provided through trainings, etc, infrastructure, political will, geographic considerations, budget, human resources. 17. What worked best for CEGAH in terms of engaging with provincial level authorities to support replication of its activities? How does that compare that to any differences in approaches that were used to engage district level stakeholders for going to scale? Probe: - Has the Regional Advocacy Forum been replicated at the district level or within other provinces using local resources? - Have the trainings on SIJAPTI, public services, SIPP and monitoring platform (SPPT-TI, SIPTL, etc.) been extended to the other districts using local resources? - Does the anti-bribery management system have the potential to be replicated or scaled up? Has any initiative or innovation of CEGAH approaches obtained buy-in from and local government entities (provincial, district level), including those approaches implemented by CSOs/sub-awardees? (example: anti-bribery management system, regional advocacy forum, commitment to continue using public satisfaction survey tool (SKM) on health care services in East Java)? EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 18. In hindsight, which CEGAH activity components were the most important to advancing USAID’s anti-corruption objectives? Which less so? Why? 19. In hindsight, do you wish you had done anything differently in the design or implementation of CEGAH? Probe #1: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc? Probe #2 Examples: national vs. local government programming, demand-based activities (university/law school trainings, CSO initiatives, and the Anti-Corruption Film Festival) vs. supply￾based (strengthened capacity/tools within judiciary)? Please explain. Which parts of CEGAH would you want to see used again in the future? 60 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 20. Has any consideration been given to engaging/partnering with state-owned enterprises (e.g., BUMD, BUMN) under a CEGAH follow-on project (for sustainability)? 21. Private sector partnerships are expected to be a significant sector for further investment in the future. What are some key lessons learned from the CEGAH efforts to engage the private sector? 61 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR CEGAH IMPLEMENTING PARTNERS GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer to not answer 62 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) ● Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. In what ways did CEGAH’s design and structure help it achieve its objectives in terms of: a) the number of activities (probe: were there too many activities/too few?); b) the size ($) of its activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) the mix of its activities (probe: probe: taken together, did activities sufficiently focus on your grantees and beneficiaries’ needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your grantees or beneficiary organizations in terms of: a) the number of activities; b) the size ($) of its activities; and c) the mix of its activities 3. Was CEGAH’s design and structure sufficiently flexible and adaptive to enable it to modify its activity approach (if/when needed) during implementation? Possible probes: - Ability to revisit/modify programmatic priorities: indicators, districts, actors (e.g., CVE in year 2; private sector engagement) - Internal management, standard operating procedures at National, provincial and district level, including activities implemented by sub-grantees 63 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report - Processes for reviewing/learning from/acting on, challenges, needs, etc. (outcome, intervention, scope, scale) 4. Was CEGAH effective in coordinating efforts between stakeholders – e.g., between grantees, between CSOs & media? If yes, what were examples of this – Probe: Among specific actors? Specific activity types? Redundancy between activities? If not, why not? 5. Did CEGAH’s activities collectively produce stronger outcomes than they would have if implemented on their own? If yes, did CEGAH’s coordination (see above) play a role in this? 6. How effectively did CEGAH measure its own progress – in terms of indicators used, data collection methods? To what extent did CEGAH use monitoring information to make course corrections and mitigate challenges? Possible probe: What was your role in monitoring the implementation of CEGAH’s grants? How well did this go? 7. To what extent were women considered in CEGAH’s design and structure? How about marginalized groups (marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, immigrants)? - Probe: Were there any specific guideline, strategies and/or tools provided by CEGAH for its partners and counterparts to facilitate the inclusion of women or marginalized groups in the implementation and monitoring of CEGAH? - Were partners able to use these guidelines, strategies, and/or tools in their implementation and/or monitoring of CEGAH’s activities? Why or why not? 8. Did the role of grantees in the structure and design of CEGAH help or hinder achievement of its results? Did grantees face any obstacles in their roles in CEGAH? If so, what were examples of this? Were they able to overcome them? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 9. In your view, which interventions were the most and the least successful. Why? Probe for example if needed. 10. In what ways did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc) Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 11. Were women able to access/benefit from the activity to the same degree that men/dominant groups were? How about other marginalized/vulnerable groups (probe: marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 12. What were the advantages/disadvantages of USAID assistance in comparison to other donors? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti￾corruption work are other donors doing? 64 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 13. To what extent did CEGAH prioritize improving the transparency and accountability of local governments. - Probing Questions: - Adoption of LAPOR system or other local government initiatives to develop local complaint handling system and integration to LAPOR SP4N system - To what extent have CSOs been involved in decision making and to improve local government accountability? - How effective is the Regional Advocacy Forum in improving local government accountability and transparency? - To what extent has the piloting of the anti-bribery management system (SNI ISO 37001) been successful and adopted by the district court/local government? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 14. What CEGAH initiatives do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 15. To what degree have CEGAH-supported entities – e.g., courts (case management system), law schools/universities, the Anti-Corruption Agency, civil society organizations, journalists – been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? - Probe: Is the GOI now funding any of CEGAH’s components that were previously supported through the activity? If not, to what extent has CEGAH given beneficiaries the capacity to secure such funding on their own? 16. What other factors, if any, have or have not increased the chances for long-term sustainability of the activities? Probe: For example, commitment of the organization, community involvement, uptake of skills provided through trainings, infrastructure, political will, geographic considerations, budget, human resources, availability of staff for training, safety of operating environment, etc. 17. What role do you think USAID is playing in promoting the sustainability of CEGAH initiatives? - Probe: co-creation on additional/future program planning? coordination with provincial and national representatives? 18. What worked best for CEGAH in terms of engaging with provincial level authorities to support replication of its activities? How does that compare that to any differences in approaches that were used to engage district level stakeholders for going to scale? PROBING QUESTIONS: - Has the Regional Advocacy Forum been replicated at the district level or within other provinces using local resources? 65 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report - Have the trainings on SIJAPTI, public services, SIPP and monitoring platform (SPPT-TI, SIPTL, etc.) been extended to the other districts using local resources? - Does the anti-bribery management system have the potential to be replicated or scaled up? Has any initiative or innovation of CEGAH approaches obtained buy-in from and local government entities (provincial, district level), including those approaches implemented by CSOs/sub-awardees? (example: anti-bribery management system, regional advocacy forum, commitment to continue using public satisfaction survey tool (SKM) on health care services in East Java)? EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 19. In hindsight, which CEGAH activity components were the most important to advancing CEGAH’s objectives? Which less so? Why? 20. In hindsight, do you wish you had done anything differently in the design or implementation of the activity? (Probe #1: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc? Probe #2 – Examples: national vs. local government programming, demand-based activities (university/law school trainings, CSO initiatives, and the Anti-Corruption Film Festival) vs. supply-based (strengthened capacity/tools within judiciary)? Please explain. Which parts of CEGAH would you want to see used again in the future? 21. Has any consideration been given to engaging/partnering with state-owned enterprises (e.g., BUMD, BUMN)? 22. Private sector partnerships are expected to be a significant sector for further investment in the future. What are some key lessons learned from the CEGAH efforts to engage the private sector? 66 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR CSO GRANTEES (including CEGAH Partners – e.g., TAF, ICW, AJI, PSHK, LeIP, and MAPPI) GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer to not answer 67 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) ● Other ___________________________ (please specify) 68 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. For “partner grantees” (TAF, ICW, AJI, PSHK, LeIP, and MAPPI) and senior-level beneficiaries: Considering all of your work with CEGAH, in what ways did the design and structure help your beneficiary organizations in terms of: a) the number of activities (probe: were there too many activities/too few?); b) size of activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities (probe: taken together, did activities sufficiently focus on your beneficiaries’ needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your beneficiary organizations in terms of: a) the number of activities; b) the size ($) of its activities; and c) the mix of its activities 3. Was CEGAH effective in coordinating efforts between stakeholders – e.g., between grantees, between CSOs & media? If yes, what were examples of this – Probe: Among specific actors? Specific activity types? Redundancy between activities? If not, why not? 4. Did CEGAH’s activities collectively produce stronger outcomes than they would have if implemented on their own? If yes, did CEGAH’s coordination (see above) play a role in this? 5. How effectively did CEGAH measure its own progress – in terms of indicators used, data collection methods? To what extent did CEGAH use monitoring information to make course corrections and mitigate challenges? Possible probe: What was your role in monitoring the implementation of CEGAH’s grants? How well did this go? 6. To what extent were women considered in CEGAH’s design and structure? How about marginalized groups (marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, immigrants)? - (Probe: Were there any specific guidelines, strategies and/or tools provided by CEGAH for its partners and counterparts to facilitate the inclusion of women/marginalized groups in the implementation and monitoring of CEGAH? - Were partners able to use these guidelines, strategies, and/or tools in their implementation and/or monitoring of CEGAH’s activities? Why or why not? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 7. In your view, which interventions were the most and the least successful. Why? Probe for example if needed. 8. What changes have happened within your beneficiary organization(s) as a result of CEGAH’s support? Which change(s) do you think were most significant? Why do you think so? Probe for example, if possible. 9. In what ways did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc)? Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 10. Were women able to access/benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (probe: marginalized groups would be considered people 69 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 11. What were the advantages/disadvantages of USAID assistance in comparison to other donors? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti￾corruption work are other donors doing? 12. To what extent did CEGAH prioritize improving the transparency and accountability of local governments - Probing Questions: - To what extent did they prioritize the adoption or integration of complaint handling system (LAPOR SP4N) or other local government initiatives to develop local complaint handling system? - To what extent have CSOs been involved in decision making and improving local government accountability? - How effective is the Regional Advocacy Forum in improving local government accountability and transparency? - To what extent has the piloting of the anti-bribery management system (SNI ISO 37001) been successful and adopted by the district court/local government? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 13. What CEGAH initiatives do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 14. To what degree have CEGAH-supported entities – e.g., courts (case management system), law schools/universities, the Anti-Corruption Agency, civil society organizations, journalists – been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? Probe: Is the GOI now funding any of CEGAH’s components that were previously supported through the activity? If not, to what extent has CEGAH given beneficiaries the capacity to secure such funding on their own? 15. What other factors, if any, have or have not increased the chances for long-term sustainability of these activities within Indonesia’s courts after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, etc. 16. What worked best for your organization in terms of engaging with provincial level authorities to support replication of its activities? How does that compare to any differences in approaches that were used to engage district level stakeholders for going to scale? - PROBING QUESTIONS: - Has the Regional Advocacy Forum been replicated at the district level or within other provinces using local resources? - Have the trainings on SIJAPTI, public services, SIPP and monitoring platform (SPPT-TI, SIPTL, etc.) been extended to the other districts using local resources? - Does the anti-bribery management system have the potential to be replicated or scaled up? Has any initiative or innovation of CEGAH approaches obtained buy-in from and local government entities (provincial, district level), including those approaches implemented by CSOs/sub-awardees? (example: anti-bribery management system, regional advocacy forum, commitment to continue using public satisfaction survey tool (SKM) on health care services in East Java)? 70 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 17. To what extent do you think the CSO networks will continue to support the supported institutions’ mission after CEGAH ends? What do they need to do? What will you do to continue their support? EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 18. Which CEGAH activity components were the most important to strengthening your beneficiaries’ objectives? Which less so? Why? 19. In hindsight, do you wish any part of CEGAH’s design or implementation had been done differently? Probe #1: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc? Probe #2 – Examples: national vs. local government programming, demand￾based activities (university/law school trainings, CSO initiatives, and the Anti-Corruption Film Festival) vs. supply-based (strengthened capacity/tools within judiciary)? Please explain. Which parts of your activity would you want to see used again in the future? 20. Has any consideration been given to engaging/partnering with state-owned enterprises (e.g., BUMD, BUMN) under a CEGAH follow-on project (for sustainability)? 21. Private sector partnerships are expected to be a significant sector for further investment in the future. What are some key lessons learned from the CEGAH efforts to engage the private sector? 71 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR GOVERNMENT OF INDONESIA INSTITUTIONS (including executive, ministerial & independent agencies) GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed: Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer not to answer 72 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) ● Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. Considering all of your work with CEGAH, in what ways did the design and structure help your government agency in terms of: a) the number of activities (probe: were there too many activities/too few?); b) size of activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?) 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your institution’s mandate, in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities; and/or d) other areas? 3. As beneficiaries of CEGAH, what kind of support was your ministry/agency/commission provided during program implementation by CSO grantees and/or CEGAH during the activity’s implementation? 73 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to your ministry/agency/commission’s ability to achieve XX results (see below)? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for explanation. Sub-Questions for Specific GOI Organizations For: KPK In what ways has CEGAH contributed to your institution’ ability to effectively investigate and prosecute all type of corruption cases? How would you describe this capacity before and now after CEGAH activities? What has CEGAH helped you to achieve that you could not achieve before this activity? Why is this achievement significant? Probe for explanation/example: indexation & annotation of corruption case indictments, research & training, UPG in North Maluku, anti-corruption education & modules, etc. For: KOMJAK, BAPPENAS, SecGen Kemenkumham In what ways has CEGAH contributed to your institution’s ability in improving the effectiveness of prosecution and adjudication of corruption cases? What has CEGAH helped you to achieve that you could not achieve before this activity? Why is this achievement significant? Probe for explanation/example: SPPT-TI business intelligence (Bappenas) and case annotation and PIK3 Index (KOMJAK) For: OJK & PPATK In what ways has CEGAH contributed to your institution’s ability in performing oversight in relation to corruption and terrorism financing activity? What has CEGAH helped you to achieve that you could not achieve before this activity? Why is this achievement significant? Probe for explanation/example. In what ways has CEGAH contributed to strengthening your institution, particularly to improve accountability through prevention &/monitoring of possible corruption, fraud, etc.? What has CEGAH helped you to achieve that you could not achieve before this activity? Why is this achievement significant? Probe for explanation/example: • BPK: movie festivals and e-learning module on performance audit • BPKP: communication assessment, MOOCs platform • IG Kemenkes: CVA & whistle blowing – training • IG Kemendikbud: risk assessment-management & whistle blowing – training • IG Kemenkumham: risk assessment-management • KemenPANRB: SP4N-LAPOR!, Public Service Information System (SIPP), mystery shopper module, etc. • SecGen Kemenkumham: complaint handling service For: KASN, ORI, KIP In what ways has CEGAH contributed to strengthening your institutions in implementing your mandate? What has CEGAH helped you to achieve that you could not achieve before this activity? Why is this achievement significant? Probe for explanation/example: 74 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report • KASN: SIJAPTI, SIPINTER, LAPOR KASN applications - mandate related to bureaucracy reform • ORI: systemic investigation, case management system (SIMPel), LAPOR!, mystery shopping manual • KIP: standard of village information service (SLID), case management system (SIMSI), PPID blueprint, etc. 5. What changes have happened within your ministry/agency/commission as a result of CEGAH’s support? Which change(s) did you think were most significant? Why? Probe for example, if possible. 6. How would you describe the capacity of your ministry/agency/commission to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen your institution’s understanding of and/ capacity to detect and prevent radicalization with the government? In what way? What work have your institution done with other stakeholders on corruption related to CVE? For BPKP, Kemenpan-RB, IG Kemendikbud & IG Kemenag, KASN Probe on the following: • BPKP: research on role of APIP in preventing radicalism in government • KemenPANRB: interview tools to detect individuals at risk of radicalization • IG Kemendikbud & IG Kemenag: IG auditors for school supervisors to promote tolerance and multiculturalism • KASN: CVE module 8. To what extent did CEGAH prioritize improving the transparency and accountability of local governments? • For KemenPANRB Probe for Adoption of LAPOR system or integration of local government initiatives and system to SP4N LAPOR! system complaint handling system • For KPK Probe for anti-corruption education and its contribution in disseminating initiatives at local level • For KASN Probe for adoption of SIJAPTI by the local government and other government organizations and to what extent SIJAPTI has been able to improve transparency of recruitment. • For KIP Probe for SLID (public disclosure system of Information Service for village governments) and its adoption 9. In what way did any external factors impact the success of CEGAH’S support to your ministry/agency/commission – e.g., enabling factors - new laws, political developments, etc)? Was your ministry/agency/commission able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 10. Were women able to access/benefit from the activity to the same degree that men were (probe: were they selected to participate in CEGAH trainings on an equal basis? Why/why not?) How about other and other marginalized/vulnerable groups (marginalized groups would be considered people 75 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 11. Were there any unanticipated outcomes, positive or negative from CEGAH activity/ies? 12. What were the advantages/disadvantages of CEGAH’s support to your institution in comparison to other donors? What have you been able to achieve through CEGAH that you couldn’t through other donors – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti-corruption work are other donors doing? 13. To what extent did CEGAH prioritize improving the transparency and accountability of local governments/institutions? Probing questions • For KPK: anti-corruption education and its role in improving transparency and accountability at the local governments/institution? Demonstrate examples, if any. • For BPK: to what extent did the e-auditee portal and learning module on performance audit contribute to improving transparency and accountability at the local governments/institution? Demonstrate examples, if any. • For IG Kemenkes: how has training on corruption vulnerability assessment helped to improve accountability of the hospitals and poltekes? • For KemenPANRB & ORI: to what extent has the LAPOR SP4N and SIPP (Public Service Information) system contributed to improving transparency and accountability at the local governments/institution? Demonstrate examples, if any. • For KASN: to what extent the applications (SIJAPTI, SIPINTER or LAPOR KASN), relevant training and e-learning program contribute to improving transparency and accountability at the local governments/institution? Demonstrate examples, if any. EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 14. What CEGAH initiatives do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 15. Has CEGAH given sufficient support for your institution to continue the implementation of this initiative, in terms of capacity to maintain system (e.g. LAPOR! or other systems)? Are you now providing some or all of the funding yourself to continue this initiative? 16. What other factors, if any, have or have not increased the chances for long-term sustainability of these activities within your ministry/agency/commission after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. 76 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 17. Which CEGAH activity components were the most important to strengthening your institution? Which less so? Why? 18. Could any part of CEGAH’s support to your institution have been implemented differently? Probe: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc? Which parts of your activity would you want to use again in the future? 19. Private sector partnerships are expected to be a significant sector for further investment in the future. To what extent has your agency established any partnerships with private sector organizations to strengthen or continue the work you have been conducting with CEGAH support? If yes, what have you learned from these partnerships thus far? KII PROTOCOL FOR JUDGES, COURT STAFF, AGO GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer not to answer 77 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: o GOI [executive/ministerial] representatives o GOI [independent agency/commission] representatives o Judges and court staff o Law school and university administrators/students o CSO grantees o CSO beneficiaries o Media representatives o USAID o CEGAH implementing partners o Private Sector (e.g., universities, think tanks, business associations/individual businesses) o External stakeholders (e.g., non-USAID donors, etc) o Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. For “partner grantees” (TAF, ICW, AJI, PSHK, LeIP, and MAPPI) and senior-level beneficiaries: Considering all of your work with CEGAH, in what ways did the design and structure help the judiciary in terms of: a) the number of activities (probe: were there too many activities/too few?); b) size of activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine the judiciary in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities? 3. As beneficiaries of CEGAH, what kind of support were the courts provided during program implementation by CSO grantees during CEGAH’s implementation? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to the courts’ ability to achieve XX result (e.g., prosecute corruption cases more effectively, adjudicate corruption cases more effectively, etc)? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for example: 78 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report a) SC: Upgrading (to version 3) of Direktori Putusan module (supports online data/analysis warehouse development) (RT 1) b) SC/AGO (Police?): Criminal Case Data Exchange between law enforcement agencies; completion of SPPT-TI development architecture/dashboard, handover of data exchange center (Puskarda) maintenance and its further development to Kemkominfo (RT 1) c) SC/AGO: Developing Prosecution and Sentencing Guidelines (MaPPI FHUI) (RT 6) d) SC Supervisory Body (BAWAS MA) / District Courts (PNs): implementation of the Anti-Bribery Management System (SNI ISO 37001). PNs of Central Jakarta and Surabaya designated as pilot courts for implementing guidelines (RT 8). e) ADDITIONAL: For: AGO: indexation & annotation of corruption case indictments, survey on corruption perception, etc. 5. What changes have happened within the courts/AGO as a result of CEGAH’s support of the judicial and law enforcement sector? Which change(s) did you think were most significant? Why? Probe for example, if possible: a) SC: Increasing acceptance/support for corruption sentencing guidelines (RT 6) 6. How would you describe the capacity of the courts to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen the courts’/prosecutors’ understanding of or capacity to counter violent extremism? In what way? What work have the courts done with other stakeholders on corruption related to CVE? 8. In what way did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc)? Were the courts/prosecutors able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 9. Were women able to access/benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (probe: marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 10. What were the advantages/disadvantages of CEGAH’s support to the courts in comparison to other donors? What have you been able to achieve through CEGAH that you couldn’t through other donors – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti-corruption work are other donors doing? 11. To what extent did CEGAH prioritize improving the transparency and accountability of local governments/institutions - Probing Questions: a) SC/BAWAS MA/PNs: To what extent has the piloting of the anti-bribery management system (SNI ISO 37001) been successful and adopted by the district court/local government? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 12. Which of these CEGAH-supported activities do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any 79 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 13. To what degree have the courts (e.g., case management system) been able to secure non-CEGAH funding to ensure continuation of this support after it ends? If it has, what is the source of the funding – Probe: A different donor, GOI, etc.? 14. What other factors, if any, have or have not increased the chances for long-term sustainability of these activities within Indonesia’s courts after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 15. Which CEGAH activity components were the most important to strengthening the courts? Which less so? Why? 16. Could any part of CEGAH’s support to the courts been implemented differently (probe: e.g., in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc)? Which parts of your activity would you want to use again in the future? 80 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR MEDIA REPRESENTATIVES GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer to not answer 81 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) ● Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. For “partner grantees” (TAF, ICW, AJI, PSHK, LeIP, and MAPPI) and senior-level beneficiaries: Considering all of your work with CEGAH, in what ways did the design and structure help your media organization in terms of: a) the number of activities (probe: were there too many activities/too few?); b) size of activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities (probe: taken together, did activities sufficiently focus on your organization’s needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine the judiciary in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities; and/or d) other areas? 3. As beneficiaries, what kind of support did you provided during program implementation by CSO grantees during CEGAH’s implementation? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to your ability to achieve XX result? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for explanation. 82 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 5. What changes have happened within your organization as a result of CEGAH’s support of your organization? Which change(s) did you think were most significant? Why? Probe for example, if possible. 6. How would you describe the capacity of [your organization] to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen your understanding of or capacity to work against violent extremism? What work have you done with other stakeholders on corruption related to CVE? 8. In what way did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc)? Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 9. Were women able to access/benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (probe: marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 10. To what extent has your support from CEGAH enabled you to report on how women and other marginalized/vulnerable groups have been affected by issues involving corruption? If it has enabled such reporting, what were examples of this? If not, what is your analysis? 11. To what extent has your work involved advocating for the transparency and accountability of local governments? To what extent did CEGAH support your work in this area? What have you been able to do as a result of this support? Probing Questions: - To what extend did you advocate for adoption of LAPOR system or other local government initiatives to develop local complaint handling system and integration to LAPOR SP4N system to be recognized and understood by public? - To what extent has your organization (or other media organizations) been involved in advocating to improve local government accountability? - To what extend did you support effectiveness of Regional Advocacy Forum in improving local government accountability and transparency? 12. What were the advantages/disadvantages of USAID assistance in comparison to other donors? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti￾corruption work are other donors doing? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 13. Do you believe your CEGAH-supported activities will be able to continue after CEGAH ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources, dissemination issue. 83 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 14. To what degree has your organization been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? If it has, what is the source of the funding – Probe: A different donor, GOI, etc.? 15. What other factors, if any, have or have not increased the chances for long-term sustainability of these activities after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, etc. EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 16. Which CEGAH activity components were the most important to strengthening your/your organization’s capacity? Which less so? Why? 17. Could any part of your support from CEGAH have been implemented differently? Probe #1: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc? Which parts of your activity would you want to use again in the future? 84 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR CSO BENEFICIARIES GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed: Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer to not answer 85 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) o Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. For “partner grantees” (TAF, ICW, AJI, PSHK, LeIP, and MAPPI) and senior-level beneficiaries: Considering all of your work with CEGAH, in what ways did the design and structure help your organization in terms of: a) the number of activities (probe: were there too many activities/too few?); b) size of activities (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your organization in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities? 3. As beneficiaries, what kind of support were you provided during program implementation by CSO grantees during CEGAH’s implementation? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to your ability to achieve XX result? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for explanation. 86 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 5. What changes have happened within your organization as a result of CEGAH’s support of your organization? Which change(s) did you think were most significant? Why? Probe for example, if possible. 6. How would you describe the capacity of [your organization] to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen your understanding of or capacity to counter violent extremism? In what way? What work have you done with other stakeholders on corruption related to CVE? 8. In what way did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc)? Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 9. Were women able to access/to benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 10. What were the advantages/disadvantages of USAID assistance in comparison to other donors? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti￾corruption work are other donors doing? 11. To what extent did CEGAH prioritize improving the transparency and accountability of local governments/institutions? - Probing Questions: - To what extent did they prioritize the adoption or integration of complaint handling system (LAPOR SP4N) or other local government initiatives to develop local complaint handling system? - To what extent have CSOs been involved in decision making and improving local government accountability? - How effective is the Regional Advocacy Forum in improving local government accountability and transparency? - To what extent has the piloting of the anti-bribery management system (SNI ISO 37001) been successful and adopted by the district court/local government? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 12. Do you believe your CEGAH-supported activities will be able to continue after CEGAH ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 13. To what degree has your organization been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? If it has, what is the source of the funding – Probe: A different donor, GOI, etc.? 87 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 14. What other factors, if any, have or have not increased the chances for long-term sustainability of these activities after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. 15. To what extent do you think the CSO networks will continue to support your institutions’ mission after CEGAH ends? What do they need to do? What will you do to continue their support? EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 16. Which CEGAH activity components were the most important to strengthening your capacity? Which less so? Why? 17. Could any part of your supporting activity from CEGAH have been implemented differently (probe: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc?) Which parts of your involvement would you want to use again in the future? 88 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR UNIVERSITY BENEFICIARIES GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed: Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer not to answer 89 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: ● GOI [executive/ministerial] representatives ● GOI [independent agency/commission] representatives ● Judges and court staff ● Law school and university administrators/students ● CSO grantees ● CSO beneficiaries ● Media representatives ● USAID ● CEGAH implementing partners ● Private Sector (e.g., universities, think tanks, business associations/individual businesses) ● External stakeholders (e.g., non-USAID donors, etc) ● Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. Considering all of your work with CEGAH, in what ways did CEGAH’S design and structure help your organization? For example: a) the number of activities under CEGAH (probe: were there too many activities/too few?); b) size of activities under CEGAH (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities under CEGAH (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your organization in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities? 3. As beneficiaries, what kind of support were you provided during program implementation by: a) CEGAH staff; and/or 2) your CSO/grantee partner, during CEGAH’s implementation? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to your ability to achieve XX result? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for explanation, such as: - Universities that incorporated ICW’s Anti-Corruption Academy into curricula (RT 33 / Grantee: ICW) 90 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report - UNUSIA & UMJ: Work with Islamic and Public Universities to conduct research and mapping for current extremism (RT 48) - LAPOR Goes to Campus events 5. What changes happened within your organization as result of CEGAH’s Program intervention? Which change(s) did you think were most significant? Why? Probe for example. 6. How would you describe the capacity of [your organization] to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen your understanding of or capacity to counter violent extremism? In what way? What work have you done with other stakeholders on corruption related to CVE? 8. In what way did any external factors impact the implementation of your CEGAH-supported work – e.g., enabling factors - new laws, political developments, etc)? Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 9. Were women able to access/to benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 10. What were the advantages/disadvantages of CEGAH (USAID) assistance in comparison to other partners (donors)? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Probe: What notable anti￾corruption work are other partners/donors doing? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 11. Do you believe your university initiative(s) will be able to continue after CEGAH ends? Is your activity self-sufficient? If not, who will support it? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 12. To what degree has your organization been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? Probe as needed: If it has, what is the source of this funding – e.g., different donor, GOI, etc.? 13. What other factors, if any, have or have not increased the chances for long-term sustainability of your activity after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 14. Which components of your CEGAH activity were the most important to strengthening your capacity? Which less so? Why? 91 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 15. Could any part of your activity have been implemented differently (probe: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc)? Which parts of your involvement could be used again in the future? 92 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR PRIVATE SECTOR STAKEHOLDERS (including think tanks, business associations, businesses, etc.) GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed Respondent sex: ● Male ● Female ● Other ● Prefer not to say 93 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: o GOI [executive/ministerial] representatives o GOI [independent agency/commission] representatives o Judges and court staff o Law school and university administrators/students o CSO grantees o CSO beneficiaries o Media representatives o USAID o CEGAH implementing partners o Private Sector (e.g., universities, think tanks, business associations/individual businesses) o External stakeholders (e.g., non-USAID donors, etc) o Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. Considering all of your work with CEGAH, in what ways did CEGAH’S design and structure affect your organization? For example: a) the number of activities under CEGAH (probe: were there too many activities/too few?); b) size of activities under CEGAH (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities under CEGAH (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?). 2. [Utilize as needed if issues with design not sufficiently addressed in 1). In what ways, if any, did CEGAH’s structure undermine your institution’s mandate, in terms of: a) the number of activities; b) the size ($) of its activities; c) the mix of its activities; and/or d) other areas? 3. What kind of support/engagement have you had with: a) CEGAH staff; and/or 2) any CEGAH￾CSO/grantee partner? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 4. In what ways has CEGAH contributed to your ability to achieve XX result(s)? What has CEGAH helped you to achieve that you could not achieve before this activity? Probe for examples. 94 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 5. What changes happened within your organization as result of CEGAH’s support/work? Which change(s) did you think were most significant? Why do you think so? Probe for example. 6. How would you describe the capacity of [your organization] to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? 7. Do you feel CEGAH’s activities have helped strengthen your understanding of or capacity to counter violent extremism? In what way? What work have you done with other stakeholders on corruption related to CVE? 8. Were women able to access/to benefit from the activity to the same degree that men were? How about other marginalized/vulnerable groups (marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 9. What were the advantages/disadvantages of CEGAH (USAID) assistance in comparison to other partners (donors)? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti-corruption work are other partners/donors doing? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 10. Do you believe your initiative(s) will be able to continue after CEGAH ends? If so, why? Is your activity self-sufficient? If not, who will support it? Probe infrastructure, political will, geographic considerations, budget, human resources. 11. What factors, if any, have or have not increased the chances for long-term sustainability of your activity after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 12. Which components of your CEGAH activity were the most important to strengthening your capacity? Which less so? Why? 13. Could any part of your activity have been implemented differently (probe: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc)? Which components of your work would you want to see used again in the future? 95 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report KII PROTOCOL FOR EXTERNAL STAKEHOLDERS (including non-USAID Donors, INGOs) GENERAL INFORMATION Name of interviewer: Date of interview: ____ June 2020 Respondent code of person interviewed ADDITIONAL INFORMATION (NOTE: Only to be asked if confirmation needed, as ET should have this information already) Location of person interviewed Province: City/town/village: Organization/affiliation of person interviewed: Respondent sex: ● Male ● Female ● Other ● Prefer not to say 96 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Organization type: o GOI [executive/ministerial] representatives o GOI [independent agency/commission] representatives o Judges and court staff o Law school and university administrators/students o CSO grantees o CSO beneficiaries o Media representatives o USAID o CEGAH implementing partners o Private Sector (e.g., universities, think tanks, business associations/individual businesses) o External stakeholders (e.g., non-USAID donors, etc) o Other ___________________________ (please specify) QUESTIONS i. What has been your experience/familiarity with CEGAH? What is your role in your organization? EQ1: In what ways did CEGAH’s design and structure help or hinder the achievement of its results? 1. Considering your knowledge of CEGAH, in what ways do you think its design and structure helped it achieve its objectives? For example: a) the number of activities under CEGAH (probe: were there too many activities/too few?); b) size of activities under CEGAH (probe: were activities’ scope too narrow/too wide – e.g., # or categories of people supported, length of time of activities; anything important that didn’t get done but should have?); c) mix of activities under CEGAH (probe: taken together, did activities sufficiently focus on your org’s needs; if not, what else should they have been focusing on?). 2. Was CEGAH effective in coordinating efforts between stakeholders – e.g., between grantees, between CSOs & media? If yes, what were examples of this – Probe: Among specific actors? Specific activity types? Redundancy between activities? If not, why not? 3. Did CEGAH’s activities collectively produce stronger outcomes than they would have if implemented on their own? If yes, did CEGAH’s coordination (in #4 above) play a role in this? 4. To what extent were women considered in CEGAH’s design and structure? How about marginalized groups (marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, immigrants)? a) (Probe: Were there any specific guidelines, strategies and/or tools provided by CEGAH for its partners and counterparts to facilitate the inclusion of women/marginalized groups in the implementation and monitoring of CEGAH? 97 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report b) Were partners able to use these guidelines, strategies, and/or tools in their implementation and/or monitoring of CEGAH’s activities? Why or why not? 5. Did the role of grantees in the structure and design of CEGAH help or hinder achievement of its results? Did grantees face any obstacles in their roles in CEGAH? If so, what were examples of this? Were they able to overcome them? EQ2: Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 6. In your view, which interventions were the most and the least successful. Why? Probe for example if needed; alternatively probe as needed: a) How would you describe the capacity of CEGAH-supported organizations to [identify, monitor, or prevent] corruption before CEGAH? How would you describe this capacity now? What do you think has changed the most? The least? Why? b) Do you feel CEGAH’s activities have helped strengthen their beneficiaries’ understanding of or capacity to counter violent extremism? In what way? 7. In what ways, if any, did any external factors impact the implementation of these interventions – e.g., enabling factors - new laws, political developments, etc) Was the activity able to change to take advantage of new opportunities or adapt to new constraints? Probe for example if needed. 8. Were women able to access/benefit from the activity to the same degree that men were? How about marginalized groups (probe: marginalized groups would be considered people living below poverty line, elderly, minority ethnic/religious group, immigrants)? If yes, what were examples of this? If not, why not? 9. What were the advantages/disadvantages of USAID assistance in comparison to other donors? What was it able to achieve through CEGAH that other donors were not – e.g., with targeted beneficiaries, in strengthening anti-corruption organizations? Potential Probe: What notable anti￾corruption work are other donors doing? EQ3: Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 10. What CEGAH initiatives do you believe will be able to continue after the activity ends? Are these activities self-sufficient? If not, who will support them? Are there any initiatives that will be more difficult to continue, and why? Probe infrastructure, political will, geographic considerations, budget, human resources. 11. To what degree have CEGAH-supported entities – e.g., courts (case management system), law schools/universities, the Anti-Corruption Agency, civil society organizations, journalists – been able to secure non-CEGAH funding to ensure continuation of this support after CEGAH ends? Probe as needed: If it has, what is the source of the funding – e.g., a different donor, GOI, etc.? 12. What other factors, if any, have or have not increased the chances for long-term sustainability of the activities after CEGAH ends? Probe: For example, commitment of the organization, uptake of skills provided through trainings, availability of staff for training, safety of operating environment, etc. 98 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report EQ4: What lessons learned in strategy and approach can and should be applied to future programming? 13. Which components of CEGAH’s activity were the most important in your view? Which less so? Why? 14. Could any parts of these activities been implemented differently (probe: in a way that would have led to better results, stronger sustainability, would have cost less to implement, etc.)? Which parts of your involvement could be used again in the future? 99 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report SURVEY PROTOCOL Survey Protocol Part 1: Survey Respondent Information Respondent sex: Respondent Province Respondent City Organization/affiliation of respondent I am a CEGAH….. select one Grantee Beneficiary Both Organization type: select one GOI [executive/ministerial/independent agency/commission] Judges, court staff, or AGO University CSO grantees CSO staff who participated in trainings Media representatives Private Sector (e.g., think tanks, business associations/individual businesses) External stakeholders (e.g., non-USAID donors, etc) Other ____________________(please specify) 100 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report I have received support through CEGAH for…. Less than 1 year 1-2 years 2-3 years More than 3 years Please name the current or most recent CEGAH activity in which you have been involved. Part 2: Scale Questions Instructions for Survey Respondent: Please choose a number on a scale of 1-4 that corresponds with how you feel about the following statements (1 = Strongly Disagree, 4 = Strongly Agree). You are welcome to elaborate on your response after choosing the number. Questions Strongly Disagree Disagree Agree Strongly Agree 1. As a beneficiary or grantee of CEGAH, my organization has received robust support from CEGAH or my grantee CSO throughout implementation of my activity. 1 2 3 4 2. CEGAH has improved my organization’s ability to identify, monitor, and/or prevent corruption. 1 2 3 4 3. CEGAH has improved my organization’s ability to understand and/or help work against violent extremism. 1 2 3 4 4. Women and other marginalized/vulnerable groups were able to access/benefit from our CEGAH￾supported activity to the same degree that men were. 1 2 3 4 5. Marginalized/vulnerable groups38 were able to access/benefit from our CEGAH-supported activity to the same degree that men were 1 2 3 4 6. CEGAH’s support strengthened my organization’s ability to fight corruption in comparison to support I received from other donors. 1 2 3 4 38 Marginalized groups would be considered people living below the poverty line, elderly, minority ethnic/religious group, and immigrants. 101 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Part 3: Open-ended Questions 7. What is the most significant change that has taken place within your organization as a result of CEGAH? 8. Could any part of CEGAH’s support to your organization have been implemented in a way that would have led to better results, stronger sustainability, or would have cost less to implement? Part 4: Multiple Choice / Yes-No Questions 9. Did the design and structure of CEGAH help your organization in terms of the following? a. The number of CEGAH activities was: i. Too few ii. Too many iii. Just right iv. Don’t know/prefer not to say b. The scope of CEGAH’s individual activities were generally: i. Too narrow ii. Too broad iii. Just right iv. Don’t know/prefer not to say c. The mix of CEGAH’s activities was: i. Not optimal – Activities were not sufficiently focused on the best sectors or types of activities to work against corruption ii. Minimally optimal – Activities were somewhat focused on the best sectors or types of activities to work against corruption iii. Optimal – Activities were sufficiently focused on the best sectors or types of activities to work against corruption iv. Don’t know/prefer not to say 9.a – 9.c: Use the space below to provide written clarification/elaboration of your responses to 8.a, 8.b, or 8.c. 102 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report 10. Did any external factors impact the success of your CEGAH-supported work (e.g., enabling factors - new laws, political developments, etc)? a. Yes b. No c. Don’t know 10.a. If yes, please briefly explain how 11. Do you believe your activity/ies will be able to continue after CEGAH’s support ends? a. Yes b. No c. Don’t know 11.a. If yes, please name the activity and briefly explain why you believe it will continue. If not, please explain why not. 103 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report CONSENT FORM CEGAH Evaluation: KII Consent Form Note to evaluator: You must read the consent script aloud to the respondent exactly as written. Introduction: Thank you for taking the time to meet with us today. We are researchers from Social Impact (SI) and we have been contracted by USAID (the US government) to evaluate CEGAH, an activity funded by USAID/Indonesia and implemented by MSI. Our team is composed of independent evaluators who are not part of USAID or the CEGAH Activity. We will ask you questions related to CEGAH’s design and structure, its overall effectiveness, sustainability of individual interventions, and any lessons applicable to future programming. Today’s interview is expected to last approximately 30-35 minutes (Note: 45-50 minutes if using a translator). Reporting and Confidentiality: The information that you and others provide will be used to write an evaluation report, which will be shared with USAID and made public. However, your responses in this interview will be kept in confidence by the evaluation team. Only the evaluation team – nobody else – will have access to the notes that are taken. Although we will report our evaluation findings in aggregate in the report, your name, position, or other personally identifying information will not appear in our notes or any of our reporting. Interview notes that do not contain any personal identifying information (such as your name) may be given to USAID, if requested. Voluntary Participation: Your participation is voluntary. If you do not want to participate or answer specific questions, you do not have to. Should you choose to participate, please know that you may change your mind and stop the interview at any point. There will be no consequences if you choose not to participate. Risks and Benefits: We do not anticipate that your participation in this interview will result in any risks or direct benefits to you, but your inputs may lead to recommendations that benefit future USAID programs in Indonesia, and therefore, the general public. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the evaluation Team Leader, Adam Reisman, at areisman@socialimpact.com. You can also contact USAID/Indonesia directly by emailing Anders Mantius at amantius@usaid.gov. ● Do you have any questions for us before we get started? [Evaluator to answer any questions] ● Do you voluntarily agree to participate in this interview? [Participant to provide answer to team interviewers, which they will document on paper] _____ Yes _____ No 104 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 5: SCOPE OF WORK Final Performance Evaluation of CEGAH I. Purpose of Evaluation The United States Agency for International Development (USAID) intends to conduct a final performance evaluation for the USAID/Indonesia CEGAH (“Prevent”) activity. The Foreign Aid Transparency and Accountability Act of 2016 defines evaluation as “the systematic collection and analysis of information about the characteristics and outcomes of the program, including projects conducted under such program, as a basis for: (A) making judgments and evaluations regarding the program; (B) improving program effectiveness; and (C) informing decisions about current and future programming.” USAID’s Evaluation Policy establishes two goals for evaluations: accountability and learning. To support accountability, evaluations should measure project effectiveness, relevance, and efficiency; match metrics to meaningful outputs and outcomes; and compare commitments and targets through valid measurement and credible analysis. To support learning, evaluations should test underlying assumptions and strategies to refine design and improve future efforts. The objective of this evaluation with regard to accountability is to determine the degree to which CEGAH achieved its stated goals, why or why not, and how. The objective of this evaluation with regard to learning is to capture lessons that may be applied to current and future programming. The final evaluation will provide USAID with an independent review of CEGAH’s performance. The evaluation is expected to highlight the achievement of expected results, factors influencing the achievement of these results, challenges faced by the activity over the course of implementation, and how the program did or did not address these challenges. The evaluation should also highlight lessons learned and other insights for USAID to consider in the design of future programming. II. Summary Information Contract details include the following: Contract Number: AID-497-C-16-00007 Contract Period: May 31, 2016 to May 30, 2021 Total Estimated Cost: $24.8 million Contracting Officer’s Representative (COR): Diah Januarti Implementing Partner: Management Systems International Chief of Party: Juhani Grossmann USAID estimates that the evaluation will be conducted over a three-month period, March 2020 to May 2020, including recruiting staff, conducting research, and drafting the final report. III.Background Description of the Development Challenge and Context Accountability in governance can strengthen democracy, prevent conflict, and spur economic growth. In Indonesia, rampant corruption and a lack of transparency hinder development and diminish public confidence in government. The United States supports Indonesia to deepen reform efforts, foster transparent governance that is responsive to citizens’ needs, and help a culture of accountability flourish. Despite progress in addressing corruption and broad public support for anti-corruption efforts, Indonesia still faces significant challenges. Transparency International ranks Indonesia 96th out of 180 countries in 105 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report public sector corruption and low scores on World Bank Good Governance indicators point to substantial accountability challenges. As a founding member of the Open Government Partnership, Indonesia has identified openness as the key to unlocking its own potential for public service improvement and economic growth. Under USAID/Indonesia’s Country Development Cooperation Strategy (CDCS) for 2014 to 2020, CEGAH forms one element of the broader Democracy, Rights, and Governance (DRG) project designed to achieve progress towards Development Objective 1, Democratic Governance and Resilience Strengthened, and the Intermediate Result of Community of Accountability Improved. In addition to contributing to USAID’s current portfolio, CEGAH has built on previous USAID investments, including Changes for Justice (C4J), the Educating and Equipping Tomorrow’s Justice Reformers (E2J) activity, and the Strengthening Integrity and Accountability Program (SIAP 1 and 2). USAID is currently designing a new CDCS, which includes a preliminary priority of Government Effectiveness. Transparency and accountability will remain a key component. This performance evaluation will help inform the design of transparency and accountability activities under the new CDCS. Description of CEGAH CEGAH falls under the umbrella of the Democratic Governance Project under USAID/Indonesia’s current CDCS. In particular, CEGAH aimed to contribute to the achievement of the CDCS’s Intermediate Result (IR) 1.1: Community of Accountability Improved and all three of its lower-level results: 1.1.1: Effectiveness of justice sector to prosecute and adjudicate corruption cases increased; 1.1.2: Key GOI corruption prevention institutions strengthened; and 1.1.3: Civil society initiatives on accountability increased. Rationale/Development Hypothesis The rationale of the activity is that USAID support to Indonesia’s “Community of Accountability” will further the county’s national reform agenda and result in a flourishing culture of accountability in Indonesian governance. The activity defines “Community of Accountability” as “ the ecosystem of institutions, rules, and reform entrepreneurs within governing institutions, civil society, and the private sector, as well as encompassing more general public attitudes and behaviors that mutually support an anti￾corruption and transparent and effective government agenda.” Its multi-dimensional, multi-faceted “supply and demand” approach aimed to address systems, processes, and inter-relationship to help civil society and the private sector “demand” greater accountability and protection of citizen rights and to help governing institutions “supply” those public goods to further Indonesia’s democratic consolidation. The development hypothesis posits that focusing strictly on the supply side can rob reform entrepreneurs within government of critical societal support and pressure for their reforms against vested interests, while focusing only on the demand side can create unmet pressure for change, which in turn can foster frustration, cynicism, and erosion of trust in democracy itself. Key Results CEGAH’s highest level result is “Community of Accountability Improved.” The activity aims to reduce corruption by addressing its root causes and bolstering the ability of the Government of Indonesia (GOI) to implement effective counter-measures, thereby strengthening the link between prevention and prosecution and ultimately improving accountability. The objective hinges on the following intermediate results being achieved: 1. Effectiveness of the justice sector to prosecute and adjudicate corruption cases is increased: Component 1 focuses on addressing corruption within Indonesia’s justice sector through capacity building and systemic reform, including: 106 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ● Strengthened Supreme Court ability to gather and analyze data on corruption cases. ● Reduced sentencing discrepancies in corruption cases. ● Improved anti-corruption education in tertiary institutions. ● Improved awareness of judges and attorneys on the linkages between corruption and CVE, complimented by improved capacity to analyze case data to improve sentencing. 2. Key GOI corruption prevention institutions are strengthened: Component 2 focuses on strengthening the capacity of key accountability institutions within the GOI to increase government transparency and prevent corruption, including: ● Strengthened integrated actions to prevent corruption by GOI institutions. ● Strengthened administrative reform efforts, such as strengthening internal controls systems and improving civil service recruitment and promotion processes. ● Enhanced engagement by accountability agencies with the media and civil society on corruption prevention campaigns. ● Strengthened prevention of corruption in budget planning and execution by selected sub-national governments, by following up audit findings. ● Improved policy on corruption related to violent extremism, leading to improvements in GOI monitoring and prevention. ● Improved GOI accountability oversight capacity in issues related to violent extremism. 3. Civil society and the media’s capacity to provide oversight and hold the government accountable are increased: Component 3 focuses on strengthening civil society organizations and the media in efforts to provide oversight and hold government accountable to its citizens. ● Increased government adoption of CSO input on transparency and accountability policies. ● Enhanced CSO advocacy capacity at the national and/or sub-national levels, regarding the accountability sector as well as the areas of health care, education and/or the environment. ● Enhance civil society capacity to research, analyze and advocate on issues related to corruption and violent extremism. These includes a 2018 modification to CEGAH’s Statement of Work to add examination of the nexus between corruption and violent extremism, which is mainstreamed across the three components. CEGAH’s approach addresses the systems, processes, and inter-relationships necessary for ensuring effective public accountability. The activity currently works with more than 19 government ministries and agencies and dozens of civil society organizations. This includes: ● Working with Indonesia’s Supreme Court and Attorney General to improve the transparency of the judiciary’s case management system, reduce sentencing disparities, and promote data exchange among law enforcement agencies. ● Improving anti-corruption education in universities and continuing education programs within the legal profession, including support to Indonesia Corruption Watch (ICW) to develop a free e￾learning platform (Akademi Antikorupsi or Anti-Corruption Academy). ● Supporting Indonesia’s Anti-Corruption Agency, the President’s Office, and the Planning Ministry in developing and implementing Indonesia’s National Anti-Corruption Prevention Strategy. ● Training government institutions such as the Ombudsman on System Investigations and the Ministries of Health and Education in areas such as citizen complaint and reporting systems, corruption vulnerability assessment management, and performance auditing and monitoring. ● Assisting with the development of the National Complaint Handling System (LAPOR!). ● Engaging with universities, civil society organizations, think tanks, and the private sector to strengthen public demand for accountability and contribute to enhanced anti-corruption strategies through evidence￾based advocacy initiatives and data-driven investigative journalism. 107 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ● Promoting public understanding of anti-corruption via the Anti-Corruption Film Festival, including filmmaking workshops, film screenings, and interactive talk shows. Though CEGAH has primarily focused on the national level, the activity also supports activities in selected sub-national locations (namely, North Maluku, North Sumatra, Banten, and East Java provinces) by emphasizing corruption prevention and enhanced accountability in strategic sectors such as health and education. Among its expected outcomes, CEGAH aims to achieve: ● More efficient and effective anti-corruption courts that adjudicate corruption cases consistently fairly and transparently; ● More capable, transparent and accountable government service delivery agencies; ● More proficient and respected CSO networks providing greater oversight of government conduct and performance; and ● Better oversight capacity on the part of the government to monitor and prevent corruption and violent extremism. CEGAH M&E Plan The CEGAH M&E Plan lays out the results framework and the accompanying performance indicators that track the achievement of the key results and is a useful resource of information in reviewing the activity’s performance. IV. Evaluation Questions The final evaluation will focus on the following questions: 1. In what ways did the design and structure of CEGAH help or hinder the achievement of its results? 2. Among CEGAH’s interventions, which were the most successful, which were the least successful, and why? What were the external factors that affected this process? 3. Which interventions and their outcomes have the greatest potential for sustainability after the end of the program? 4. What lessons learned in strategy and approach can and should be applied to future programming? V. Evaluation Methodology The evaluation team should consider a mixed-method evaluation approach to the extent possible. The methodology should combine a review of quantitative data and application of qualitative evaluation techniques to obtain information, opinions, and data from counterparts, contractors, partners, GOI entities, beneficiaries, and other counterparts as appropriate. The approach should be participatory and involve the use of appropriate data collection tools. By using a mixed approach, the evaluation team will gain insight on the impact of CEGAH interventions (mostly from quantitative data collected by the activity and others) and the processes (mostly qualitative information provided by the project staff and key informants) that lead to those impacts. Sequential and iterative approaches should be used to integrate the mixture of methods at various stages of the evaluation. The evaluation team will draft an evaluation methodology/design for USAID approval. The detailed methodology of this evaluation will be described by the evaluation team in the Work Plan; this will include presentation of an evaluation matrix that will explicitly link evaluation questions to particular data collection approaches and data sources. The team should start deciding on specific methodologies for the various activities during the evaluation team’s initial planning meetings. The following essential elements should be included in the methodology as well as the additional methods proposed by the team: 108 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Document Review: USAID/Indonesia DRG will provide the Team Leader with a core list and/or copies of the activity’s Statement of Work (SOW), performance reports, relevant assessment reports, and other key Mission, GOI, and implementing partner documentation before the evaluation begins. The Team Leader will be responsible for expanding this background documentation as appropriate, review, prioritize, and distribute it to other team members for their review. All team members will review relevant documentation before their initial team meetings. Key informant interviews: The team will conduct interviews to obtain feedback from a representative number of stakeholders including (list is not exhaustive): ● Various USAID offices and other USG offices in Indonesia ● CEGAH Implementing partner at both HQ and field level ● Key GOI representatives across various government ministries and agencies ● Judiciary ● Civil society and media ● Private Sector: think tanks, universities representatives of business associations and business networks; individual businesses ● Other stakeholders, e.g. major donors ● Staff from other relevant USAID implementing organizations. Focus Group Discussions: The team will conduct group discussions with a range of representatives across the CEGAH beneficiaries and stakeholders to obtain feedback on the impact of the activity. Discussion groups will include balanced numbers of men and women to the extent possible; in addition, as appropriate to local circumstances sex- or age-segregated discussion groups will be used to promote free discussion by women and men. Site visits: In addition to interviewing stakeholders to solicit views on the overarching evaluation questions, the Evaluation Team will conduct site visits to selected activity locations (national and potentially subnational) to collect and/or validate data and inform findings related to the evaluation questions. Site visit locations will be selected in consultation with the DRG office. Staff from Social Impact (SI), with support from USAID/Indonesia and CEGAH, will assist in organizing logistics for site visits for the evaluation team (e.g., making introductions; CEGAH will not accompany the team on any of the site visits). Data Analysis: Team members will analyze CEGAH documents and information gained from key informant interviews and site visits in order to inform their findings and recommendations. The team will keep a record of meetings that take place and record the summaries of each meeting. Some quantitative analyses may be featured, for example, in the review of CEGAH’s performance monitoring data or in the analysis of the activity’s efficiency. The Evaluation Team will analyze the information collected to establish credible answers to the questions and provide major trends and issues. In accordance with USAID requirements, the evaluation should examine gender issues within the context of CEGAH activities. Where appropriate, the methodology should employ gender-sensitive data collection methods, with an analysis of differential impacts of interventions and their outcomes on men and women. Limitations: Since key informant interviews will be a major source for the validation of information available from the activity, chances of bias are likely. The evaluation should carefully decide on the methodology and select interviewees in a way that the possibility of bias is avoided or reduced to a minimum. The Evaluation Team should also take systematic actions to counter any biases in interpretations of collected data by the team. The methodology narrative of the evaluation report should discuss the merits and limitations of the final evaluation methodology. The evaluation team will design appropriate tools for collecting data from various units of analysis. The tools will be shared with USAID during the evaluation and as part of the evaluation report. 109 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report VI. Existing Sources of Information USAID will provide the evaluation team with access to the necessary documents for the evaluation. These documents will include the CEGAH SOW; annual work plans; quarterly and annual performance reports; monitoring, evaluation, and learning (MEL) plans; and technical reports (Annex 1). USAID will facilitate access to other key documents as identified by the Evaluation Team. The evaluation team will work with USAID to determine the list of key informants to be interviewed. VII. Evaluation Deliverables The evaluation will include the following minimum deliverables: 1. Evaluation Work Plan The evaluation team will develop a work plan that has the methodology, including evaluation questions, main features of data collection instruments, data analysis plan, a discussion of methodological limitations, activities, milestones, associated deliverables, and timeline. Work plan to include planned schedule, logistics, evaluation milestone, site visit schedule, interview lists, and additional data requests. 2. Evaluation In-briefing The evaluation team will conduct an in-briefing with USAID/Indonesia to discuss the draft work plan, arrive at a common understanding of the evaluation’s purpose and methodology and overall plan for implementation of the evaluation. 3. Evaluation Debriefings After the field work is completed, the evaluation team will conduct a debriefing with USAID to present and discuss a summary of findings, conclusions, and recommendations. Additional debriefings, including utilization workshops, may be held with the implementing partner, other U.S. government stakeholders, and GOI counterparts. The evaluation team will share a slide presentation of draft findings, conclusions, and recommendations at least 1 day before the debriefing. 4. Draft Evaluation Report Draft Evaluation Report clearly describing findings, conclusions, and proposed recommendations. USAID will provide comments on the Draft Evaluation Report within 10 days of submission. 5. Final Report Final report that follows USAID standard evaluation report format and branding guidelines, within 10 days of receiving Mission comments on the draft report. The format of the report is presented in Section IX of this SOW. VIII. Evaluation Schedule Performance Evaluation Timeframe Date or Duration Activities Important Considerations/ Assumptions Feb 24 – Mar 23 Recruitment of Team Leader and team members Mar 16 – 27 Document Review by Team Leader/SI and eventually by all team members of the Evaluation Team All team members are contracted by Mar 23 110 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Date or Duration Activities Important Considerations/ Assumptions Mar 24 – Apr 6 Submission of Inception Report by April 6 (COB) Preparation of the work plan and evaluation design Local holidays, season/weather, transport availability. NB: March 25, 2020 is an American and Indonesia Holiday Apr 7 – 16 Feedback and comments received from USAID by Apr 16 (COB) USAID review and approval of the work plan NB: April 10 (Good Friday) Apr 7 – 30 Instrument Design and finalization of work plan based on USAID feedback Development of instrument and submission for IRB review and approval (by April 16). Apr 21 (COB) Submission of Instrument for USAID input and feedback, and the revised Work Plan Apr 30 The final Work Plan approved by USAID May 29 In-briefing Availability in the Mission May 29 - Jun 1 Finalization of data collection instrument Jun 2 – 19 Data collection Number of sites, methods, sectors, etc. Jun 19 – Jun 25 Data analysis and findings Amount and type of data quantitative and qualitative Jun 26 Debriefing – rough findings based on field notes and observation during field work Availability in the Mission Jun 29 – Jul 21 Report writing Length of time to meet report requirements and any additional requests/products. Jul 21 (COB) Submission of Draft Report to USAID Jul 21 – 31 Feedback and comments received from USAID by Jul 31 (COB) USAID review of draft report Length of time for all relevant stakeholders to read and provide feedback. Aug 1 – 7 Incorporation of USAID comments and submission of final report Length of time to reconcile feedback from varying stakeholders and comply with formatting requirements. Aug 7 (COB) Submission to USAID Fully address USAID comments 111 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Estimated LOE Activity LOE Team Leade r Eval/Tec h Specialis t Eval/Tech Specialist Review of project documents Preliminary analysis of CEGAH results 4 2 2 Travel to Jakarta 1 - - In-country Evaluation Team Meetings Draft Work Plan, including evaluation methodology, proposed schedule, logistical arrangements, team member responsibilities, evaluation milestones. Methodology to include list of tools, site visit schedule, interview list, additional data requests, evaluation design matrix linking evaluation questions to data sources, analysis) 6 6 6 In-brief with USAID Review key issues with USAID, CEGAH, and key stakeholders 1 1 1 Develop interview questionnaires / other tools to be used in data collection 3 3 3 Key informant interviews, FGDs, site visits, other data collection 16 16 16 Analysis of data, including gender disaggregation Development of initial findings 5 5 5 Final Briefing with USAID 1 1 1 Travel from Jakarta 1 - - Draft report and submission to USAID 16 9 9 Incorporation of USAID feedback Submission of Final Report 5 2 2 TOTAL 59 45 45 SCHEDULING AND LOGISTICS Funding and Logistical Support The proposed evaluation will be funded and implemented through Monitoring, Evaluation, and Learning Support (MELS) mechanism, under the management of the MELS COR in the USAID/Indonesia Program Office. The evaluation must be independent from the CEGAH implementing partner, its staff, and subcontractors. MELS will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate consultants. They will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. The evaluation team should be able to make all logistic arrangements, including the vehicle arrangements, for travel within and outside Jakarta and should not expect any logistic support from the Mission. The team should also make their own arrangements on space for team meetings, and equipment 112 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report support for producing the report. USAID and the CEGAH implementing partner may, in some cases, assist in arranging meetings and site visits. However, they shall not participate in any meetings or interviews related to this evaluation. The DRG Office drafted the evaluation SOW with review by the Program Office. The technical team will provide support to the evaluation by ensuring that the implementing partner, GOI and other CEGAH stakeholders are aware of the evaluation’s purpose, the timeline, and the scope of their expected engagement; ensuring that the evaluation team has all relevant background materials detailed in the SOW; providing technical input during the review of the evaluation design and draft evaluation report; and participating in discussion of post-evaluation action planning. IX. Final Report Format The total pages, excluding references and annexes, should not be more than 35 pages. The following content (and suggested length) should be included in the report: Table of Contents 1. Abstract - the evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. 2. Executive Summary - concisely state the project purpose and background, key evaluation questions, methods, most salient findings, and conclusions (plus recommendations and lessons learned as appropriate) (3-4 pp.). 3. Evaluation Purpose - purpose, audience, and synopsis of task (1 pp.). 4. The Development Problem - highlight the development problem and the context in which intervention took place (1 pp.). 5. USAID’s Response - brief overview of CEGAH activity, and USAID CDCS, and implemented in response to the problem (2-3 pp.). 6. Evaluation Questions - state the evaluation questions as identified in the SOW (0.5 pp). 7. Methodology - describe evaluation methods in detail, including limitations (strengths, constraints, and gaps related to issues such as data availability; any potential bias, such as measurement, interviewer, response etc., must be disclosed and implications on conclusions drawn. A summary of methodology can be included in the body of the report, with the full description provided as an annex. (2-4 pp.). 8. Findings and Conclusions - describe and analyze findings for each objective area using graphs and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcome (12-17pp.). 9. Recommendations - prioritized for each key area or evaluation focus; should be separate from conclusions and be supported by clearly defined set of findings and conclusions (3-4pp.). 10. Lessons Learned39- provide a brief of key technical and/or administrative lessons that could be used for future project or relevant program designs (3-5pp.). 11. References - including bibliographical documentation, meetings, interviews and focus group discussions. 12. Annexes – to include statement of work, documents reviewed, evaluation methods, data generated from the evaluation, tools used, interview lists and tables, conflict of interest forms for each evaluation team member, and summary information about evaluation team members including qualifications, experience, and role on the team. Annexes should be succinct, pertinent, and readable. 39 This evaluation focuses on more adaptive management and learning focused approach. 113 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report The final report should have the following criteria to ensure its quality: ● The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. ● The evaluation report shall address all evaluation questions included in the scope of work. ● The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the COR. ● The evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. The Final Evaluation Report should be submitted in Word and PDF formats and 12-point type font should be used throughout, with page margins one-inch top/bottom and left/right. The Mission should receive five hard copies of the final version of the report and an electronic copy of the final report. The evaluation team will submit the Final Evaluation Report to USAID’s Development Experience Clearinghouse. X. Evaluation Team Qualifications and Composition All evaluation team members will be required to submit written disclosure of conflicts of interest. The Evaluation Team shall consist of three (3) individuals with substantial experience in transparency, accountability, evaluation, and learning approaches. Collectively, the evaluation team should possess the following qualifications: ● Experience in the evaluation of large, complex projects, including skills in interview techniques and data collection and analysis. ● Expertise in transparency, accountability, justice and bureaucratic reform, and related areas. ● Understanding of the Indonesian context, including Bahasa Indonesia language skills (or the ability to engage translators/interpreters as necessary). ● Organizational and logistical capacity. 1. Team Leader (International Staff) - The Team Leader will be responsible for direct liaison with USAID, managing evaluation team personnel and assignment responsibilities, and producing high-quality evaluation deliverables, including the evaluation methodology and work plan, evaluation presentations, and the draft and final reports. The Team Leader will work under the direct supervision of the Chief of Party for Indonesia Monitoring and Evaluation Support Project (implemented by Social Impact) and will coordinate with, and be supported by, Social Impact evaluation specialists and research assistants. Qualifications: ● Advanced degree in a relevant discipline, such as research, international development, public administration, political science, or a related democracy and governance field. ● Minimum of 10 years of experience in development, with a significant experience in one or more of the following areas: transparency, accountability, and bureaucracy reform. ● Significant performance evaluation experience, including prior experience designing data collection instruments. 114 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ● Experience analyzing and synthesizing large amounts of data and information into meaningful conclusions and recommendations for strategic input and implementation. ● Have a proven track record in leadership, coordination, and evaluation of development projects. ● Have excellent writing, oral presentation, and organizational skills. ● Prior work experience in Southeast Asia is required, and prior experience in Indonesia is preferred. ● Familiarity with USAID democracy and governance programming approaches is an advantage. 2. Evaluation/Anti-corruption Specialist (Indonesian Staff). This position will work under the supervision of the Evaluation Team Leader and will be responsible for contributing to all aspects of the evaluation, including supporting the development of the evaluation’s methodology, data collection and management, data analysis, and writing the draft and final reports. Qualifications: The evaluation technical specialist will be selected based on a mix of the relevant skills that will be required for the assignment: ● Advanced degree in a relevant discipline, such as: public administration/policy; research skills; international development, political science, law school or a related DRG field. ● Minimum of seven years of experience in development, with a significant experience in one or more of the following areas: accountability, transparency, and anti-corruption. ● Minimum of five years of performance evaluation experience, and/or experience designing data collection instruments, conducting structured interviews and managing qualitative data. ● Knowledge of accountability and transparency and justice sector issues and programming approaches; ● Familiarity with USAID democracy and governance programming approaches is an advantage. 3. Evaluation/Bureaucracy Reform Specialist (Indonesian Staff). This position will work under the supervision of the Evaluation Team Leader and will be responsible for contributing to all aspects of the evaluation, including supporting the development of the evaluation’s methodology, data collection and management, data analysis, and writing the draft and final reports. Qualifications: The evaluation technical specialist will be selected based on a mix of the relevant skills that will be required for the assignment: ● Advanced degree in a relevant discipline, such as: public administration/policy, international development, political science, or a related DRG field. ● Minimum of seven years of experience in development, with a significant experience in bureaucracy reform. ● Minimum of five years of performance evaluation experience, and/or experience designing data collection instruments, conducting structured interviews, and managing qualitative data. ● Knowledge of accountability and transparency and justice sector issues and programming approaches. ● Familiarity with USAID democracy and governance programming approaches is an advantage. 115 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 6: LIST OF CEGAH REQUIRED TASKS RT 1: Use data tools to increase tracking and monitoring of corruption and related criminal cases. RT 2: Support interventions to better use available open data (including the Direktori Putusan court decision database and the SIPP case tracking system) to facilitate better quality corruption judgments. RT 3: Support the adoption and implementation of the International Framework for Court Excellence in the Special Courts on Corruption Crimes. RT 4: Support performance audits within the Supreme Court focusing on standards of court services, using the International Framework for Court Excellence and/or the Ministry of Civil Service Management and Bureaucracy Reform’s Assessment System for Bureaucracy Reform. RT 5: Technical assistance and training to improve the SIPP case tracking system to include more sectoral and sub-sectoral tags (i.e. corruption cases involving the environment in addition to other classifications for the education and health sectors) to improve research and analysis of cases by the GOI and the public. RT 6: Support the Supreme Court and AGO to develop and implement common sentencing guidelines for corruption cases. RT 7: Support effective coordination between the KPK, AGO and other law enforcement entities to strengthen corruption indictments. RT 8: Support the Supreme Court, including the Special Courts on Corruption Crimes, to properly adjudicate corruption cases in accordance with the laws. RT 9: Facilitate the participation of CSOs in the development and implementation of the common sentencing guidelines for corruption cases. RT 10: Support an outreach strategy to campaign and mainstream the necessity for integrating anti￾corruption modules that emphasizes on evidence-based and experiential learning into tertiary education curricula all across Indonesia through the Ministry of Higher Education and Research and Technology. This task must be carried out in collaboration and/or coordination with other USAID/Indonesia higher education activities. RT 11: Support the development and strengthening of university/CSO anti-corruption networks to strengthen anti-corruption legal education clinics (including using paralegals), and to produce high quality research and analysis on judicial reform (including the use of open data sources such as Direktori Putusan and SIPP), and to develop outreach strategy to further disseminate the research and analysis. RT 12: Support the building of relationships between private sector entities and CSOs and NGOs to further promote anti-corruption education within and beyond universities. RT 13: Support a comprehensive prevention action strategy by GOI accountability agencies including prioritization and implementation, with clear engagement of the KPK’s coordination role and authorities so it would be linked better to prosecution. This will include support of GOI efforts to implement Inpres No. 7/2015 and the National Integrity System (SIN). 116 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report RT 14: Support key accountability agencies to strengthen compliance with existing corruption prevention mechanisms, for example the annual wealth reports (LHKPN) and gratuities reports by government officials. RT 15: Support stronger integrated action among the key accountability agencies to prevent fraud, waste and abuse in flagship government programs such as national health insurance (run by BPJS), health subsidies (Kartu Sehat), education subsidies (Kartu Pintar), and/or forestry and environmental issues (such as: overlapping mining licenses; illegal logging; or illegal, unreported and unregulated [IUU] fishing) RT 16: Provide capacity building and training to enhance these accountability institutions’ professional knowledge in order to leverage capabilities and tools to prevent corruption and related crimes. RT 16 (BPK): Provide capacity building and training to enhance these accountability institutions’ professional knowledge in order to leverage capabilities and tools to prevent corruption and related crimes. RT 16 (KPK): Support the Corruption Eradication Commission in Drafting Anti-Corruption Law to Accommodate UNCAC and Preventing Unclear Provisions in the Current Law RT 17: Support accountability agencies in improving the quality and implementation of their whistleblower policies. RT 18: Support KemenPAN-RB, the Ombudsman and the Office of the Presidential Staff in the development and implementation of the National Complaint Handling System as the flagship program under Component 2 to deter and prevent corruption as well as to improve quality of agencies’ services delivery, including the integration of previous UKP4-managed efforts using the OGP – LAPOR application. RT 19: Support KASN and other GOI efforts to implement Law No. 5/2014, including transparent and merit-based recruitment and promotion of civil service officers at all levels. RT 20: KemenPAN-RB-led efforts to implement administrative reforms within ministries/agencies. This could include promoting annual ethics training, implementing fraud control mechanism, and streamlining conflict of interest regulations and/or codes. RT 21: Support collaboration of key government accountability agencies such as KemenPAN-RB and BPKP to encourage ministries/agencies at the national and sub-national levels to reduce red tape and improve the quality and efficiency of public service delivery, thereby reducing opportunities for corruption. RT 22: Support KemenPAN-RB in better measuring government performance by: aligning it with budgeting and programming; adopting simplified measures that are more pragmatic, less duplicative, and more transparent; and strengthening coordination among the institutions that work on these issues (KemenPAN-RB, Ministry of Finance, BAPPENAS, BPK and BPKP). RT 23 Support government efforts to promote data transparency, including establishment of an effective nationwide information support system for public services. RT 24: Support key accountability agencies’ public communication departments to promote their efforts in fighting and preventing corruption. 117 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report RT 25: Support accountability agencies to develop collaborative prevention efforts that involve media and CSOs, to follow up on audit findings and recommendations, highlight and analyze bureaucracy reform challenges, and promote social auditing of key public services. RT 26: Promote better public access to information on government policies, budgets, expenditures, and evaluations. RT 27: Support media and CSOs to develop data and conduct research to directly assist GOI accountability agencies with more effective inter-agency collaboration and corruption prevention. RT 28: Support accountability agencies to work with selected local governments to introduce a comprehensive internal controls framework that adheres to international standards, in order to reduce opportunities to engage in corrupt practices. RT 29: Support selected local governments to improve public services by following up audit recommendations, particularly regarding GOI flagship programs in key sectors such as education, health care, and the environment RT 30: Assist local government agencies to adopt citizen charters to inform the public about their rights and provide detailed and clear information about the requirements for obtaining a public service. RT 31: Support dialogue, consultations and strengthening relationships/network among civil society, private sector, media, and government regarding transparency and accountability policies and practices. RT 32:Direct support to GOI or through CSOs in research, analysis, and dissemination of findings to support GOI in evidence-based decision-making regarding transparency and accountability policies, including (when appropriate) gender-disaggregated data on the impact of those policies and of corrupt practices. RT 33: Technical assistance and training to CSOs to improve their policy advocacy capacity, including regarding gender aspects of accountability. RT 34: Technical assistance and sub-grants to civil society organizations to support their corruption prevention, monitoring and research work, including the gender aspects of this work. RT 35:Provide targeted training and capacity support for women’s organizations and their leaders to improve institutional capacity and efficacy regarding anti-corruption advocacy. RT 36: Support the integration, improvement, and strengthening of CSOs/networks on accountability, legal reform, and anti-corruption at national and sub-national levels. RT 37: Support private sector associations that accommodate principles of good business practices, accountability, and corporate social responsibility. RT 38: Support CSO initiatives to identify and advocate for Indonesian institutions that promote accountability support initiatives to improve accountability and law enforcement in sectors such as health care, education, and the environment. RT 39: Support initiatives to improve accountability and law enforcement in sectors such as health care, education, and the environment. 118 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report RT 40: Training for Judges of the Corruption Court to better understand VE issues and their links with corruption. RT 41: Supporting the AGO and Supreme Court on Corruption and Terrorist case management, for prosecutors and judges to increase their analytical skills for sentencing improvement and guidelines. RT 42: Engagement with the Office of the President’s Staffs (KSP) to Increase the Capacity of GOI in Monitoring of Corruption Extremism Links. RT 43: Providing input to Government through tracking the linkage between accountability and the intolerance survey conducted by CSO. RT 44: Supporting the GOI agencies such as the Ministry of Civil Service Management and Bureaucratic Reform/KemenPAN-RB and the Civil Service Commission/KASN in monitoring and preventing the radicalization of civil service prevention program. RT 45: Strengthening the ability of financial oversight bodies (Financial Intelligence Unit/ PPATK, Financial Services Authority/OJK, and Corruption Eradication Commission/KPK) to track illegal financial flows, especially financial flows that relate to corruption and extremism activities. RT 46: Capacity development for the Inspectorate Generals (IGs) at the Ministry of Law and Human Rights (KemenkumHAM) on oversight of prison accountability as related to violent extremism; the IG at the Ministry of Manpower in preventing the radicalization of migrant worker programs and; the IG at the Ministry of Education in strengthening its monitoring capacity on the intolerance at public schools (secondary schools) and universities. RT 47: Supporting the Indonesian external and internal oversight bodies such as the Ombudsman Office (ORI), the Supreme Audit Agency (BPK), and the Development and Finance Oversight Agency (BPKP) and the IGs of the government in strengthening capacity to monitor government allocations of social funds (bantuan sosial/bansos) at the regions/districts that have potential violent extremist supporters. RT 48: Work with Islamic and Public Universities to conduct research and mapping for current extremism and its funding support and its support bases for awareness on the terrorism/extremism and for improving the alert system, in the high-risk and fragile areas. RT 49: Joint efforts with leading Indonesian public opinion research firms to standardize language of CVE￾related surveys. Important quality control measures that will allow compatibility and comparability among various donor-funded research efforts. RT 50: Research into the linkages between corruption in the bansos/dana hibah and the funding of extremist organizations especially in the high-risk and fragile areas. RT 51: Support CSOs in CVE advocacy efforts for social funds that are distributed in areas that have potential violent extremist supporters. RT 52: Supporting the CSOs in reviewing the Local Government regulations related to public services that discriminate against certain people. 119 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report ANNEX 7: COMMENTS FROM USAID CEGAH ON THE EVALUATIONS REPORT AND THE RESPONSES FROM THE EVALUATION TEAM USAID Indonesia appreciates the efforts of Social Impact, Inc. (SI) and the Evaluation Team in conducting the final performance evaluation for the USAID CEGAH project. USAID Indonesia also appreciates the feedback from USAID CEGAH on the evaluation report. The annex documents the comments from USAID CEGAH on the evaluation report, submitted to USAID Indonesia as a memorandum, as well as the responses from the Evaluation Team to the comments. MEMORANDUM FOR : Mr. Anders Mantius, COR, USAID CEGAH Ms. Fitria Wahid, COR, Monitoring Evaluation and Learning Support to USAID/Indonesia FROM : CEGAH SUBJECT : Comment on the Draft Social Impact Technical Evaluation Report on USAID CEGAH, dated August 10, 2020 DATE : September 21, 2020 This memorandum is submitted with reference to the Draft Final Evaluation Report of USAID CEGAH prepared and submitted to USAID/Indonesia on August 10, 2020 by Social Impact, Inc. (SI) under the project, Monitoring Evaluation and Learning Support to USAID/Indonesia. In an email dated September 11, 2020, CEGAH COR Anders Mantius agreed that CEGAH could provide a quick review of the report to ensure that any concerns pertaining to factual elements stated in the report are properly addressed. Accordingly, this memorandum is submitted for USAID/Indonesia’s consideration. Summary CEGAH appreciates this opportunity to both review the report and learn from its conclusions. Many of the conclusions are valuable both for CEGAH’s remaining implementation period and the design of follow￾on anti-corruption interventions. We appreciate that conducting an evaluation in times of COVID-19 is exceptionally difficult and therefore hope that our comments are accepted in the spirit that they are provided: to improve the overall effort and therewith future anti-corruption programming. Overall, we find that the utility of some findings and recommendations is constrained by both the limitations in methodology and a lack of contextual awareness and background information. SI Response: We address these specific points below. Comments on the Overall Evaluation Methodology Regarding the overall methodology, CEGAH understands the difficulties encountered by the SI evaluation team (ET) in obtaining information and gathering data given the present COVID-19 pandemic restrictions on travel, mobility and personal interaction. It was in this spirit that upon the ET’s request, CEGAH’s senior management team provided full support and cooperation to SI, to ensure that it can gain access to information and data that could properly inform the evaluation and generate accurate and reliable results. 120 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report SI Response: We appreciate the CEGAH team’s support. Formulation of the sample frame. Because of the sizable number of required tasks (RTs), it is not clear how the key informants (KIs) were identified from the larger beneficiaries and grantees list for the evaluation sample frame, and how representative was the sample frame of the population and the range of CEGAH’s components, clusters and geographic scope. The same concern also applies to the process to identify the grantees and beneficiaries to be part of the online survey pool, where only 26 responses were returned out of a total of 126 selected potential respondents to whom the survey link was sent. Given the broad coverage of CEGAH components and its range of RTs, it is possible that with these limitations, not all of the components and tasks may have been fully assessed and evaluated by respondents who were fully knowledgeable of the RTs they were asked about. SI Response: The ET provides a detailed explanation of the approaches employed to identify both key informants as well as survey respondents within the report’s Evaluation Design and Methodology section. Please see the “Data Collection” and “Sampling” subsections from pages 10 – 14. For both its KII and survey samples, the ET drew from the complete list of beneficiaries and grantees representing the full range of RTs that CEGAH provided to the ET. As noted in the report, the ET purposively selected RTs and their corresponding beneficiaries and grantees within the list for the qualitative portion of the evaluation. Such an approach targets those with diverse and important perspectives and is the most common approach taken for qualitative methods. While it was not possible to talk to every single stakeholder, the ET is confident in having reached saturation with its qualitative methods, ensuring that the findings represent a broad range of perspectives. For its quantitative data collection, the ET sent the survey to all beneficiaries on the list as well as additional beneficiaries from a small group of purposively selected RTs that represented CEGAH’s three primary components, were implemented in different regions of the country, and consisted of beneficiaries who have participated in CEGAH-supported trainings relatively recently. The ET encourages CEGAH to listen to the feedback provided by these qualitative and quantitative responses from stakeholders. Incomplete information to assess relative success or failure of project interventions. Among other questions, key respondents were asked which CEGAH interventions were the most successful and which were the least successful. They were also asked what were the external factors that affected this process. In the report, the ET acknowledged the challenge inherent in the number and diversity of CEGAH’s “required tasks”, and they described several steps to ensure that fairly reliable data was generated in response to the questions stated above. The steps undertaken by the ET to address this challenge includes: a) narrowing the sample by excluding certain tasks that were ended earlier; b) establishing a three-part criterion to objectively determine the answers to the evaluation questions; and c) triangulating the probable answers to the questions by taking into account the answers of grantees and implementing partners, both using a response count and qualitative responses, and application of the three￾part criterion. While CEGAH appreciates the efforts of the ET to triangulate the most accurate answers across the various research approaches deployed, the questions that asked which interventions where most successful and least successful obviously required a certain level of knowledge and familiarity of the success of specific tasks in relation to others. Developing reliable results using this approach can be quite difficult. This is particularly relevant with regards to more niche interventions that operate in an exceptionally difficult environment, such as the CVE interventions. 121 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report It would be helpful if the ET clarified its measurement of “success.” Requiring respondents to assign programming into either successful or unsuccessful buckets suggests that there is no in-between. Also, it would be helpful if the ET could clarify whether all respondents were encouraged to pass judgment on all programming, including some that they were not involved in or for which they had limited information. If this was this case, there is a serious risk that respondents downgraded success for those interventions they knew little about. This could explain why CVE, which is no ministry’s first job (not even of the Ministry of Religious Affairs), fared particularly poorly. At the same time, asking the Pancasila agency about CVE engagement would probably have resulted in highly positive results, demonstrating the weakness of such inherently subjective ratings. Overall, we fear there is a risk that the evaluation and its findings may have relied heavily on perceptions of persons without the relevant information, and are thus unable to adequately judge the relative merits of different tasks or interventions. SI Response: We appreciate CEGAH’s concerns. As noted above, however, our sampling strategy included diverse perspectives across the breadth of CEGAH’s RTs, which the ET used to provide nuanced findings. Per evaluation question #2, the ET identified those activities that, based upon its identified criteria, were considered most/least successful. In identifying these activities, the ET didn’t solely highlight their positive components (for the most successful RTs) or their challenging ones (for the least successful RTs). Rather, as with SP4N LAPOR!, the ET detailed both its successes as well as several challenges faced by this intervention – while still being responsive to the evaluation question. Regarding the CEGAH team’s concerns about CVE programming, the ET’s findings were based on a variety of perspectives from stakeholders that had directly engaged in these activities, including beneficiaries, grantees, and implementing partner staff directly involved in CVE tasks. Use of contribution analysis. The ET listed contribution analysis as one of the data analysis methods deployed to identify key findings, draw conclusions and make recommendations (p. 15). It is unclear however from the draft report how contribution analysis may have been used. The ET has not ascertained and traced the logic of the relationship between the TOC and the 52 RTs, and how this relates to their evaluation design. The bias that may manifest in this situation is that the results were more likely generated by triangulating perceptions from stakeholders and validating this with quantitative data from the online survey and other sources, rather than analysis that would have permitted assessment of relative contribution of individual RTs or clusters of RTs. SI Response: The ET has revised this sub-section. We attribute the challenge of utilizing contribution analysis as part of our analytical approach in part to the difficulties encountered in utilizing some of CEGAH’s indicators – e.g., at the outcome level – to attempt to properly assess whether changes had occurred or not within CEGAH’s logical framework’s outputs and outcomes due to its RTs (please see “Challenges measuring CEGAH’s impact” subsection under EQ1). Comments on the Executive Summary Content Balanced representation from various counterpart government institutions in trainings. On p. 4 and p. 43, the section on expanding training opportunities suggests that having more judges participate in one training would have been more beneficial. CEGAH agrees that a more balanced participation in that particular instance would have been desirable. This is why CEGAH organized multi-agency trainings where possible, and expended tremendous efforts to ensure participation in accordance with plans. However, the following realities are relevant: a) the nomination of participations for any training, especially multi￾agency trainings with several senior government officials, such as judges, are notoriously unpredictable in 122 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Indonesia, b) due to technical, substance, funding, and administrative reasons, an even distribution of participants in multi-agency trainings is not always possible or even desirable, and c) that this instance is not representative of the project’s overall approach. However, it is important to further note that actual composition and representation from various participating agencies may be affected by availability of staff to be trained, nature and subject matter of the training and selection decisions by the hosting agencies. SI Response: The ET is glad that this evidence provided by stakeholders was helpful for CEGAH. No edits to the report required. Suggestion for a stronger learning culture. On p. 5, the ET urges the creation of a stronger learning culture. It would be appreciated if the ET could specify what kind of events the ET suggests, beyond the already existing efforts to share knowledge across the project through grantee coordination meetings, multi-grantee presentation of project and research findings, workplan and quarterly report sharing, multi￾agency training and events, the regular newsletter, and sharing of grantee events to encourage cross￾participation. SI Response: The nature of these learning sessions is explored in greater detail in the full Recommendations section in the body of the report. We have also amended this recommendation to clarify that such sessions could be integrated into current CEGAH events or be organized as stand-alone sessions. Passing of CVE expertise. On p. 6, the ET suggests the passing of CVE expertise to HARMONI. CEGAH is pleased to report that this has already taken place. SI Response: Thank you for this additional information. No edits to the report required. Comments on Findings and Conclusions Observations from stakeholders that CEGAH has too many tasks and too little time and resources devoted to each task. On p. 19 and pp. 43-44, the ET refer to grants being too short in duration as a shortcoming. We respectfully submit that the annual program planning cycle and administrative requirements meant that the period of performance of grants typically were a little less than a year. We did not have any grants of a 3-4 months duration except for a tiny number of narrowly focused grants seeking to implement a specific activity. However, the project has issued many no-cost extensions, and dozens of cost extensions for high-performing grantees. This approach is praised on the following page (p. 20), including by all grantees and beneficiaries. We would therefore request the ET to clarify how these integrated approaches across the project can be seen as both good and negative. SI Response: The ET discussed the utilization of follow-up grants to enable “next steps” or additional progress among some RTs. These were seen as useful by interviewed beneficiaries and grantees, but most if not all indicated that longer RTs (versus one shorter RT followed by a follow-on RT) would have been preferable – e.g., these were viewed as more seamless, easier for building capacity, and simpler for tracking impact. Duplicative programming between Component 1 and ICW Akademi Antikorupsi. With reference to remarks on p. 20, we submit that the original design for ICW’s Akademi was targeting activists, not university students, while the Component 1 engagement focused on incorporating anti￾corruption education into the formal university curriculum. These are quite different purposes. As the Component 1 engagement proved challenging, and ICW’s Akademi attracted substantial interest from 123 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report educational institutions of higher learning, we increased the funding to ICW several times to support the rapid widening and deepening of the Akademi to meet demand. Not all programs are equally successful, so focusing on the programs that have traction and adjusting strategies to match realities on the ground seemed like a prudent approach. SI Response: The relevant part of this finding – i.e., the Akademi’s original design for this programming – has been clarified within the report. Please note that the modified content still reflects the ET’s evidence that there was increasing recognition over time that these two tasks were performing similar work. In the end, the evaluation found CEGAH’s actions in this case to be positive – ending one program that was performing one activity and continuing/expanding another that was doing similar work but was able to achieve better results. Multitude of required reports. On p. 21, the ET refers to complaints about a multitude of activity reports. This is confusing since our grant agreements were FAAs and thus did not require activity reports unless they were included by the grantee during the agreement negotiations. The reporting requirements were constantly scrutinized and simplified while still ensuring accountability of grantee performance and adherence to USG regulations. The CEGAH grants team spent countless staff hours supporting grantees who struggled to write reports that were required under FAA guidelines. This has been a constant CEGAH policy, implemented to this date. SI Response: It would be helpful for the CEGAH team to clarify the extent to which they required reporting (of any kind) on its FAAs. In this feedback, the CEGAH team indicates that reports were not required under its FAAs but then later says that countless hours were spent supporting grantees in writing required reports. This feedback is contradictory. The ET acknowledges CEGAH’s efforts to simplify its reporting requirements (indeed, one of the report’s findings is that some grantees/beneficiaries found its administrative requirements and work culture to be relatively flexible and simple in comparison to their work with other donors). However, a sizeable separate group of grantees/beneficiaries stated that this reporting burden could be high. The required reports respondents spoke of were not activity reports, per se, but reports based on grants’ milestones, for example, as well as the activities and outputs within these milestones. We have provided clarifying language within this paragraph in the report. Higher-level indicators. On p. 22, the ET refers to the type of indicators. CEGAH respectfully submits that the indicators mentioned, including the Rule of Law Index listed by the ET, are requirements in the CEGAH contract spanning all the way back to the RFP stage. SI Response: We acknowledge that some of these metrics are required (others were “suggested,” according to CEGAH’s revised scope) and/or have been in use since the beginning of the project, but that does not change the ET’s findings that several of these are not effective as outcome indicators – e.g., the ROL index would be more suitable as a context indicator, as the results of an outcome indicator are supposed to be attributable to the activity. Comments on Specific Program-Related Findings CEGAH project design. In the summary (p. 2), the ET both praise and lament the project design and its magnitude of tasks. Having a fixed budget divided by a high number of required tasks (which is cited as good) leaves modest funding per task (which is seen as bad). It is unclear what lessons are to be drawn from such contradictory conclusions, which it would be helpful to specify and explain. 124 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report SI Response: The ET’s finding did not state that a high number of lower-value tasks was good. However, stakeholders did say they appreciated CEGAH’s design for building/strengthening a network of supportive actors across the GOI and judiciary; for the flexibility to work with many stakeholders across ministries; and for the opportunity to build relationships between and within these ministries, agencies, commissions, offices and courts, among other findings. A lesson to be drawn from this would be seeking ways to maintain this network while supporting a smaller number of longer tasks. The ET discusses specific solutions to accomplish these aims in both the “Lessons Learned” and “Recommendations” sections of the report. CVE interventions. With regard to the ET’s findings on the CVE interventions as the least successful, CEGAH appreciates the feedback and shares an appreciation for some of the program’s challenges. It is worth noting that by its very nature, inter-disciplinary programming is high-risk programming and none more so than when two exceptionally complex and sensitive fields, corruption and extremism, are addressed jointly. CEGAH’s program proposal in this field stressed the inherent challenges associated with studying a new field (the very definition of innovative programming). The various intersections between the two fields were presented as hypotheses which were going to be tested during the program. This is precisely what the program did and as the ET correctly observes, few of the hypotheses bore fruit. According to the ET, this means the value of the research was “relatively limited” (p. 3). We argue, however, that establishing a lack of correlation does not constitute a lack of program success – disproving a relationship can be as valuable as proving one, especially since it will allow future USAID programs from avoiding similar engagements. SI Response: Thank you for this additional context. No edits to the report required. The second finding, that there were few supporters of the program in government, is hardly surprising: the political environment around extremism in civil service has become increasingly toxic. It was never intended as a consensus-based program among all government agencies. Rather, we worked with those that expressed an interest. During the course of the program, six ministries were enthusiastic to use TEPAT for their new civil servant recruitment in 2019. SI Response: Thank you for this additional context. No edits to the report required. The final finding, that the TePAT engagements “were misused by some agencies and raised human rights concerns” (p. 2, Abstract) is a strong criticism that requires far more substantiation by the ET. Subsequent narrative language in the SI report is far softer, referring to “a potential for misuse,” (p. 3, Summary). We respectfully request that the various sections of the document be aligned appropriately and reflect the full picture, which is as follows: SI Response: This language has been modified in the abstract of the report, but the overall finding remains; a finding that was supported by interviewed CEGAH stakeholders and the activity’s own documentation (Q2 2020 Quarterly Report: “…USAID CEGAH discontinued its support for TePAT in fear of possible misuse of the tool by those who do not give sufficient attention to possible human rights consequences during its application.”). Context. The civil service entrance requirement existed before CEGAH’s arrival and included questions on Pancasila before CEGAH ever entered the picture. At the time of the design of the program, the GOI 125 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report together with CEGAH identified a need to review the questions and improve these. Realizing the growing concern about civil servant radicalization, CEGAH set out to do this with substantial GOI and USAID encouragement, engaging moderate Islamic scholars from Nahdlatul Ulama. SI Response: Thank you for this additional context. No edits to the report required. Relevant political developments and CEGAH action. Almost as soon as CEGAH started engaging on this issue, the broader political environment shifted significantly, with an increasing stand-off between the President and his team, and the civil service (a majority of which had voted for his opponent in the presidential elections). CEGAH was neither the source nor driver of this stand-off and remained engaged in the process precisely because it saw the potential for abuse. It employed human rights experts to provide input and frequently consulted with USAID and outside CVE experts. When the program saw that it could no longer influence the deterioration of the situation from inside and was risking giving legitimacy to a process with uncertain outcomes, it withdrew its support for TePAT. All this was done in close coordination with USAID and the final withdrawal was in fact taken proactively by CEGAH. SI Response: Thank you for this additional context. We are not disputing that these developments took place, but TEPAT nonetheless raised human rights concerns and, in some cases, was used as the sole criteria by which to disqualify applicants. Thus, the finding that TEPAT was one of the less successful CEGAH RTs remains. Risk was factored in and duly anticipated. The program clearly recognized the risk inherent in this program. This risk is not unusual, however. All anti-corruption programming includes risks for human rights abuses. Prioritizing the rule of law vs. human rights is a fine balance in dynamic political environments. There is, for example, a real risk of KPK turning into an agency that is used for political retribution. Will this mean that all of USAID programs supporting KPK were contributing to human rights abuses? CEGAH urges the ET to consider the context when describing the activity and include relevant circumstances, including the efforts taken by CEGAH to limit the risks. Alternatively, if context is not taken into account and program accurately assessed in relation to relevant circumstances, this will provide a disincentive to innovative programming in the future. SI Response: The ET acknowledges that CEGAH was cognizant of the risk with TePAT and took steps to attempt to mitigate it (the team has added a sentence to this effect within the draft). TePAT was the sole example brought up by key informants of a CEGAH task posing human rights concerns. SOLUSI. We agree with the ET’s evaluation of the SOLUSI program, which is precisely the reason why CEGAH did not extend the grant. The original purpose was to provide support to the private sector to navigate Indonesian anti-bribery regulations, an expressed priority. However, when it became clear that efforts to promote the hotline generated only a few inquiries and limited results, the program ended the engagement. SI Response: Thank you for this additional information. No edits to the report required. LAPOR! interventions. CEGAH appreciates the ET’s observation that SP4N LAPOR! has achieved significant progress on several fronts under CEGAH’s direction (p. 28). Indeed, the achievements generated from CEGAH’s support for LAPOR! was a testament to the strong collaboration between key government ministries and CEGAH, and a strong buy-in and ownership on the government side. In addition, we offer the following supplemental information: 126 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report a) With regard to the recommendation made by the ET for the provision of offline means to file complaints (p. 5), it is worth noting that an offline and manual system to file complaints existed before LAPOR!, and this is still available at present. The main purpose of LAPOR! was to provide an internet-based online means of filing complaints and reports on government services, accessible from anywhere in the country; thus, it is a channel parallel to the offline one. It is also important to note that precisely for areas with low bandwidth, the SMS functionality of the system remains in place and is actively utilized by citizens. Following through on the expansion of existing offline functions, while increasing access for some groups, would come at a tremendous cost in a country the size of Indonesia and would hardly meet the ET’s scaling criteria. b) Further, to supplement the recommendation of the ET for USAID to consider supporting the formulation of a LAPOR! roadmap that addresses future needs and inclusion requirements, it is useful to note that KemenPAN-RB has an existing roadmap. In Year 3, CEGAH proposed to revise and update the roadmap, but in the course of donor coordination efforts, it was agreed that UNDP, with KOICA support, will work on the roadmap while CEGAH will focus on application enhancement, public education campaigns, and technical assistance to national government agencies and local government units to connect to the reporting platform. This decision was taken in close coordination with USAID to avoid donor duplication and we request that this important background be reflected in the report and the recommendations/findings be adjusted accordingly. c) On p. 25, the ET notes that the dashboard was an important feature to encourage the monitoring and supervision of complaints, yet at the same time, the ET expresses concern about insufficient supervision of LAPOR!. It would be helpful to clarify which other supervision measures the ET envisions for future program planning. d) Finally, the ET expresses concern on p. 22 about the focus on the number of complaints received. CEGAH respectfully submits that the number of complaints and their increase was a major focus for USAID throughout the program. CEGAH sought to be responsive to USAID’s priority and, accordingly, conducted numerous efforts to expand the reach and availability of the platform, and increase the complaints received by the system. With regards to the observation that random text might have been entered as complaints, we can only assume that this relates to “Lapor goes to Campus!” events where participants were encouraged to download the app on the spot. As part of the efforts to generate excitement around the app, attendees were engaged in a practical exercise to think of recent concerns about a public service they had and to file a complaint about it on the spot. Other engagements featured visits to remote communities where citizens, including those who had internet access challenges, were provided information and support in filing complaints about public services. These activities are meant to inform the community that there is a channel through which they can submit their complaints and reports regarding quality or availability of government services, a channel that they can easily access wherever they are. It is unclear what “gimmicks” the ET is referring to. SI Response: The ET thanks CEGAH for these clarifications on the SP4N LAPOR! initiative. We provide the following in response to this input: a) The ET has modified the relevant content in both the report’s Executive Summary and Recommendations sections to expand on offline and other options for filing complaints under LAPOR! With regard to expanding existing offline functions, the ET feels that promotion (e.g., more advertising) of existing solutions or of those potentially available to local governments (a Whatsapp-based system) is necessary to continuing the growth of LAPOR, even if it is done incrementally. 127 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report b) The ET has modified the appropriate content accordingly with regard to the UNDP’s role on the KemenPAN-RB’s roadmap under CEGAH. c) Regarding the monitoring of complaints under LAPOR!, the ET found that the launch of the dashboard, while an important monitoring/supervisory development, does not obviate the need to address other supervisory issues the ET identified within the system – e.g., rapid staff turnover in system administrators at the local levels, staff carrying double burdens (system administration plus original jobs responsibilities). The ET discusses potential solutions to this in its recommendations – e.g., strengthening the capacity and incentives for SP4N LAPOR! administrators. d) With regard to the number of complaints received under the LAPOR! system, key informants interviewed by the ET stated that they needed to provide incentives to encourage students to download the app and file complaints. The referenced example was seen as such a “gimmick” by one grantee. CEGAH’s COVID-19 programming. CEGAH appreciates the ET’s recognition of its quick response to COVID-19 through a set of activities that seeks to promote transparency, accountability and prevent corruption during the pandemic. The activities mentioned in the report include the following: a) The development of an advance filter feature in SP4N LAPOR! which facilitates the easy filing of complaints and reports regarding COVID-19 and drive quick action by the concerned government agency (p. 34). b) The conduct of specialized training to provide auditors of the IG of the Ministry of Health with the necessary skills to conduct corruption vulnerability assessments of emergency procurement for pandemic response and prepare risk mitigation and corruption prevention plans (p. 35). Though not captured in the report, CEGAH also implemented the following important COVID-19 interventions: a) Supported Indonesia Corruption Watch to conduct monitoring of COVID-19-related procurement and distribution of social safety nets in 11 areas. b) Provided a grant to support the Ministry of Education to conduct corruption vulnerability assessment of the BOS School Fund Program which seeks to reallocate huge amounts of state funds to support online learning platforms and materials to allow public schools to conduct online and distance learning. c) Technical assistance to KPK to promote JAGA Bansos, an online monitoring and reporting platform for complaints of corruption in relation to the distribution of social safety nets and direct cash subsidies during the pandemic. Finally, the proposed study mentioned in the report (p. 34-35), to be conducted by PUKAT UGM on the new law seeking to promote financial stability during COVID-19, did not materialize due to prioritization constraints on the part of potential government counterparts. SI Response: Thank you for noting the additional COVID-related efforts the CEGAH team has undertaken. The ET has added the first example, which was also found in CEGAH’s activity documentation and which was completed during the evaluation window (which ended in late June). The ET has also 128 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report corroborated that the proposed study to be conducted by PUKAT UGM on the COVID 19 law is no longer moving forward and have taken it out of the report. Planning process. The ET highlights on p. 20 that “one grantee” stated that differences in the culture of CEGAH beneficiaries were not always taken into account in the planning process. It is unclear what conclusion can be drawn from such a finding, so we request the ET to clarify what the opportunities for improvement are in this regard, preferably substantiated with more than one grantee’s comment. SI Response: The ET has clarified that with respect to the culture of CEGAH beneficiaries not always being considered in the planning process, more than one grantee voiced similar opinions. Rather, the ET, in citing specific examples, was merely trying to illustrate the broader point. The ET has taken one sentence out of this finding in the executive summary, as we acknowledge it appears out of context there. We also note that content elsewhere in the report addresses how such cultural differences can be considered in the planning of future interventions – e.g., see final paragraph of “Expand and strengthen training opportunities” subsection“ in “Lessons Learned section”). Still on the issue of the planning process, the ET recommends on p. 41 for CEGAH to work with KPK and AGO in the planning process. The record will show that CEGAH engaged with both KPK and AGO early in the planning process and sought to bring their representatives together in various events, such as trainings and workshops, and for some projects, such as sentencing guidelines development. However, the dynamics between both institutions is complex and often unpredictable, factors that were not fully considered in the draft report. In this regard, CEGAH made early adjustments to the approach that would take into account the nature of the relationship between AGO and KPK, and encouraged the Supreme Court to take the lead with the expectation that the two agencies will follow. SI Response: The ET is not saying that CEGAH’s planning with both the KPK and AGO did not take place but is suggesting that the timing and substance of such training could potentially be tweaked to yield stronger results. The ET also notes that it provides additional possible ways to engage with KPK/AGO, including through the use of an MOU, modifying/strengthening prominent and top-down communications and relationships with GOI and judicial entities, such as prosecutors within the AGO, and the Supreme Court’s development of criminal proceedings during the current pandemic as another option to incentivize AGO participation in their drafting. Separately, the ET notes that we did consider that the dynamics between the KPK and AGO institutions are complex and often unpredictable – please see pgs. 20-21. Staffing changes. The ET expresses concern regarding “unanticipated” (p. 2, Abstract) staffing rotations which resulted in “changing levels of commitment” (p. 3). CEGAH takes exception to the finding that these staffing rotations were unanticipated; this is a reality in many bureaucracies in the region, not just in Indonesia, and was duly considered in the planning process. In addition, mitigation efforts during the implementation phase of the activities were undertaken by CEGAH, and it is suggested that the evaluation reflect these efforts adopted to prepare for such rotations. A noteworthy measure in this regard is the tireless effort by the LAPOR! team to enshrine the system’s functionality in local regulation including the assignment of dedicated staff, precisely to minimize the risk of waning political support following leadership and/or staff rotation. SI Response: The ET has adjusted this wording but notes that multiple grantees appeared surprised by or otherwise experienced challenges with such staffing changes, which forced them to conduct trainings again for their replacements. 129 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Sustainability issue pertaining to risk management program. On page 19, relating to the formulation of guidelines/SOPs on risk management for planners and auditors in the Inspectorate General, a ministerial beneficiary mentioned that they have not been able to implement the guideline fully. This is because there has been no movement from the Secretary General to take the necessary steps to implement the guideline. The ministerial beneficiary further mentioned that they have asked for CEGAH’s assistance to push things forward but have not received feedback. On this issue, the program team at CEGAH had exerted efforts to reach out to the Inspectorate General and Secretary General on a number of occasions to address this concern, but an apparent lack of interest was noted. This suggests that, as in many development projects involving direct assistance to government institutions, coordination and communication between project implementors and government counterparts, and among government counterparts as well, will remain a challenge and should be addressed in advance to prevent it from undermining long-term sustainability of project gains. SI Response: We note and thank you for this additional context, but it does not change the team’s findings. Guidelines on Corporate Criminal Liability. On p. 18, the development of the guidelines on investigating and prosecuting Corporate Criminal Liability (CCL) was attributed to CEGAH’s network. While CEGAH supported promotional efforts of the new regulation among businesspeople, it was not responsible for the development of CCL guidelines; they were developed sometime around the commencement of CEGAH implementation by another organization or network. SI Response: The ET has modified the appropriate language in the report. Early support for SIPP. On page 25, it was mentioned that KemenPAN-RB took the initiative to create and improve the SIPP (the National Public Service Information System). In this regard, it is useful to add that the SIPP was supported by CEGAH from the early stages and up to the present time. SI Response: Thank you for this additional context. No edits to the report required. No explicit KPK dual approval process. On page 37, it was stated that it is now a requirement for CEGAH to secure approval of the Director of Prevention for every activity proposal, in addition to the Secretary General of KPK’s program implementation unit. The relevant paragraph further stated specific instances where CEGAH proposals were rejected by the Director of Preventions on the ground that it was outside the mandate of KPK. In this regard, it is significant to note that prior consultations with government counterparts are critically important to building ownership and buy-in for any project initiative. In the course of these consultations, various issues and concerns are discussed, including the mandate of the government agency, its priorities, and present challenges that may require CEGAH support or assistance. These discussions between CEGAH and prospective government beneficiaries are important to inspire mutual trust and build ownership, and while they may lead to actual activities or interventions, there are instances that they do not, for a variety of reasons; but these instances should not be considered as reflective of a parallel approval or vetting process. While we accept the facts as presented, we would also like to highlight the potential sensitivity of this issue for GOI-USG relations in regards to future programs and encourage the ET to revisit the wording of this section in case of plans for public release. SI Response: The ET acknowledges the sensitivity of this issue, but our responsibility is to provide a complete accounting of our findings to USAID. The team therefore defers to USAID regarding any potential issues of sensitivity and whether to redact any portions of the report from public sharing. 130 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report Engaging Integrity Zones. On p. 41, the ET recommends engaging Integrity Zones to continue the success of ISO 37001. We respectfully disagree as these are totally different programs with totally different levels of efficacy. We believe that suggesting the adjustment of internationally recognized ISO 37001 standards with the Integrity Zone efforts of questionable efficacy would severely decrease the utility of the former. SI Response: We understand that there may be significant differences between ISO 37001 and Indonesia’s Integrity Zones, and if there is an option to continue the ISO 37001 certification process within the courts, it should be pursued. But the reality appears to be that there is no funding available as of yet to continue the ISO 37001 certification by the courts after CEGAH ends. Therefore, integrating these ISO 37001 standards within an existing Indonesian anti-corruption certification platform (Integrity Zones) may be the difference between these standards continuing or not, even if the Integrity Zones currently support different standards. Lack of top-level government access. On p. 42, the ET expresses concern about the lack of top￾level government access. CEGAH is concerned by this statement since its program teams have consistently maintained open lines of communication and solid top-level access to all relevant beneficiaries and government counterparts. It would be helpful for the ET to clarify this statement. SI Response: The ET understands that program teams may maintain such lines of communication and top-level access, but this does not mean that this access filters down to the grantee level. Option to buy materials with own resources. On p. 43, the ET suggests that beneficiaries ought to be able to copy training materials and purchase trainings themselves. CEGAH always provided soft copies of training materials unless there were copyright concerns from the training provider. SI Response: The ET notes that interviewees understood that there were intellectual property concerns, so they instead wanted to be able to purchase the trainings themselves, which they believed wasn’t possible under the arrangement with CEGAH. Link between an anti-bribery system and speed of case resolution. On p. 46, one of the ET recommendations refers to the long-term impact of ISO 37001 as the speed of court case resolution. It is, however, unclear how an anti-bribery certification is related to case resolution speeds. SI Response: The ET has adjusted this language and now discusses the speed at which cases are assigned to judges, as other USAID-supported judicial anti-corruption activities are adopting case-assignment systems that are designed to increase judicial accountability by more quickly and randomly assigning cases to get judicial proceedings more quickly underway. Closing Statement CEGAH appreciates this opportunity provided by USAID/Indonesia and Social Impact, Inc. to address any factual concerns that may be relevant to improve the present Draft Evaluation Report. If there are further concerns that require project attention, CEGAH stands ready to respond and assist in any way. SI Response: The ET thanks the CEGAH team for providing this feedback on its evaluation report. Where appropriate, we have adjusted the language in the report to address this additional information and provide clarifications that may be useful to readers. 131 MEL Support to USAID/Indonesia USAID CEGAH Final Performance Evaluation – Final Evaluation Report United States Agency for International Development Jakarta, Indonesia