April 1, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Patrick Fn’Piere, Ashari Edi, Isma Yusadiredja, and Amanda Stek of Social Impact. FINAL PERFORMANCE EVALUATION OF POLICY CLUSTER APPROACH Program Representasi (ProRep) FINAL PERFORMANCE EVALUATION OF POLICY CLUSTER APPROACH Program Representasi April 1, 2016 #AID-497-C-11-00002 COVER PHOTO Bukit Indah Farmers Group, one among eight groups who gained a Community Forest Utilization Permit (IUPHKm) after receiving assistance from ProRep partners, having a discussion at their farm. Photo Credit: Josh Estey/USAID/Program Representasi on January 14, 2016. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS 1Acronyms .......................................................................................................................................................................... 1 Executive Summary ........................................................................................................................................................... 3 Background ..................................................................................................................................................................... 3 Findings and Conclusions ............................................................................................................................................. 4 Recommendations ......................................................................................................................................................... 5 Evaluation Purpose and QuestionS ................................................................................................................................ 7 Evaluation Purpose ........................................................................................................................................................ 7 Evaluation Questions .................................................................................................................................................... 7 Project Background ........................................................................................................................................................... 9 ProRep Phase I and II .................................................................................................................................................... 9 Policy Cluster Approach ............................................................................................................................................ 11 Evaluation methods and limitations ............................................................................................................................. 12 Evaluation Methods ..................................................................................................................................................... 12 Biases and Limitations ................................................................................................................................................. 15 Findings, conclusions and Recommendations ........................................................................................................... 18 Question 1 .................................................................................................................................................................... 18 Question 2 .................................................................................................................................................................... 22 Question 3 .................................................................................................................................................................... 25 Question 4 .................................................................................................................................................................... 28 Question 5 .................................................................................................................................................................... 31 Question 6 .................................................................................................................................................................... 33 Annexes ............................................................................................................................................................................. 35 Annex I: Evaluation Statement of Work ................................................................................................................ 35 Annex II: Data Collection Instruments .................................................................................................................. 43 Annex III: Evaluation Sources of Information ....................................................................................................... 55 Annex IV. Data Collection Schedule ....................................................................................................................... 59 Annex V: List of Grantees ......................................................................................................................................... 62 Annex VI. Grant Mechanisms ................................................................................................................................... 64 Annex VII: List of Regulations Enacted ................................................................................................................... 65 Annex VIII. Policy Clusters ........................................................................................................................................ 66 Annex IX. Evaluation Matrix ..................................................................................................................................... 71 Annex X. Conflict of Interest ................................................................................................................................... 78 Annex XI. Evaluation Team Biographies ................................................................................................................ 81 Annex XII. ProRep Activity Map .............................................................................................................................. 82 Annex XIII. Social Impact Response to USAID Comments on Draft Report .............................................. 83 Tables and Figures Table 1: KII Respondents by Institution ..................................................................................................................... 13 Table 2: Cluster Formalization Factors ...................................................................................................................... 28 Figure 1: Site Locations .......................................................................................Error! Bookmark not defined. Figure 2: Cluster Engagement with National and Local Level Government Entities ....................................... 21 Figure 3: ProRep Cluster’s Core Elements............................................................................................................... 27 1 ACRONYMS CDCS Country Development Cooperation Strategy COP Chief of Party COR Contracting Officer Representative CSO Civil Society Organization DEC Development Experience Clearinghouse DFAT Department of Foreign Affairs and Trade DPD Dewan Perwakilan Daerah / Indonesia’s Regional Representative Council DPR Dewan Perwakilan Rakyat / Indonesia’s House of Representative DPRD Dewan Perwakilan Rakyat Daerah / Indonesia’s Local Parliament DO Development Objective DRG Democratic, Rights, and Governance ET Evaluation Team FAA Fixed Amount Award FOG Fixed Obligation Grant FGD Focus Group Discussion F2H Frontiers for Health GOI Government of Indonesia hKm Hutan Kemasyarakatan / Community-Based Forest Management ICEL Indonesia Center for Environment Law ICW Indonesia Corruption Watch IP Implementing Partner IUPHKm Community Forestry Utilization Permit JABAT House of Parliament Member Constituent Outreach Program KII Key Informant Interview KMSTP Koalisi Masyarakat Sipil untuk Transparansi Pendidikan / Civil Coalition for Education Transparency KSI Knowledge Sector Initiative LoE Level of Effort LoC Letter of Collaboration LPEM FEB UI Lembaga Pengembangan Ekonomi dan Manajemen, Fakultas Ekonomi dan Bisnis, Universitas Indonesia / Institute for Economic and Social Research, University of Indonesia M&E Monitoring and Evaluation MOEC Kementrian Pendidikan dan Kebudayaan / Ministry of Education and Culture MoEF Ministry of Environment and Forestry MoH Ministry of Health 2 MP Member of Parliament MPPS Masyarakat Peduli Pendidikan Surakarta /Civic Coalition for Education Surakarta NCT National Coordination Team NGO Non-Governmental Organization OGP Open Government Partnership Perda Peraturan Daerah / Local Legislation PIRS Performance Intermediate Results PMP Performance Management Plan PPPI Paramadina Public Policy Institute PRN Policy Research Network ProRep Program Representasi RMI Rimbawan Muda Indonesia / Indonesian Youth Forester RPJMN Rencana Pembangunan Jangka Menengah Nasional / National Mid-Term Development Plan SAF Strategic Activities Fund SK Surat Keputusan / Decree SI Social Impact SMS Short Message Service SOW Scope of Work SSTC South-South and Triangular Cooperation TPM Team Planning Meeting USAID United States Agency for International Development WRI Women Research Institute YAPPIKA Yayasan Penguatan Partisipasi, Inisiatif dan Kemitraan Masyarakat Indonesia / Foundation for Strengthening Participation, Initiatives, and Partnership of Community in Indonesia YSKK Yayasan Satu Karsa Karya / Foundation of Unified Intention and Creation 3 EXECUTIVE SUMMARY BACKGROUND As Program Representasi (ProRep) completes its final year in April 2016, the United States Agency for International Development (USAID) requested that the implementing partner (IP), Chemonics, conduct a final performance evaluation. The focus of this evaluation is to evaluate ProRep’s year four and five use of policy clusters to improve the policymaking process and promote policy reforms; and to generate key lessons learned and identify promising practices related to the policy cluster approach for replication in future USAID programs (in various sectors). Program Representasi (ProRep), a five-year United States Agency for International Development (USAID) project implemented by Chemonics, was designed to strengthen democracy and good governance in Indonesia by promoting better informed and more representative legislative and policymaking processes. The project was implemented in two phases, with Phase II as the focus of this performance evaluation. Phase II of the project was implemented during year four and five. In Project year four, and in line with USAID’s 2014 – 2018 County Development Cooperation Strategy (CDCS), ProRep institutional support to Indonesian parliaments ended. Institutional strengthening support to civil society and think tanks was scaled back. ProRep shifted to facilitating multi-sector policy communities or “policy clusters.” This approach built on experience from phase 1 but a) sought to bring CSOs, think tanks, and policymakers together to collaborate in support of specific policy reforms in the sectors of education, health, and environment and b) set the project in-line with USAID/Indonesia’s new CDCS. These policy communities were designed to become platforms to facilitate interactions among inter￾connected actors and to further inculcate an inclusive and evidence-based approach to policymaking. Specifically, six questions were used to evaluate the cluster approach in the sectors of Health, Education and Environment: 1. In what ways is the policy cluster approach, as implemented through ProRep, affecting the following objectives: bringing about better informed and more representative legislative and policymaking processes; and, promoting the passage of policy improvements or better policy implementation. 2. According to stakeholders, what are contributing and inhibiting factors toward the achievement of these objectives? What could be done to increase the effectiveness of the clusters in achieving each of these objectives? 3. In what ways did the policy clusters under ProRep influence national vs local level policy? What are the differences or similarities in how clusters operate at these levels? 4. How has the policy cluster/community approach been operationalized differently (or similarly) in the sectors of environment, health, and education through ProRep? Have different methods of operationalization affected achievement of cluster objectives? 5. How did ProRep’s management of partnerships (through grants, letters of collaboration, and other mechanisms) contribute to or inhibit the achievement of policy cluster objectives? 6. What aspects of sustainability supported or promoted by ProRep are present in the policy clusters? What are the challenges to sustainability for each cluster? The Evaluation Team (ET) used several techniques to complete this evaluation which entailed a mix of mutually reinforcing qualitative methods that reflected the program logic and research questions being addressed. The team utilized document review, qualitative data collection, and data analysis to capture the diversity of opinions and perceptions of stakeholders about the policy cluster approach, including its strengths and weaknesses. The qualitative data collection, which included 41 key informant interviews 4 (KII), three focus group discussions (FGD) and two site visits was conducted in Jakarta, Solo (Central Java), and Bengkulu (Southwest Sumatra) in January and February 2016. The ET faced several limitations during data collection, namely availability of contacts, limited Government of Indonesia respondents at the national level, and limited time for evaluation data collection. Additionally, the ET was challenged in assessing the outcomes of the policy cluster approach due to the length of the cluster approach as implemented by Prorep and by the selection of only ¾ of the clusters for inclusion in the evaluation. The above limitations, however, did not prevent the ET from gathering information and data needed to produce findings, conclusions, and recommendations for this evaluation. FINDINGS AND CONCLUSIONS The Evaluation Team (ET) found that ProRep’s cluster approach, as piloted, met its objectives in two distinct ways (evaluation question 1 and 2). First, the ProRep cluster was designed as a new tool for CSOs to move from pure advocacy to more effective engagement in participatory governing. They did so during the grant periods. Secondly, the cluster approach was effective at taking advantage of the opportunity gained through decentralization, i.e., the move of implementation of policy from concentrated national offices to local governments. Preliminary indications are that the cluster approach leads to solid policy outcomes. ProRep had 10 policy wins in year 4 and 5; 11 other policy objectives are still under consideration at the time of this report writing. By targeting achievable and pragmatic policy objectives, ProRep clusters positively influenced government officials’ views of how CSOs and think tanks could influence policy, and built confidence in CSOs and think tanks in their ability to influence policy. Most CSOs reported that by focusing their cluster activities on “policy/rule-making” (as opposed to the pure advocacy approach they had used to influence change previously), they acquired new practical knowledge and “critical engagement” skills that are being internalized into their respective organizations’ missions. At the same time, both the implementing partner and clusters faced significant program constraints due to unsure future funding during both final years of the project (evaluation question 2). Concurrently, a short ramp-up in the initial development of a “cluster model” and equally brief periods for turning around proposals, implementing cluster activities, and producing policy wins were also problematic. The one-year time frame to accomplish the range of requirements was simply too short to see all policy objectives through to conclusion. ProRep CSOs and think tanks also had some complaints about ProRep’s management of the cluster process (evaluation question 5). Most CSOs perceived the cluster process as “cumbersome” with too many “meetings.” Compounding this perception was that ultimate success in the cluster approach rested on individual cluster members’ ability to (as seamlessly as possible) integrate their organization’s activities into the broader objectives of the whole cluster without the benefit of significant financial support to expand their internal capacity. Despite these programming challenges, ProRep successfully initiated an inclusive, iterative process to frame a vision for how clusters could be developed and what over-arching sector policies/objectives would be undertaken (evaluation question 4). The process involved discussions with multiple stakeholders, with significant input from USAID/Indonesia, informed by quickly-developed expert working papers. The national/local structure of the clusters increased access to multiple-levels of government among cluster members and increased each member’s political effectiveness, operational capacity, and reach in a manner that did not increase overhead for their respective organization (evaluation question 3). In effect, by cluster members collaborating on discrete policy objectives, which drew on relative strengths 5 of participating members, members created a policy-making function for their own organization greater than they could or would have done alone. The ET found that Fixed Amount Awards (FAA) and Letters of Collaboration (LOCs) were appropriate and flexible tools to allow clusters to meet their objectives (evaluation question 5). FAA grantees were required to meet milestones (e.g., sponsor an event) to receive funds, and did not have to submit receipts or share accounting records. This lessened the administrative and financial burdens for grantees that allowed them more time to focus on programmatic milestones. Through LOCs, gaps in grant proposals were filled or unanticipated issues were handled through in-kind activities or purchases from ProRep. LOCs helped grantees maintain momentum in their activities that was important due to short grant time frames. Meeting the demand by issuing LOCs was very labor intensive for the implementing partner organization. With more upfront planning, the LOC demand likely would have been reduced. The ET found that it is too early to confirm the sustainability of the clusters supported by ProRep (evaluation question 6). However, notable cluster achievements show promise. First, one of the three clusters, the Education Cluster, has formalized membership through the creation of a newly registered national organization with a membership of over 40 organizations committed to utilizing the cluster approach. Second, members of the Health Cluster have, in principle, agreed to jointly pursue future funding as a cluster. While not conclusive of the long-term durability of clusters, the ET found that these two outcomes evidenced forward momentum beyond the ProRep project. RECOMMENDATIONS Based on the ETs findings and conclusions, 12 specific recommendations for USAID are provided: 1. The ET recommends that before any new sector cluster formation in Indonesia is encouraged by USAID, a diagnostic study should be conducted on the existing laws, regulations, and practices affected by decentralization to become better acquainted with the range of options and responsibilities afforded the sub-national government units related to the policies the cluster plans to influence. This will allow the cluster to map the local-national policy linkages, actors, needs as well as potential supporting resources such as political opportunities, allies and constituencies. 2. Specific time should be allocated within grants to develop work plans prior to launching specific cluster activities. Work plans should better align internal capabilities and operational reach of CSOs to meet negotiated, common policy objectives. 3. In the Indonesian context, with a history of adversarial policymaker-CSO attitudes, the ET recommends that USAID use its reputation to enlist noteworthy/trusted individuals to help clusters gain access to policy markets. Early personal contact and communication with key government officials by respected individuals within a cluster or senior advisors to the cluster should coincide with cluster formation activities. 4. Informal communications/engagement by various cluster members with policy makers is important for building relations and such communications have a cost. Grants should allow for this cost. 5. As part of future cluster-type grants, USAID implementing partners should look to standardize training of key personnel in cluster CSOs on government and legislative protocols (e.g. hearings, policy mark-ups, budget and legal requirements, etc) at both national and sub-national units of government. 6. As part of the cluster operational plan, the ET recommends a standalone strategic communication plan be developed by the cluster alongside its policy reform/implementation work plan. The communication plan should outline how the cluster will maximize broad public 6 awareness of cluster objectives and accomplishments, to highlight the work of the cluster(s) as a new way of governing. 7. The ET believes that future consideration of policy cluster designs and formations by USAID and/or its implementing partner(s) would continue to benefit from facilitated dialogues among potential cluster members and other interested parties. In the future, such dialogues should be guided by policy mediation experts who possess substantive expertise in government processes, planning, or a related discipline to save time and facilitate actionable agreements early on that ProRep clusters spent working out over time (which would result in lowering transactional cost). 8. USAID should consider using a co-creation process to design any follow-on cluster related project. Co-creation allows for more dialogue and input from more stakeholders and creates a more transparent process for project design. Since this is an evolving approach to both program design and policy reform, getting additional ideas on how clusters might be formed and operated could enrich a follow-on project design. 9. USAID should continue the utilization of FAAs and increase use of LOCs for timely tactical assistance to grantees. The experience of having to meet milestones to receive payment was seen to have value, while the requirement of previous grants to respond to USAID administrative requirements was not seen to add value to the organizations’ operations. 10. The USAID implementing partner conducting the cluster approach in the future should develop a grants management database system that goes beyond monitoring and evaluation (M&E) functions to include overall program operations. The addition of a database grants management platform could standardize management practices and reporting among the clusters to simplify program management by an implementer. 11. Encourage the formation of cluster secretariats to replace the USAID implementing partner as lead convener of cluster activities; and, as feasible, the ET recommends that cluster grants include provisions for clusters to obtain their own institutional support, with the caveat that protocols should be established that ensure that secretariats do not become a competitor to its members. 12. Encourage joint solicitation of new funding by sector clusters particularly in the environment sector where there appears to be significantly high interest from private sector and international donors. 7 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE As Program Representasi (ProRep)1 completes its final year in April 2016, the United States Agency for International Development (USAID) requested that the implementing partner (IP), Chemonics, conduct a final evaluation as per Section F.7.12 of the Project Contract.2 In compliance with Contract requirements, ProRep, in agreement with USAID and through subcontractor Social Impact (SI), conducted an independent final evaluation of the ProRep Project, focused on the policy cluster approach. The objective of the evaluation, as stated in the Contract, was to identify which elements of the project were most impactful, which were not, and how to integrate end-of-of project learning into future USAID programming. USAID clarified in November 2015 that “the project”, or the focus of the evaluation, should be the policy cluster approach, as was implemented in year four and five. The specific objectives of the revised evaluation scope are as follows: 1. To evaluate ProRep’s Year four and five use of policy clusters to improve the policy making process and promote policy reforms; and, 2. To generate key lessons learned and identify promising practices related to the policy cluster approach for replication in future USAID programs (in various sectors). The primary intended user of this evaluation is USAID/Indonesia, particularly the Democracy, Rights and Governance Office (DRG) and Mission management. In addition, USAID/Indonesia’s Development Objective (DO) teams and USAID/Washington bureaus will use the results of this evaluation to inform planning of future programming involving the use of evidence-based policy-making. The secondary audience are local Indonesian institutions, other donors, and other USAID Missions worldwide. EVALUATION QUESTIONS The evaluation reviewed, analyzed, and evaluated ProRep’s implementation of the policy cluster approach according to the six questions: 1. In what ways is the policy cluster approach, as implemented through ProRep, affecting the following objectives: bringing about better informed and more representative legislative and policymaking processes; and, promoting the passage of policy improvements or better policy implementation. 1 Award Number AID-497-C-11-00002, with contract dates of April 19, 2011 to April 18, 2016. The total value of the contract was USD 20,160,527.47 2 The contract states the following: The Contractor will conduct a final independent evaluation of the project within the final quarter of the final contract performance year. The focus of the evaluation will be to assess the achievements of the project versus the stated objectives and goals, to identify which elements of the project had the most significant impact and which did not, and which aspects of project design need to be considered for continuation under future projects. The evaluation team is to be comprised of technical experts/evaluators who are independent of the Contractor and the Contractor’s staff. The final evaluation report will be submitted to the COTR, with a copy provided to the Contracting Officer, no later than the final date of the contract performance period. 8 2. According to stakeholders, what are contributing and inhibiting factors toward the achievement of these objectives? What could be done to increase the effectiveness of the clusters in achieving each of these objectives? 3. In what ways did the policy clusters under ProRep influence national vs local level policy? What are the differences or similarities in how clusters operate at these levels? 4. How has the policy cluster/community approach been operationalized differently (or similarly) in the sectors of environment, health, and education through ProRep? Have different methods of operationalization affected achievement of cluster objectives? 5. How did ProRep’s management of partnerships (through grants, letters of collaboration, and other mechanisms) contribute to or inhibit the achievement of policy cluster objectives? 6. What aspects of sustainability supported or promoted by ProRep are present in the policy clusters? What are the challenges to sustainability for each cluster? 9 PROJECT BACKGROUND PROREP PHASE I AND II ProRep is a five-year USAID-funded project designed to strengthen democracy and good governance in Indonesia by promoting better informed and more representative legislative and policymaking processes. Figure 1 below is a map identifying areas where ProRep activities were conducted during the life of the program. ProRep partners held activities in 25 provinces. Figure 1: Site Locations *An additional map has been included in Annex XII from the ProRep Final Report 2016. ProRep’s results framework was developed at the launch of the program and was further refined in 2014 and 2015.3 The Program Objective is at the pinnacle of the ProRep Results Framework; a ‘Better informed and more representative legislative and policy-making processes’. ProRep’s Program Intermediate Results (PIR), contribute to the realization of the Program Objective, and are three in number. They are:  ‘Representative capacity of membership- and constituency-based CSOs strengthened’  ‘Research institutions’ capacity to conduct and disseminate policy-relevant research and analysis on key policy and governance issues strengthened’  ‘More effective, responsive and transparent legislative processes’ The project was implemented in two phases. Phase I was conducted during years one, two and three of the project, or the base contract period. During this period, ProRep focused on strengthening the three sectors listed below (items 1 – 3), and conducted special initiatives as agreed with USAID (item 4): 1. Civil society organizations (CSO), particularly non-profit non-governmental organizations (NGO) that are membership-or constituent-based.4 Most such organizations are national in 3 See ProRep’s 2015 PMP for a more detailed description of all indicators included in the results framework. 4 ProRep defined CSOs as groups or associations organized by citizens to promote or protect common interests. 10 purpose. Many represent women and other marginalized groups and engage in advocacy on key policy issues. 2. Public policy research institutions, including research groups engaged in applied or policy research in universities, research centers, and CSOs.5 3. Parliament’s two national houses, the Indonesian House of Representatives, Dewan Perwakilan Rakyat (DPR) and, to the extent agreed with USAID, Indonesia’s Regional Representative Council, the Dewan Perwakilan Daerah (DPD). 4. Timely assistance for special initiatives needed to protect or advance democratic governance. At its mid-way point, the ProRep midterm evaluation revealed that the Project had helped selected CSOs and CSO networks develop larger and more defined constituencies, expand memberships and stakeholders, and become more informed and constructive advocates in the policy process. ProRep also assisted them in finding positive ways to work with parliaments, especially the DPR, as well as other local- and national-level policymakers and implementers. Second, ProRep worked with research organizations to improve the quality of policy research and research on legislation, and to enhance its dissemination to both national- and local-level policymakers, allowing civil society, parliament, and others to develop better-informed approaches to formulating and implementing new laws, to overseeing how laws are actually implemented and their impact on people, and for parliament members to better respond to their constituents. Third, the project worked to meet a number of defined needs of the DPR (and DPD, as needed) and improve its national standing by helping members represent their constituents more effectively, transparently, and in a better-informed manner. The Special Initiatives component (also called Strategic Activities Fund - SAF) was designed to enable USAID to respond flexibly and rapidly to address other unanticipated needs and opportunities pertaining to the protection and advancement of democratic governance in Indonesia. Under this component ProRep implemented a number of activities to advance democratic governance, including study visits and peer-to-peer exchanges, and carried out a DPR Member Constituent Outreach Program (Jangkau dan Libatkan - JABAT) to help bridge the wide gap between DPR Members and their constituents, providing venues and opportunities for greater, and more meaningful contact between citizens and the DPR Members (Members of Parliament - MP) than has been the case in the past. Phase II of the project was implemented during year four and five. In early 2013, ProRep, through subcontractor SI, conducted a mid-term evaluation, resulting in a number of findings, conclusions, and recommendations presented to USAID. The revised program design in Phase II and an updated Performance Management Plan (PMP) submitted to USAID addressed a number of these recommendations. In Project Year four, and in line with USAID’s 2014 – 2018 County Development Cooperation Strategy (CDCS) released in November 2013, ProRep institutional support to Indonesian parliaments ended. Institutional strengthening support to civil society and think tanks was scaled back, with the exception of continued institution-building assistance to the ProRep initiated Policy Research Network (PRN - an association of think tanks and research orientated CSOs). ProRep’s assistance shifted to facilitating multi-sector policy communities or “policy clusters.” This innovative systems approach built upon ProRep’s experience over the previous three years, and sought to bring CSOs, think tanks, and policymakers together to collaborate in support of specific policy reforms in the sectors of education, health, and environment. These policy communities were designed to become platforms to facilitate interactions among inter-connected actors and to further inculcate an inclusive and evidence-based approach to policymaking. 5 ProRep defined research partners as independent ‘think tanks,’ research institutes, and non-state-funded educational bodies. 11 Additionally, the Special Initiatives or SAF component continued, and began supporting, among other initiatives, Indonesia’s National Coordination Team (NCT) in their development of a South-South and Triangular Cooperation (SSTC) program; a program to reactivate the Indonesian Secretariat for the Open Government Partnership (OGP): and a special initiative to counter specific threats to Indonesia’s progress in eliminating corruption. In early 2015 ProRep launched a fourth cluster in the area of democratic rights and governance. This cluster aims to enhance the advocacy, coordination and policy reform efforts of groups promoting greater accountability in Indonesia. POLICY CLUSTER APPROACH During ProRep’s base period (described above), the project increasingly brought together CSOs, policy experts, and policymakers to work together for policy reforms. Building on this experience, and recognizing that achieving many of the objectives of USAID’s new CDCS would require national and local level policy changes, the USAID DRG Office and ProRep collaborated to develop the policy cluster approach. Through the implementation of the policy cluster approach, ProRep continued to work towards its overall program objective of “better informed and more representative legislative and policymaking processes”6 while also promoting the passage of policy improvements or better policy implementation, addressing critical development areas for Indonesia (i.e. health, education, environment, democracy). Policy and law-making in Indonesia, as in many nations, tends to take place within ministries and parliament, with little if any interaction with other sectors. This occurs without taking into account evidence-based policy recommendations made by policy experts and local constituents’ policy interests represented by CSOs. Neither tends to have sufficient understanding of how policies are made, or how to be effective advocates for policy change. These policy clusters or policy communities (terms used interchangeably) were designed in this context to bring together national and local-level policymakers from the executive and legislative branches, experts, and advocates, to help them collaborate more effectively in improving policy or policy implementation in selected policy-focus areas. Policy clusters are designed to be (1) inclusive, involving multiple stakeholders, from many sectors, (2) evidence-based, using the results of empirical research, and, (3) to involve the media, so that clusters’ positions have an impact on the opinions of decision-makers and the public. ProRep has supported these clusters through the provision of grants to CSOs and research organizations, technical assistance, support for focus group discussions, meetings, and parliamentary hearings, and other means of support. Depictions of ProRep policy clusters, by sector and year, are included in Annex VIII. 6 Ultimate goal as detailed in the Project PMP. 12 EVALUATION METHODS AND LIMITATIONS EVALUATION METHODS To gather data required for this evaluation, the Evaluation Team (ET) used several techniques which entailed a mix of mutually reinforcing qualitative methods that reflected the program logic and research questions being addressed. The team utilized document review, qualitative data collection, and data analysis to capture the diversity of opinions and perceptions of stakeholders about the policy cluster approach, including its strengths and weaknesses. The qualitative data collection, which included key informant interviews (KII) and focus group discussions (FGD), provided the local context and allowed for the collection of concrete examples that illustrate the findings in greater detail. The ET conducted the evaluation in a participatory manner which involved the USAID Mission in Indonesia, as well as the implementing partner, Chemonics. The evaluation was conducted by team leader, Mr. Patrick Fn’Piere, an expert on governance and legislative bodies; Mr. Ashari Edi, an Indonesian Democracy and Governance specialist; and Ms. Isma N. Yusadiredja, an evaluation specialist. The three-person team had sufficient expertise in evaluation, civil society, and the policy-making processes in Indonesia to be able to assess the policy cluster approach and make recommendations regarding future program direction. See Annex XI for short biographies of each team member. The phases of the evaluation process are detailed below. See Annex IV for a detailed evaluation schedule. Phase One: Project Planning and Desk Review Upon finalization of the Statement of Work (SOW), the ET conducted a thorough desk review. For the desk review, the team spent one week reviewing available documents supplied by the ProRep team and USAID. The desk review also continued throughout the data collection period. The review included, but was not limited to, the documents listed in Annex III. The evaluation relied on project data and documents to corroborate findings from KIIs and FGDs. Before the commencement of data collection, the team conducted internal coordination meetings to on￾board all team members, coordinate logistics for fieldwork, discuss drafting of the Evaluation Work Plan, and determine roles and responsibilities for the duration of the evaluation. Upon arrival in-country, the team held an internal Team Planning Meeting (TPM) on January 19. The team then conducted an in-brief with USAID/Indonesia on January 20th, which was attended by the project Contracting Officer Representative (COR) and USAID Mission Monitoring and Evaluation (M&E) Lead. During the in-brief, the team clarified expectations and discussed future utilization of the evaluation to ensure that the work plan was feasible and achievable within the time frame of the evaluation and was responsive to the Mission’s needs. Following the in-brief with USAID/Indonesia, the ProRep team presented the ET an overview of the project and policy cluster approach and achievements of the project to-date. The ET then finalized the work plan and all data collection instruments and submitted them to USAID/Indonesia on January 21, before the launch of fieldwork. Phase Two: Data Collection The initial target sample of respondents included individuals or institutions who were directly involved in ProRep. The list of target respondents was provided by the ProRep team to USAID before the ET 13 arrived in country. USAID then narrowed the list and selected priority individuals and institutions. After review of the list, the ET suggested additional informants. KIIs and FGDs included individuals or institutions that were program staff and partners; grantees who are involved in ProRep; Government of Indonesia (GOI) Ministry, local offices, and local parliament members; as well as USAID. For a complete list of ProRep grantees during year four and five, see Annex V. Additionally, several individuals and institutions who were not grantees or partners but were regarded as key sources of information were also chosen to further ground the evaluation in the Indonesian context. The final list of informants, therefore, is a result of convenience and snow ball sampling. Key Informant Interviews The ET conducted KIIs to triangulate the data collected in the desk review and gain further insights into perceptions of the program’s use of the policy cluster approach. The ET conducted KIIs with institutions, as listed in Table 1. Phone interviews were also conducted with respondents outside Jakarta or respondents unable to attend scheduled KII times. They have been included in this list as well. The full list of KII respondents is included in Annex III (listed according to institution in order to protect the anonymity of respondents). Table 1: KII Respondents by Institution Institution Total USAID 6 ProRep (Chemonics) 9 Ministry of Environment and Forestry (MOEF) 1 Former Governor of Bengkulu 1 Municipality Education Office, Surakarta 2 Municipality Health Office, Surakarta 1 Frontiers for Health (F2H) 2 Women Research Institute (WRI) 2 Watershed Management Office, Bengkulu 1 District Planning Agency Bengkulu 1 AKAR Bengkulu 1 Pattiro Jeneponto 1 HR Kemitraan 2 LepMil 1 YIPD 1 Sulawesi Institute 1 New Indonesia* 1 Rimbawan Muda Indonesia (RMI) 2 USAID – Kinerja Project* 1 USAID – Prioritas Project 3 Knowledge Sector Initiative (KSI)* 1 United Nations Development Programme (UNDP) 1 TOTAL 42 * Denotes respondents that were not in original USAID or implementer list In order to ensure that interview respondents felt as comfortable as possible, the ET, depending on the context and the population being addressed, strategically chose to have particular facilitators or individuals with similar demographic qualities administer or participate in the interviews. The semi￾structured interview protocols were finalized by the team after in-depth discussions with USAID 14 personnel and implementer staff during the week of January 20 (in Indonesia). KII questions and protocols are included in Annex II. Focus Group Discussions In Jakarta, the ET conducted separate FGDs with the following institutions: • Education Cluster (Koalisi Masyarakat Sipil untuk Transparansi Pendidikan/Civil Coalition for Education Transparency - KMSTP) including: o Indonesia Corruption Watch (ICW) o Forum Guru (Teacher’s Forum) o New Indonesia o Yayasan Penguatan Partisipasi, Inisiatif dan Kemitraan Masyarakat Indonesia (Yappika) o Expert Individual • Environment Cluster including: o Epistema Institute o RMI • PRN including: o Lembaga Pengembangan Ekonomi dan Manajemen, Fakultas Ekonomi dan Bisnis, Universitas Indonesia (LPEM-FEB UI) o Paramadina Public Policy Institute (PPPI) All FGDs were conducted by the ET; the three ET members detailed above were facilitators. To ensure FGD respondents felt as comfortable as possible, the ET, depending on the context and the population being addressed, strategically chose to have particular facilitators or individuals with similar demographic qualities to administer or participate in the discussions. FGD questions and protocols are included in Annex II. Site Visits During the second week of data collection, the ET divided into two teams; one team conducted a site visit to Bengkulu and the second team visited Surakarta (Solo). Two site visits were selected based on evaluation budget and input from USAID and the IP. The site visits provided the ET an opportunity to collect in-depth information regarding two specific policy objectives pursued by clusters during ProRep. From January 23 - 26, 2016, the first team traveled to Bengkulu. The team interviewed the Akar Foundation and stakeholders related to the environment cluster as listed below: o Akar Foundation Director and Staff at their office o Forestry Agency of Bengkulu Province o Watershed Management Office of Bengkulu Province o Former Governor of Bengkulu Province who granted hutan kemasyarakatan (hKm - community-based forest management) From January 25 – 28, 2016, the second team traveled to Solo to conduct KIIs and FGDs with the following key individuals and institutions to investigate the health and education policy clusters at the local level: o Yayasan Satu Karsa Karya (YSKK) Director and staff o Education Agency of Surakarta Municipality o Parliament members of Surakarta Municipality (Member of Commission IV) o Masyarakat Peduli Pendidikan Surakarta (MPPS) o Pattiro Surakarta Director and staff o Health Agency of Surakarta Municipality 15 In Solo, KIIs and FGDs were conducted with the health and education cluster respondents separately. All data collection methodologies considered the privacy and confidentiality of respondents; the ET also included gender appropriate questions in all KII and FGD protocols. Both women and men were included in the sampling of stakeholder groups. Lastly, the ET ensured that KIIs and FGDs were conducted at times and places accessible to both men and women equally. At the conclusion of data collection, the ET held an out-brief meeting with USAID and Chemonics on February 9,, 2016. During the out-brief, the team presented preliminary findings and conclusions. The team considered and collected all input from attendees to be incorporated in additional analysis and the final report. Phase Three: Data Analysis The data collected for this evaluation, as described in Phase one and two above, contributed evaluation question findings. In the evaluation work plan, the ET carefully detailed how data would align with each question. This matrix is included in Annex IX and shows how the team used document review (including program indicators), KIIs, FGDs and observation notes to address each evaluation question. During data collection and analysis, the ET conducted debriefs at three-day intervals to prepare a raw data matrix including all KII and FGD respondent answers to key questions. The debriefs and raw data matrix were conducted as part of critical fieldwork analysis in order to discuss a) evidence collected, patterns, and discrepancies that helped answer the evaluation questions; and b) any adjustments that were needed in the evaluation schedule. The evaluation relied on two main sources of data: existing program documents and other relevant reports as a secondary source, and a number of KIIs, FGDs and site visits as a primary source (as detailed above). Parallel analysis was used to analyze the evidence from both sources. In this analytical approach, each type of data for an evaluation question was first analyzed according to its respective source, and then across both data sources. For example, the team developed preliminary findings by first analyzing interviews with key informants; then, formulated complementary preliminary findings from the key documents and other secondary materials. All KIIs and FGDs were audio-recorded. The ET wrote a summary for each completed KII and FGD and then transferred responses to the raw data matrix. All respondent names were kept confidential during data collection and analysis. Pre-determined identification numbers were used on data collection forms (on KII/FGD guidelines and field notes). Given that the overall approach of this evaluation relied on in-depth interviews with a broad range of stakeholders as a primary data source, the collected primarily qualitative data was analyzed using appropriate qualitative analysis methods that allow for generating both “horizontal” analysis (across main themes to create the big picture) and “vertical” analysis (in-depth understanding of the most important issues). The ET began by discussing and developing codes or themes together, emphasizing the significance of consistency of analysis across different individuals. Given the large amount of information gathered, the ET utilized qualitative analysis matrices (in Excel) to handle all data with speed and accuracy. BIASES AND LIMITATIONS The evaluation encountered some biases and limitations inherent to the design of the evaluation and during its fieldwork in Indonesia. 16  First, recall bias may have been present, such as key informants responding to team questions with answers related to a different assistance program or their interaction in other “clusters” or policy communities. To mitigate this potential bias, the team used the introduction time before each interview or focus group to re-emphasize the focus of the evaluation on the ProRep support to cluster activities.  Second, response bias may have been a problem. For example, respondents could only provide the interviewer positive remarks about an activity or experience in a cluster because they would like to participate in additional funded projects. To mitigate this, the ET triangulated responses and considered desk review and interviews across stakeholders before identifying the answer to an evaluation question.  Third, selection bias in the form of contacts provided by the implementer and USAID may have meant that the team only heard from people with positive experiences. The team mitigated this challenge, however, by including a meeting with each of the following non-ProRep respondents: the UN, DFAT’s KSI Project, USAID’s Kinerja Project, and local CSO, New Indonesia  Fourth, conformation bias may have been present, meaning there may have been a tendency for respondents to readily accept conclusions that agreed with one’s beliefs, and discard conclusions that disagreed with them. This was of particular concern during FGDs, when usually FGD respondents with lower hierarchical levels tended to agree with beliefs expressed by individuals of a higher hierarchical level. To mitigate this potential bias, the ET did their best to ensure that FGDs in particular were made up of respondents of similar work stations/levels. To minimize bias, the ET used multiple sources of data to triangulate on evaluation questions. By combining information found in documents or interviews from multiple sources, any single piece of biased data did not skew the analysis. In addition, the ET used direct observation based on nuanced knowledge of the evaluation environment to assist in identifying bias in responses. The team also faced several limitations during data collection, as follows:  Availability of contacts: Interviews with GOI officials, organizations, and experts were difficult to schedule because of existing demands on the individuals’ time or the need to accommodate last minute scheduling changes. To mitigate this concern, the team remained flexible to extent possible in order to accommodate as many key informants and focus group participants as possible. The team also coordinated with the ProRep team prior to and during the data collection period to arrange schedules with potential participants. Another method used to mitigate availability changes was using phone interviews with stakeholders that had conflicting schedules or were outside of Jakarta and the site visit areas. The team was able to meet with all but three respondents from the USAID-provided list.7  Limited GOI official contacts at the national level: The ET intended to have several interviews with high-level government officials at the national level. However, after consultation with USAID/Indonesia, it was decided to drop the interviews with the Minister of National Education due to protocols required by the USAID Mission.  Length of Policy Cluster approach: The team was assessing an approach that had only been implemented for two years. The team took this into consideration when drawing conclusions and developing recommendations based on findings from the methods of data collection detailed above.  Selected policy clusters for evaluation: Based on USAID guidance, the ET did not investigate other clusters or forums promoted by ProRep, including the Democracy and Governance cluster addressing corruption. This cluster was formed under different circumstances and operated 7 The ET was unable to interview one expert, a GOI representative from the Ministry of Education and Culture, and a member of the Association of Parliament Expert Team (Asosiasi Tenaga Ahli Parlemen/ATAP). 17 uniquely when compared with the other sectors. USAID and the IP felt that cross comparisons would be more useful among the health, education and environment clusters.  Time allotted for evaluation: The ET spent three weeks in Jakarta and identified sites for fieldwork. During this three week period, the team had to assimilate data from KIIs, FGDs, as well as the desk studies and prepare an out-brief presentation. Given the scale and the scope of the ProRep Project, this is a short time period to undertake an evaluation, and therefore only a limited number of sites were visited/observed. The above limitations, however, did not prevent the ET from gathering information and data needed to produce findings, conclusions, and recommendations for this evaluation. 18 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS This section presents findings, conclusions and recommendations per evaluation question. All recommendations are written to USAID as the main end user of this evaluation. If the recommendations are targeted at different stakeholders, the ET has noted as such in the text. QUESTION 1 In what ways is the policy cluster approach, as implemented through ProRep, affecting the following objectives: a) bringing about better informed and more representative legislative and policymaking processes; and, b) promoting the passage of policy improvements or better policy implementation? Findings A review of ProRep quarterly and annual reports confirmed that overall, among the three clusters investigated for this evaluation (Environment, Health, and Education), there were over 300 policy￾focused events conducted (including more than 60 multi-stakeholder forums), involving nearly 9,000 people and approximately 100 organizations and government officials. ProRep clusters produced 10 enacted policy objectives in two years--two national level policies were passed (Presidential Decree on Ministry of Environment and Forestry structure and Priority of Social Forestry in the National Mid-Term Development Plan (RPJMN) 2015-2019) and eight subnational level polies were passed (see Annex VII). Eleven other policies were in advanced stages of development when the ET was in Jakarta for data collection. In meeting with Indonesian public officials at both national and local levels, the ET consistently received positive feedback about their participation in policy cluster initiatives as supported by ProRep. Respondents noted their positive feedback in regards to both cluster objectives, namely improved policymaking processes and passage of policy. One official declared that his experience in the cluster proved to him that CSOs could move from providing only “noise” to real “knowledge” regarding policy and regulations. The same official acknowledged that the information and research produced through the ProRep-supported Health Cluster added significant legitimacy and expertise to the policy agenda for the year by extending his committee’s data collection and outreach capacity to local communities well beyond their normal practice. The Director General of one national ministry stated that the cluster approach he participated in (an Environment Cluster) made him believe that at least some CSOs saw that “there are good people in the bureaucracy who are also looking for change”. A good example of a cluster’s strength in improving the Indonesian policymaking processes (objective A in question 1 above) is from the grantee Epistema Institute, a national driver of an Environment Cluster who worked hand in hand with local CSOs (namely RMI, AKAR Bengkulu, LePMIL and Sulawesi Institute) on local regulations and policies regarding customary land and the expansion of people￾managed areas. With ProRep support, Epistema Institute produced, among several grant products, detailed evidenced-based policy briefs and policy papers (Naskah Akademik) which were then formulated into a legal draft of local regulations to recognize the existence of the Kasepuhan indigenous community in Lebak District, Banten Province. The “how to aspects” of achieving this type of policy were developed by working with local government officials, and then were put into a Guidebook on Recognizing Indigenous Communities. In the Environmental Cluster, respondents explained to the ET that members 19 exchanged expertise to complete this product. For example, RMI’s expertise in community organizing and Epistema’s expertise in research and legal matters were key capacities that, when combined, extended each other’s respective expertise and improved the final output. The result of their collaboration and of their combined knowledge of the policymaking process in the district were essential factors that increased their relevance to government officials. The local ordinances were in turn aggregated along with other environment cluster contributions to form the basis of recommendations to the National Environment and Forestry Ministry on long-term planning. These efforts resulted in the passage of policy, detailed in Annex VII (objective B in question 1 above). ProRep implementing staff interviewed who had participated in earlier ProRep activities confirmed that ProRep’s cluster approach was a significant departure from original programming in which ProRep provided institutional support to the national legislature, national-level think tanks, and membership￾oriented CSOs to enhance representation in law-making in the national parliament--DPR. As detailed in the ProRep Project’s Midterm Evaluation, this was an inherently slow process that yielded few policy victories. The cluster approach, while still focused on increasing participation in decision-making, moved away from “law-making” in the parliament to policy-making in the government at large. Policy-making is primarily undertaken by the executive branch through national ministerial or presidential decrees. These policies require rule-making for implementation. That occurs primarily at sub-national levels of government. Rule-making at sub-national levels rather than at the national level provides opportunities for adaptation to different sub-national characteristics. The variety of sub-national experiences can then be aggregated to inform national policy improvements. ProRep cluster members interviewed by the ET stated that the policy cluster approach created interdependent (national and local level) relationships among stakeholders who advanced their organizations’ missions, while positioning them as more equal partners to government. The work of the environment cluster, detailed above, illustrates this finding. At the national level, representatives of an environment cluster indicated with pride that constructive engagement between their cluster and the President Joko Widodo’s administration resulted in the adoption of a number of the cluster’s recommendations with regard to the merging of the Environment and Forestry Ministries through Presidential Decree No. 16/2015 and has led the way to future collaborations.8 Also at the national level, Epistema Institute led a coalition involving NGOs, think tanks, public intellectuals, and others to formulate and endorse recommendations for Indonesia’s RPJMN 2015-2019. In Indonesia’s system of national development plans, RPJMN is the five year-reference for development planning at any level of government agencies including national, provincial, and local governments. Though it is hard to determine the impact of this recommendation, given the short period of performance of the Project, the ET confirmed consideration of the recommendation in the RPJMN, as noted in ProRep’s 5th Annual Report. “There are some similarities of what the coalition proposed and what the final document (RPJMN) contains, inter alia on the set target for social forestry areas, the presence of state in natural resources management, the strengthening of license moratorium in peat land, to name just a few.” These national works, in-line with new policy opportunities provided by the Constitutional Court Rulings No. 45/2011 and No. 35/2012 that exclude customary forests from state owned forests, have been simultaneously followed by local policy initiatives carried out by LePMIL, AKAR Bengkulu and Sulawesi Institute to initiate local regulations and policies that recognize customary land and the 8 See Annex VII for details on the regulations passed. 20 expansion of people-managed areas. In conducting all these national and local initiatives, as validated by policymakers interviewed, the ProRep grantees proved to be independent and credible resources. The ET found that many ProRep grantees reported that their previous experience working with communities through grassroots organizing did involve utilizing participatory assessment of policy objectives, opportunities, or challenges. However, According to respondents, the ProRep cluster approach encouraged grantees (and other participating organizations) to reorient themselves to what was possible to achieve, to identify who to work with, and to plan within the confines of resources and identified political will for reform. While common cluster objectives did not always meet advocacy priorities of individual organizations, the practical outcomes of a negotiated common agenda in the cluster and participation in administrative rule-making processes produced notable “wins” that incrementally advanced each organization’s mission. Cluster members moved ahead their negotiated policy agenda taking particular note of rule-making processes and procedures. Cluster members reported that the reorientation of their organizations to rule-making helped them “critically engage” with government on issues they believed were achievable. ProRep’s grantees reported that they had to not only adapt to the government rulemaking calendar, but also to the political environment, tight schedule of policy phases, and research needs of government officials. Interviews with the grantees revealed that all these considerations necessary in cluster work in order to advance negotiated common agendas reoriented the way they worked as organizations. An activist of AKAR and Sulawesi Institute, for instance, stated that they had never been in a project as tight as ProRep in terms of time and required output. They thought that this experience provided feedback to their organizational approach to doing policy projects and advocacy. This was also confirmed by other cluster members from the health sector. Conclusions Overall, the ET found that ProRrep’s cluster approach, as piloted, met its objectives (A and B) in two distinct ways: first, with regard to better informed and more representative processes. The ProRep cluster was designed as a new tool for CSOs to move from advocacy to more effective engagement in governing at both national and sub-national levels, requiring them to more systematically work with think tanks to better understand and subsequently produce materials relevant to government officials to achieve a successful policy outcome. Secondly, ProRep CSOs, think tanks, and government officials interviewed relayed that the cluster approach was effective at taking advantage of the opportunity gained through decentralization, i.e., moving substantial authority, resources, and responsibility for policy implementation to local governments. However, the Evaluation Team also found uneven understanding among government officials, CSOs and think tanks of the full range of options and responsibilities afforded the sub-national government units related to implementing policies. The cluster approach served as both a means (i.e. a set of coordinated actions) and an end (i.e., formalized working groups that energized citizen’s participation). By design, the process involved multiple actors at the local and national level in the process of governing. Preliminary indications are that the cluster approach does lead to policy outcomes. Learning, as well as relationships and expectations, were built. The cluster approach achieved success in objectives A and B (of question 1) by employing a strategy that included the following:  A focus on improving enacted national policy rather than promoting new policy; 21  Determining common objectives among cooperating CSOs, think tanks, and national level policy makers (aka clusters) related to the national and local policy;  Identifying opportunities in local level rule-making processes and procedures to advance agreed￾upon objectives;  Aggregating and analyzing local level results for national-level implications; and,  Producing recommendations for national level policy refinement based on local level experience. Figure 2 below illustrates these points by detailing the policy clusters’ engagement with national to local level entities. Figure 2: Cluster Engagement with National and Local Level Government Entities The ET found the cluster approach successfully created functional and complementary relationships among cluster members to advance targeted local level policy implementation objectives, which in turn sought to support policy reforms at the national level. However, as in any inherently political government activity, there could be many unanticipated events or roadblocks (e.g., competition among political leaders, powerful third party interest, or too small or narrowly focused policy objectives that may not have broad national applications) that could adversely affect cluster success. One of the key elements to the functioning of the cluster was the level and type of decentralization present in Indonesia. The work of clusters in local policymaking during the ProRep Project would not have been possible if there were no authorities granted to local government. The policy clusters approach increased CSO and think tank tactical skills in interacting with government to influence policy objectives. Conventional advocacy usually focuses on voicing demands. CSOs 22 involvement in the very beginning of setting common and negotiated agendas, and subsequently working hand-in-hand with government by infusing evidence and recommendations needed along the policymaking stages and processes, is quite a rarity. When working in the policy cluster approach, grantees were conditioned to adjust by mastering tactical skills to influence policy making. The ET believes that the approach may have yielded more results if grants were sequential in distribution (i.e., one small grant leading to a policy objective would lead to a second and third grant building on ascending achievements). With a six-month time frame in each of the two years, there simply was not enough time to implement such a strategy. Additionally, while the cluster approach did not deliberately reinforce the strategic values of community outreach, it did seem to utilize local communities in carrying out grant activities. Therefore, synchronizing cluster members’ capacity for community outreach and research at national and local levels appears necessary to ensure effectiveness in cluster activities. Lastly, ProRep’s cluster policy approach achieved sector-specific policy objectives while continuing to enhance USAID/Indonesia’s DO 1 of “Democratic governance strengthened.” The ET found that ProRep successfully moved from providing institutional support to the DPR and national level CSOs and think tanks to improve their representational and policy-making capacity (Phase I), to building organization capacity among CSOs and think tanks to effectively engage in policy making at local and national levels of government (Phase II). The organizations used this capacity to focus on sector-specific, pragmatic policy outcomes (consistent with USAID’s new Indonesia CDCS). They also continued to advance greater “representation” in policy making at multiple levels of government (primarily in the executive branch of government and local level legislative bodies). They supported broader and more robust civil society interactions and further consolidated gains made over two decades in decentralizing government. Recommendations *Executer(s) of recommendations are bolded.  The ET recommends that any new cluster sector formation in Indonesia supported by USAID or implemented by implementing partners should conduct a diagnostic study on the existing laws, regulations, and practices affected by decentralization to become acquainted with the range of options and responsibilities afforded the sub-national government units related to the policies they plan to influence. This will allow the clusters to map the local-national policy linkages, actors, needs as well as potential supporting resources such as political opportunities, allies and constituencies.  Specific time should be allocated within grants by implementing partners to develop work plans prior to launching specific cluster activities. Work plans should better align internal capabilities and operational reach of CSOs to meet negotiated, common policy objectives. QUESTION 2 According to stakeholders, what are contributing and inhibiting factors toward the achievement of these objectives? What could be done to increase the effectiveness of the clusters in achieving each of these objectives? Findings Contributing Factors In interviews with ProRep grantees, the ET found concurrence among grantees that the overall enabling environment for greater participation in policy making had improved significantly after the 2014 presidential elections in which Joko Widodo became President of Indonesia. The new President, who has a reputation for being a reformer and for seeking citizens’ input on government activities, has 23 brought new momentum across government for greater accountability and transparency. Most grantees interviewed voiced optimism about their current ability to engage government actors based on their interactions with high-level ministry officials in all three sectors (health, education, and environment). They credited President Widodo’s leadership as one of the primary reasons for this increased level of engagement. They reported that while, in practice, their experience with lower-level bureaucrats in sharing governance responsibilities varied (with some bureaucrats still working through how to implement more accountability into their respective offices), they were finding patterns of “less resistance” for more collaboration. One example was given by an Education Cluster member in describing the reception they have had thus far with the Ministry of Education. The President’s appointment of a former university president who is a champion of efforts to improve education throughout Indonesia9 as a case in point. The Education’s Cluster’s work on school management and teacher training under ProRep were not only aligned with the Minister’s vision, but he has encouraged key education directorates within the ministry to be active participants in discussion of policy and implementation with members of the Education Cluster, now known as KMSTP. Two decades of experience decentralizing government in Indonesia were also found to be a critical factor, further improving the environment for more inclusive participation in government decision making. Through decentralization, sub-national units of government have gained considerable authority, resources, and responsibility for enacting policies which were largely created at the national level. According to numerous respondents interviewed, many of those national policies lacked specificity for implementation, leaving local authorities with significant latitude to determine how policies would be carried out. ProRep’s focus on developing common policy objectives or agendas grounded in research was often voiced as an important contributing factor to cluster success. Respondents cited the policy cluster approach as different from conventional advocacy, in which CSOs traditionally worked outside of the governing processes to bring their agenda to government. A local bureaucrat asserted, “a CSO may bring good aspirations and ideas of policy changes; however, without an ability to build a common agenda with governments, such initiatives may not be able to be implemented.” CSOs interviewed noted another positive contributing factor from the cluster approach was the utilization of national/local networks. Coordinated national-local plans among CSOs were seen as magnifying the operational capacity of individual organizations. For example, in the environment cluster, a national think tank, experienced in bill drafting at the DPR level, guided and provided valuable support to several local CSOs who were engaged with provincial level Bureau of Law on local ordinances regarding forestry management. The national think tank, in return, received new evidence-based research important to their work with the national Ministry in Jakarta. The limited research capacity of subnational units of government provided an entry point for clusters. Through PreRep, local government officials expressed gratitude that CSOs were able to augment their capacity and meet their needs for more comprehensive data collection, as noted in question 1 above. The focus on time-sensitive policy objectives increased operational discipline among cluster members. Interviews with grantees in education, health and environment clusters revealed that the 9 by pioneering a program of sending volunteers, mostly fresh university graduates, to isolated, and poor Islands in Indonesia as teachers 24 ProRep grants required cluster members to organize their work around policy making stages of government and that their grant products were a series of inputs in each stage requiring more intense communications among cluster members and government agencies’ to ensure deadlines were met. The high rate of women’s participation in the cluster approach increased the representative nature of the activities, as noted during field interviews. Additionally, women leadership in the clusters was pronounced, and gender-focused and gender-led issues were supported by cluster objectives, such as the Health Cluster’s work on reforming insurance policies related to the inclusion of mid-wife services in health provision. Lastly, the selection of which CSOs and individuals to include in a cluster was key to helping the project gain credibility among government and think tanks. ProRep was good at identifying stakeholders who have not only strengths in a specific policy interest and who are well connected or influential, but also those who have demonstrated the constructive strategies for engaging with government. As explained earlier, in order to build a common agenda, it is necessary to have figures and institutions that have good track records, credibility and technical capacity. Inhibiting Factors By and large, policy cluster members agreed that the most challenging aspects of achieving policy objectives utilizing the cluster approach had less to do with external factors given the positive enabling environment noted above (although several cluster members mentioned that moving from pure advocacy towards participating in governance did require shifting behavioral norms within their own organizations) and more to do with internal capacity gaps, division of labor, and coordination amongst cluster members. These were seen as the largest challenges to cluster effectiveness (and are discussed in greater detail in question 4). Another inhibiting factor was that the rank and file in the civil service were not used to interacting with CSOs at an operational level; and without high-level buy-in from government officials, such participation by them was not always welcomed given the adversarial history of CSOs with government. The policy cluster approach requires think tanks and CSOs to internalize the governments’ perspectives as a central part of the cluster approach to governance. In other words, CSOs are no longer just external actors in governing, whose tasks are only to give inputs. However, several think tanks and CSOs voiced concern that they guard against being co-opted by government. They felt they needed to be sure that their engagement with government was still undertaken with a critical eye. Conclusions As the Indonesian government continues to support more transparency and accountability in its operations, and as local governments continue to exercise more authority, the cluster approach should gain more utility. The presence of a national enabling environment that accepts citizen participation in rule making and of government officials and bureaucrats who will participate in governance is important for cluster success. In the ProRep policy clusters, CSOs assumed a co-governing position in which they were integrated policy actors. Policy clusters negotiated common agendas (that were time-sensitive and focused), utilized local/national networks, and augmented individual cluster members’ capacities to achieve objectives. ProRep clusters also offered an opportunity for women to participate in advocating for policy objectives related to a variety of issues. ProRep also ensured the selection of capable cluster members. These factors contributed to the success of the clusters. 25 The clusters faced challenge and resistance when it came to working with civil service staff that have not yet caught on to the President’s reform agenda (and more welcoming attitude toward civil society). Clusters also faced expected challenges involving member coordination (division of labor, negotiation of common agenda, schedule of meetings) and individual organization/member internal capacities (including the shifting of organization behavior norms). Recommendations *Executer(s) of recommendations are bolded.  In the Indonesian context with a history of adversarial policymaker-CSO attitudes, the ET recommends that USAID use its reputation to enlist noteworthy/trusted individuals to help clusters gain access to policy makers. Early personal contact and communication with key government officials by respected individuals within a cluster or senior advisors to the cluster should coincide with cluster formation activities.  Informal communications/engagement by various cluster members with policy makers is important for building relations and such communications have a cost. Grants made by implementing partners should allow for this cost. QUESTION 3 In what ways did the policy clusters under ProRep influence national vs local policy? What are the differences or similarities in how clusters operate at these levels? Findings Research conducted by ProRep in 2011 and subsequent research by Australian Aid (now managed under the Department of Foreign Affairs and Trade - DFAT) in the KSI project indicated that evidence-based research from multiple sources was utilized in discrete and often limited ways by Indonesian national ministries or subnational units of government. In a seminal research paper entitled “The Political Economy of Policy- Making in Indonesia--Opportunities for Improving the Demand and Use of Knowledge,” KSI researchers noted: Ministers were expected to formulate policy and provide directives to echelon one level bureaucrats (Deputy Ministers, Director Generals – DGs – and Executive Secretaries) to implement policy. However, as policy-making became more complicated and wide reaching (e.g. due to globalization and urbanization) and particularly in cases where Ministers had limited technical expertise, Ministers increasingly formulated normative high level goals (perhaps reiterating presidential directives) and delegated responsibility for practical policy formulation or for rulemaking to put the policy into action to DGs and, more recently, (since their emergence) to Deputy Ministers.10 The research also indicated that Indonesia’s implementation of large-scale decentralization resulted in providing considerable power and resources to districts while reducing central government capacity (referenced above as a contributing factor to cluster success): 35 percent of the national budget and three million civil servants were transferred from central to local authority.11 These two factors are critical underpinnings and part of the political/historical context into which the cluster approach was introduced by ProRep. 10 Datta, Ajoy, Harry Jones, Vita Febriany, Dan Harris, Rika Kumala Dewi, Leni Wild, and John Young. The Political Economy of Policy-Making in Indonesia. Working Paper 340. London: Overseas Development Institute, 2011. 11 Ibid. Pp. 25. 26 The ET found that ProRep linked cluster members at national and local levels to achieve complementary policy objectives. At the local level, where policies were implemented, ProRep grants focused on rule￾making protocols associated with national policies. National ProRep cluster members utilized research and practice generated through their local cluster counterparts to provide specific national-level policy reform recommendations that were grounded in evidence and lessons accumulated through local level practical applications (as explained in question 1). As previously noted, ProRep clusters achieved cluster 10 objectives: eight at the local level and two at the national level. These objectives differed in that the focus at the local was on how to implement policy emanating from the national level. As described above, many of the local victories were related to new procedures. For example, the development and deployment of new protocols for an Short Message Service (SMS) platform (SMS Gateway) in Central Java to provide local communities real-time incidence reporting and monitoring of local health facilities was an important element/value added research for the Health Cluster’s national level objective of ”improving health policies and their implementation through enhanced public participation.” Over time and with additional successful local implementation of national policy, the cluster would, working with national ministries, recommend reforms consistent with what the cluster found worked at the local level. The national level policy contributions from the Environment Cluster illustrate this cycle. See the Epistema grant description in question 1 findings and Figure 1. Whether at the local or national level, effective communication strategies with officials in government agencies was key, as noted by respondents. Respondents cited the importance of producing user-friendly policy briefs and draft language for government officials to use, providing background materials to media on cluster policy objectives, and participating in support of their objectives in multiple hearings. They found these actions on their part were significant in helping officials move a policy objective through the official process. Informal communication often played a determining role in establishing trust between policymakers and think tanks/CSOs. Trust was paramount since civil society-state relationships had been historically characterized by conflicts. While informal communication strategies were effective, several cluster grantees stated that the ProRep grants were insufficient for them to conduct the range of communication activities that cluster members felt were needed. In fact, the number of discussions required for following-up formal workshops exceeded ProRep’s support. Some grantees stated that their organizations had to make further commitments to do additional non-budgeted activities for the sake of their policy reform agenda. Several CSOs pointed out that part of the success their cluster achieved in working with government officials was related to the fact that key individuals, whose expertise was already recognized by officials, built initial trust between grantees and government (establishing or building on existing credibility). For instance, even though KMSTP had nationally known CSOs like ICW in their cluster, they cited the participation of one nationally known expert in basic education as a key factor in opening doors at the Ministry of Education and Culture (MOEC). In the health cluster, as confirmed in the interview with F2H, the role of a professor as one of their experts was crucial to opening doors at the Ministry of Health (MOH) because the professor has been serving as expert/consultant to Ministry in other matters. LePMIL and Sulawesi Institute also validated this by asserting that part of their success is due to the personal relationships that helped open more institutional engagement between them and the government. 27 Conclusions The cluster approach supported by ProRep did not differ in its operationalization at the national vs local level. The approach was effective at utilizing the respective strengths of cluster members at both the national and local level to achieve policy objectives. Whether clusters were organized at national or local levels, they agreed to several core elements: 1. consisted of representatives from CSOs, think tanks, and policy making officials/institutions; 2. identified a common and achievable policy objective that was in the manageable interest and mission of participating CSOs; 3. developed an action plan based on identified objectives; 4. had practical familiarity and understanding of government decision-making processes/protocols; and, 5. provided evidenced-based research that advanced specific policy or rule-making objectives at the local level or policy reform at the national level. Figure 3 below displays these core elements in a pie chart. The policy cluster approach seemed to build confidence among members, and responded well to the emerging patterns of policy implementation at the local level and policymaking at the national level by working through both national and local organizations that empowered local level policy implementation while building momentum for national level policy reform. ProRep’s provision of tactical support to grantees government rule￾making processes was instrumental to cluster members’ selection of national level policy reforms that would be supported by policy implementation efforts of their local-level affiliates. Additionally, the use of government-friendly policy briefs and continuous feedback among cluster members were vital to ensuring momentum toward achievement of policy objectives among clusters at both the national and local level. Recommendations *Executer(s) of recommendations are bolded.  As part of future cluster-type grants, implementing partners should look to standardize training of key personnel in cluster CSOs on government and legislative protocols (e.g. hearings, policy mark-ups, budget and legal requirements, etc) at both national and sub-national units of government.  As part of the cluster operational plan, the ET recommends that future implementing partners support the development of a standalone strategic communication plan by the Cluster alongside its policy reform/implementation work plan. The communication plan should outline how the cluster will maximize broad public awareness of cluster objectives and accomplishments to highlight the work of the cluster(s) as a new way of governing. Figure 3: ProRep Cluster’s Core Elements 28 QUESTION 4 How has the policy cluster/community approach been operationalized differently (or similarly) in the sectors of environment, health, and education through ProRep? Have different methods of operationalization affected achievement of cluster objectives? Findings As background, ProRep initiated an inclusive, iterative (but somewhat precipitous) process to frame their vision for how clusters could be developed and what over-arching sector policies/objectives would be undertaken. The process involved multiple stakeholders, with significant input from USAID/Indonesia, along with quickly-developed expert working papers. Eleven grants were made in the second quarter of 2014 organized around education, environment, and health policy reforms with flexible implementation strategies. These grants included the stipulation that all grants had to produce results quickly--because there were no assurances that funding would continue beyond end of project year four, which meant grants had to end prior to August to allow for orderly close-out if future funding was curtailed. The ET found little difference in how the three clusters operated, but did find some differences among the Education, Health, and the Environment Clusters in the way they are organized at the national level. Over time, the Education Cluster formalized its national partners into a newly registered stand-alone entity called the Civil Coalition for Education Transparency (KMSTP) with over 40 members. The Health Cluster is organized as a semi-formal body, meeting regularly; but thus far, has no registered entity. ProRep’s national environmental groups utilized the cluster approach more autonomously (individual organizations led national and local coalitions) in pursuit of policy objectives. Respondents noted several factors that contributed to how formalized their clusters were at the national level. These factors are outlined in Table 2. Table 2: Cluster Formalization Factors Level of Cluster Formalization Contributing Factors More Formal  Consensus among members on priorities and policy objectives  Alignment of common policy objectives/agendas with individual cluster members’ agendas  Synchronization of policy makers’ calendars with cluster members’ priorities Less Formal  International funding disparities in sectors12  Ideological differences among members  Potential competition among cluster members for funding For example, one respondent stated that success in the education cluster is in part predicated on the fact that the sector has better and longer institutional relationships at all levels (paraphrasing). Additionally, the respondent observed more consensus among the (education) cluster members on specific policy agendas than in the environment sector (paraphrasing). 12 In certain sectors like environment, this created different incentives for cluster membership. 29 While sector clusters in the three sectors (and clusters at the national vs local level, as discussed in question 3) varied in their level of formality, the ET found that all clusters shared core elements in how they operated that made them ProRep Clusters, as detailed in question 3. ProRep grantees described the cluster policy approach as both an end, i.e., the formal establishment of a community of like-minded CSOs, think tanks, and targeted policy makers organized around a broad policy objective; and as a means, i.e., a set of evolving protocols or a methodology utilized to effect a specific policy objective/policy implementation (usually at the local level). The majority of respondents agreed that the cluster approach had the following similar operational characteristics, regardless of sector or level:  negotiated process for selecting policy objectives;  focused attention on policy making rules and the needs of policy makers to achieve a specific policy objective that was formally under consideration;  emphasized collaboration among cluster members for mutual gain over competition;  encouraged more comprehensive engagement with respective bureaucracies e.g., understanding the needs of other key government units that are part of the rule-making process such as legal and budget functions;  institutionalized feedback processes, particularly between national and local counterparts, in a mutually supportive environment; and,  enhanced political/government communication skills among cluster members. The ET found that among all three clusters, grantees gave generally positive feedback regarding the value of the cluster approach. At the same time, many CSOs and think tank members in each of the three clusters stated that the cluster approach was at times cumbersome, requiring “too many meetings” to develop goals and work plans. Moreover, cluster grants had short turn-around times to achieve results. To some CSOs, these times were too short and somewhat constrained their ability to take on more challenging policy objectives. Further, one local-level cluster participant noted that they had to work at a faster pace than their normal habit and engage more extensively with local government officials. Had they not already had longstanding (mostly informal) relationships at the local level, implementing their ProRep activities would have been significantly harder to achieve. Virtually all CSOs and think tanks were quick to note that ProRep’s use of Letters of Cooperation (LOCs)--which were in-kind expenditures either directly contracted by ProRep or provided by ProRep staff--were timely and important to their meeting grant milestones. Additionally, CSO respondents uniformly expressed gratitude for the added administrative, planning, communications, and other supports provided by ProRep. They credit their ability to generate useful research to ProRep’s support; and they credit the cluster’s national/local nexus for improving their access and engagement with multiple levels of government. One externality noted by several CSOs was that in many instances, the policy objective achieved by the cluster required further follow-up with implementation that the grant did not cover (also referenced in question 2). Without follow-up, they worried that their newly enacted policy could be at risk of being weakened or even distorted during implementation. None the less, several CSOs stated that the early success they had with the cluster approach set higher expectations from local policy makers who received useful research that was admittedly beyond their somewhat limited capacity. Additionally, most CSOs and think tanks interviewed, regardless of sector, concurred that ProRep’s insistence of connecting local policy implementation (rule-making) activities to national level policy reforms increased their confidence that the research they generated could have meaningful impact on their overall mission and enhanced their credibility with participating policy officials. 30 Conclusions The ET did not find any significant structural or fundamental differences among sectors that would require different modalities for operationalizing sector clusters in the future; therefore, the cluster approach in each sector did not impact achievement of objectives differently. Rather, ProRep utilized learning from one sector cluster over time to influence or improve other clusters formation. ProRep grantees in all three clusters grantees gave positive feedback regarding the value of the cluster approach However, a general willingness to negotiate policy objectives among cluster members who may have ideological differences and disparities in funding opportunities, mostly because of differing priorities for international donors, were observed as conditions that could affect the level of formal cluster cohesion over time. Additionally, complaints raised by ProRep CSOs and think tanks concerning their perception of the cluster process as “cumbersome” indicates that the current process can be a disincentive for participation. The ET believes that the ultimate success in the cluster approach rests on individual cluster members’ ability to (as seamlessly as possible) integrate their organization’s activities into the broader objectives of the whole cluster. Recommendation *Executer(s) of recommendations are bolded.  Today, many complex public policy issues are the domain of professional public policy mediation experts who are skilled at facilitating multiple parties in reaching consensus on common agendas and action plans. “Public policy mediation creates a (highly structured) forum for deliberative negotiations among government, stakeholders and, when appropriate, the public. The parties’ contributions of their technical expertise and the resulting greater knowledge of participants’ preferences, are woven into discussions that increase mutual understanding and lead to otherwise untapped opportunities for consensus agreements. The intended result is an agreement that sets forth the terms of the future relationships and responsibilities among the parties with regard to the issues they discussed.”13 Future consideration of policy cluster designs and formations by USAID would continue to benefit from facilitated dialogues among potential cluster members and other interested parties. In the future such dialogues should be guided by policy mediation experts, who possess substantive expertise in government processes, planning, or a related discipline to save time and facilitate actionable agreements early on that ProRep clusters spent working out over time (which would result in lowering transactional cost). Policy mediation experts would systematically conduct confidential interviews with multiple stakeholders and joint fact-finding; establish cluster ground rules; negotiate themes prior to actual goal-setting; develop contingency plans to mitigate risk; and sequence agreements. Policy mediation experts act as honest brokers and execute multiple tasks associated with multi-party coordination through more structured dialogues designed to facilitate and drive parties to agreements. Such experts are skilled at transforming negotiated outcomes into formal operating mechanisms that will assist USAID to better anticipate events during the life of the cluster initiative. By streamlining the front-end of cluster formation through the use of skilled policy mediation expertise, the ET believes that concerns noted by CSOs of the cumbersome process would be minimized. 13 Public Policy Mediation by Howard S. Bellman and Susan L. Podziba, Dispute Resolution Magazine, Winter 2012. 31 QUESTION 5 How did ProRep’s management of partnerships (through grants, letters of collaboration, and other mechanisms) contribute to or inhibit the achievement of policy cluster objectives? Findings Respondents noted that Fixed Amount Awards (FAA) and Letters of Collaboration (LOCs) were flexible tools that allowed clusters to meet their objectives. FAA grantees were required to meet milestones (e.g., sponsor an event) to receive funds, and did not have to submit receipts or share accounting records. This lessened the administrative and financial burdens for grantees that allowed them more time to focus on programmatic milestones. Through LOCs, gaps in grant proposals were filled or unanticipated issues were handled through in kind activities or purchases from ProRep. LOCs helped grantees maintain momentum in their activities that was important due to short grant time frames. The ET found that the use of FAAs and LOCs to manage partnerships, however, significantly increased the quantity and quality of duties for ProRep project staff. For a list of mechanisms utilized in ProRep for management of partners, see Annex VI. ProRep staff had to assume multiple roles as: grantor, convener, facilitator, technical expert, financial oversight manager, negotiator, planner, mentor, and coach throughout the short lifespan of individual grants. Additionally, the ET found that the short turnaround time for actual cluster programming was a constraint. Essentially there were approximately six months of program activities wedged between start-up and the close out of grants14 which was an important factor that contributed to how clusters were formed, the type of policy initiatives targeted, as well as the selection of cluster grantees both in terms of range of participants available and their relative capacity to engage and produce results quickly. These factors significantly expanded the array of day-to-day responsibilities for ProRep staff. One staff member conveyed that the need for speed and results increased “transactional costs”, in terms of the time and variety of interventions, required by each staff member to effectively implement critical aspects of the project (which included numerous multi-stakeholder forums, enhanced communications training and the rapid production of research and policy briefs, and more one on one coaching and mentoring of individual grantees, etc). In effect, ProRep had to be an operations supervisor, utilizing its own internal infrastructure to supplement that of grantees to ensure that clusters functioned smoothly. Cluster grants were of modest size (averaging approximately $60,000 each) and focused on short-term achievable results. However, that did not mean that USAID grants compliance standards were any less required or minimized. Indeed, a significant amount of assistance to grantees revolved around grant administrative compliance which fell to ProRep staff to ensure. One notable example involved a grantee who, during the implementation of its ProRep grant, experienced significant employee turnover. That turnover signaled to ProRep staff potential problems in performance and possible delays in meeting cluster policy objectives/milestones. ProRep knew in advance through their standard pre-award audit that the grantee had sufficient experience in carrying out the activities of the grant, but lacked advanced management systems. ProRep flagged this concern and included financial management standards training and at least three site-visits during implementation as part of the grant. By investing in this early exercise, ProRep staff was able to both identify and assist the grantee in a timely manner. They helped the grantee reconcile financial issues which were related to staff turnover and provided real-time support and training to use a “simple Excel-based accounting system, payment forms requiring clear segregation of duties, inventory tracker and check-out forms”, and by conducting an intensive organizational planning exercise to clarify the grantee’s organizational and management structure. The 14 ProRep’s final two years were on a year-to-year option basis. The base contract included three years. 32 grantee completed their grant successfully, achieving a policy victory in securing the issuance of a provincial governor’s decree. Multiple grantees noted that responsibilities such as the need for continuous feedback loops among cluster members and ProRep staff, required that they had to become “Master of Milestones” and develop more robust operational protocols to keep pace with program objectives while still meeting other institutional objectives of their respective organizations. During ET field visits, no complaints of micro-management by ProRep staff were offered by cluster grantees. Virtually all of the cluster grantees interviewed were appreciative of ProRep’s assistance. They indicated that many of the administrative and reporting burdens usually associated with USAID standard grants were minimized through the use of FAAs. At the same time, they indicated that use of FAAs demanded more precise forward planning and day-to-day management to ensure they met their targets. Several CSOs went so far as to praise the faster pace of activities through FAAs, citing the discipline required as instructional for increasing their organization’s overall operational capacity to pursue multiple goals. The ET found that ProRep’s operational support was critical to the grantees’ success. Conclusions The ET found that ProRep’s use of FAAs and LOCs (the two primary procurement instruments) were critical and appropriate management mechanisms for providing technical and administrative support to grantees to achieve demonstrable policy victories for sector clusters. But there also appeared to be a high correlation between the institutional capacity of the implementer (Chemonics) to provide a wide range of technical and administrative support in a timely manner to ensure cluster success, suggesting that a ProRep grantee CSO had to have more than minimal internal capacity to manage time and resources within their respective organizations to partner with ProRep. Absent the constraint of uncertain future funding and the need to have demonstrable short-term results in yearly increments, which impacted the selection of grantees, the ET believes that given enough time, FAAs and LOCs could be utilized to build institutional capacity of more grantees, particularly those who may not already have high levels of administrative capacity but who may potentially be future critical lead organizations in a cluster. Second and third FAAs to respective CSOs within a cluster could be incorporated as institutional capacity of individual cluster members rise and perhaps be utilized to achieve multiple policy outcomes in that respective sector. FAA grants and the use of LOCs lend themselves more readily to the dynamic environment in which policy-making and policy implementing takes place. Given the fact that highly-dynamic policy implementation take place mainly at the sub-national level (involving government official rotation and changing political interests), FAA and LOCs were ideal for ProRep grantees in order to achieve their milestones, since these mechanisms are very practical and flexible to deal with changes. Given that ProRep cluster grants focused on grantees achieving operational milestones, the use of LOCs is essential to filling operational and programmatic gaps of cluster members during the life of a grant to ensure smooth functioning. ProRep acted as convener, funder, negotiator, and technical resource for potential cluster members interested in applying for grants under the new cluster approach, while simultaneously phasing-out key institutional support elements, activities, and partners from its first three years of operation. The ET believes these were too many roles to fill for the implementer, which had an impact on virtually every aspect of cluster programming. Recommendations *Executer(s) of recommendations are bolded.  USAID should consider using a co-creation process to design any follow-on cluster related project. Co-creation allows for more dialogue and input from more stakeholders and creates a more transparent process for project design. Since this is an evolving approach to both program 33 design and policy reform, getting additional ideas on how clusters might be formed and operated could enrich a follow-on project design.  USAID should continue the utilization of FAAs and increase use of LOCs for timely tactical assistance to grantees. The experience of having to meet milestones to receive payment was seen to have value for future operations, while the requirement of previous grants to respond to USAID administrative requirements was not seen to add value to the organizations’ operations.  Future implementing partners should develop a grants management database system that goes beyond M&E functions to include overall program operations. The addition of a database grants management platform could standardize management practices and reporting among the clusters to simplify program management by an implementer. QUESTION 6 What aspects of sustainability supported or promoted by ProRep are present in the policy clusters? What are the challenges to sustainability for each cluster? Findings The ET found that it is too early to confirm the sustainability of any of the clusters supported by ProRep. However, the ET found notable cluster achievements that may contribute to sustainability in the future. First, the Education Cluster has formalized membership through the creation of a newly registered national organization with a membership of over 40 organizations committed to utilizing the cluster approach. Second, members of the Health Cluster have, in principle, agreed to jointly pursue future funding as a cluster. While not conclusive of the long-term durability of clusters, the ET found that these two outcomes evidenced forward momentum beyond the ProRep project. The ProRep cluster approach had several underlying characteristics that appear to be scalable: 1. By targeting achievable and pragmatic policy objectives, clusters built confidence in their ability to influence policy. ProRep had 10 policy wins along with 11 other policy objectives still under consideration in relatively short-time spans. 2. Several ProRep cluster members volunteered that the policy objectives achieved through the cluster approach have engendered follow-up implementation responsibilities and/or activities for cluster CSOs within their respective constituents. 3. Most CSOs reported that by focusing attention on “policy/rule-making,” they had acquired new practical knowledge and “critical engagement” skills that are being internalized into their respective organization’s mission. 4. Finally, the national/local structure of the clusters, the subsequent mutually re-enforcing collection of data, and increased access to multiple-levels of government among cluster members increased each member’s political effectiveness, operational capacity, and reach in a manner that did not increase overhead for their respective organization. In effect, by cluster members collaborating on discrete policy objectives, which drew on relative strengths of participating members, members created a policy-making function for their own organization greater than they could/would have done alone. During the course of the field interviews, several CSOs reported that demand for the type of research generated through their cluster approach (i.e. very specific and comprehensive data relevant to a policy/rule making currently under consideration) had increased. One public official interviewed relayed to the ET that future participation in a ProRep policy cluster initiative would be welcomed. 34 The ET found that the comprehensive and targeted (rule-making) nature of the research generated through the cluster approach increased and expanded the depth and capacity of government research (more so at the local level) while also enhancing the relevance of participating CSOs in policy￾making/implementation. Moreover, virtually all of the public officials interviewed agreed that the process was effective in demonstrating the government’s commitment to greater transparency in a way that was mutually beneficial. The ET found the single most reported challenge to the longer-term viability of a cluster or the cluster approach involved potential competition among cluster members for funding. Several CSOs noted that coalitions and collaborations among civil society actors was nothing new; indeed many of the ProRep CSOs are or have participated in collaborative efforts in the past, with mixed results. Several national cluster leaders in environment noted that some of past collaborations had produced a coalition entity that became a competitor to its members. Conclusions While ProRep’s approach to cluster formation has resulted in formal, semi-formal and non-binding cooperation platforms for different sectors, no model has been proven. More experience with this approach is needed to draw a conclusion about sustainability. The ET notes, however, that respondents reported aspects of sustainability (or commitment to the core elements found in the cluster approach noted above as having already been internalized into their respective organizations). Aspects of the cluster approach may be attractive to donors. It is a cost-effective approach to achieving policy outcomes or objectives, and clusters demonstrated the ability to enhance institutional capacity of cluster member organizations. The cluster approach effectively facilitated the creation of policy/legislative arms within each member organization. These are promising achievements. Recommendations *Executer(s) of recommendations are bolded. • USAID should encourage the formation of cluster secretariats to replace the implementing partner(s) as lead convener(s) of cluster activities and as feasible. The ET recommends that cluster grants from implementing partners include provisions for clusters to obtain their own institutional support, with the caveat that protocols should be established that ensures that secretariats do not become a competitor to its members. • USAID and implementing partners should encourage joint solicitation of new funding by sector clusters, particularly in the environment sector where there appears to be significantly high interest from private sector and international donors. 35 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK Indonesia Program Representasi Scope of Work Evaluation of Policy Cluster Approach I. Summary The Indonesia Program Representasi (ProRep) Project requires the services of a team of three (3) consultants to conduct an evaluation of the policy cluster approach as developed and implemented by the program; the findings from which will be utilized in the determination of future USAID programming in multiple sectors in Indonesia. II. Project Overview Program Representasi is a five-year USAID-funded project designed to strengthen democracy and good governance in Indonesia by promoting better informed and more representative legislative and policymaking processes. Phase I (Project Years 1-3): During its base contract period, Years 1 – 3, ProRep focused on strengthening the three sectors listed below (items 1 – 3), and conducted special initiatives as agreed with USAID (item 4): 1. Civil Society Organizations (CSOs), particularly non-profit NGOs that are membership-or constituent-based. Most such organizations are national in purpose; many represent women and other marginalized groups and engage in advocacy on key policy issues. 2. Public policy research institutions, including research groups engaged in applied or policy research in universities, research centers, and CSOs. 3. Parliament’s two national houses, the Indonesian House of Representatives, Dewan Perwakilan Rakyat (DPR) and, to the extent agreed with USAID, Indonesia’s Regional Representative Council, the Dewan Perwakilan Daerah (DPD). 4. Timely assistance for special initiatives needed to protect or advance democratic governance. ProRep helped selected CSOs and CSO networks to develop larger and more defined constituencies, expand memberships and stakeholders, and become more informed and constructive advocates. ProRep also assisted them in finding positive ways to work with parliaments, especially the DPR, as well as other local- and national-level policymakers and implementers. Second, ProRep worked with research organizations to improve the quality of policy research and research on legislation, and to enhance its dissemination to both national- and local-level policymakers, allowing civil society, parliament, and others to develop better- informed approaches to formulating and implementing new laws, to overseeing how laws are actually implemented and their impact on people, and for parliament members to better 36 respond to their constituents. Third, the project worked to meet a number of defined needs of the DPR (and DPD, as needed) and improve its national standing by helping members represent their constituents more effectively, transparently, and in a better-informed manner. The Special Initiatives component (also called Strategic Activities Fund/SAF) was designed to enable USAID to respond flexibly and rapidly to address other unanticipated needs and opportunities pertaining to the protection and advancement of democratic governance in Indonesia. Under this project component ProRep implemented a number of select activities to advance democratic governance, including study visits and peer-to-peer exchanges, and carried out a DPR Member constituent outreach program (JABAT) to help bridge the wide gap between DPR Members and their constituents, providing venues and opportunities for greater, and more meaningful contact between citizens and the DPR Members (MPs – Members of Parliament) than has been the case in the past. Phase 2 (Project Years 4-5): In Project Year 4, and in line with USAID’s 2014 – 2018 County Development Cooperation Strategy (CDCS) released in November 2013, ProRep institutional support to Indonesian parliaments ended. Institutional strengthening support to civil society and think tanks was scaled back, with the exception of continued institution- building assistance to the ProRep initiated Policy Research Network (an association of think tanks and research orientated CSOs). Most of ProRep’s assistance shifted to facilitating multi-sector policy communities or “policy clusters.” This innovative systems approach built upon ProRep’s experience over the previous three years, and sought to bring CSOs, think tanks, and policymakers together to collaborate in support of specific policy reforms in the sectors of education, health, and environment. These policy communities were designed to become platforms to facilitate interactions among inter-connected actors and to further inculcate an inclusive and evidence-based approach to policymaking. Under the Special Initiatives or SAF component, ProRep began supporting, among other initiatives, Indonesia’s National Coordination Team (NCT) in their development of a South- South and Triangular Cooperation (SSTC) program, a program to reactivate the Indonesian Secretariat for the Open Government Partnership (OGP). Also under the SAF component, to counter specific threats to Indonesia’s progress in eliminating corruption, in early 2015 ProRep launched a fourth cluster in the area of democratic rights and governance (DRG). This cluster aims to enhance the advocacy, coordination and policy reform efforts of groups promoting greater accountability in Indonesia. In early 2013, ProRep, through subcontractor Social Impact, conducted a mid-term evaluation, resulting in a number of findings, conclusions, and recommendations presented to USAID. The revised program design in Phase 2 (Project Year 4 and 5) and an updated Performance Management Plan (PMP) submitted to USAID addressed a number of these recommendations. III. Policy Cluster Approach During ProRep’s base period (Years 1 – 3 described above), the project increasingly brought together CSOs, policy experts, and policymakers to work together for policy reforms. Building on this experience, and recognizing that achieving many of the objectives of USAID’s new CDCS would require national and local level policy changes, the USAID Democracy and Governance Office and ProRep collaborated to develop the policy cluster approach. Through the implementation of the policy cluster approach, ProRep continued to work towards its overall program objective of “better informed and more representative legislative and policymaking processes” while also promoting the passage of policy improvements or better policy implementation, addressing critical development areas for Indonesia (i.e. health, education, environment, democracy). 37 Policy and law-making in Indonesia, as in many nations, tends to take place within ministries and parliament, with little if any interaction with other sectors. This occurs without taking into account evidence-based policy recommendations made by policy experts and local constituents’ policy interests represented by CSOs. Neither tends to have sufficient understanding of how policies are made, or how to be effective advocates for policy change. These policy clusters or policy communities (terms used interchangeably) were designed in this context to bring together national and local-level policymakers from the executive and legislative branches, experts, and advocates, to help them collaborate more effectively in improving policy or policy implementation in selected policy-focus areas. Policy clusters are designed to be (1) inclusive, involving multiple stakeholders, from many sectors, (2) evidence-based, using the results of empirical research, and, (3) to involve the media, so that clusters’ positions have an impact on the opinions of decision-makers and the public. ProRep has supported these clusters through the provision of grants to CSOs and research organizations, technical assistance, support for focus group discussions, meetings, and parliamentary hearings, and other means of support. IV. Evaluation Context and Objectives Program Representasi will be completing its final year in April 2016, and USAID has requested that Chemonics conduct a final evaluation as per Section F.7.12 of the Project Contract states the following: The Contractor will conduct a final independent evaluation of the project within the final quarter of the final contract performance year. The focus of the evaluation will be to assess the achievements of the project versus the stated objectives and goals, to identify which elements of the project had the most significant impact and which did not, and which aspects of project design need to be considered for continuation under future projects. The evaluation team is to be comprised of technical experts/evaluators who are independent of the Contractor and the Contractor’s staff. The final evaluation report will be submitted to the COTR, with a copy provided to the Contracting Officer, no later than the final date of the contract performance period. In compliance with Contract requirements, ProRep, in agreement with USAID and through subcontractor Social Impact, will conduct an independent final evaluation of the ProRep Project, focused on the policy cluster approach as developed and implemented by the program. The objective of the evaluation, as stated in the Contract, will include identifying which elements of the project were most impactful, which were not, and how to integrate end-of-of project learning into future USAID programming. USAID clarified in November 2015 that “the project”, or the focus of the evaluation, should be the policy cluster approach, as was implemented in Year 4 and 5. The three-person team (one expatriate and two Indonesians) will, among them, have sufficient expertise in evaluation, civil society, and in- depth understanding of policy-making processes in Indonesia, to be able to assess the policy cluster approach and make recommendations regarding future program direction. USAID will provide technical concurrence for the selection of each of the evaluators on the team, and any changes to this Scope of Work. The objectives of the evaluation are as follows: 5. To evaluate ProRep’s Year 4 and 5 use of policy clusters to improve the policy making process and promote policy reforms; and, 38 2. To generate key lessons learned and identify promising practices related to the policy cluster approach for replication in future USAID programs (in various sectors). V. Evaluation Questions The evaluation will review, analyze, and evaluate ProRep’s implementation of the policy cluster approach according to the questions below. In answering these questions, the Evaluation Team should assess the performance and role of both USAID and its implementing partner(s) (IPs). 1. In what ways is the policy cluster approach, as implemented through ProRep, affecting the following objectives: a) bringing about better informed and more representative legislative and policymaking processes b) promoting the passage of policy improvements or better policy implementation 2. According to stakeholders, what are contributing and inhibiting factors toward the achievement of these objectives? What could be done to increase the effectiveness of the clusters in achieving each of these objectives? 3. In what ways did the policy clusters under ProRep influence national vs local level policy? What are the differences or similarities in how clusters operate at these levels? 4. How has the policy cluster/community approach been operationalized differently (or similarly) in the sectors of environment, health, and education through ProRep? Have different methods of operationalization affected achievement of cluster objectives? 5. How did ProRep’s management of partnerships (through grants, letters of collaboration, and other mechanisms) contribute to or inhibit the achievement of policy cluster objectives? 6. What aspects of sustainability supported or promoted by ProRep are present in the policy clusters? What are the challenges to sustainability for each cluster? VI. Draft Methodology The detailed methodology of the final evaluation will be designed by the evaluation team and presented in the work plan. Some illustrative methods (to be finalized by the evaluation team in coordination with the Mission), include:  Conduct background reading and preparation prior to beginning work in Indonesia. ProRep will provide the team with electronic copies of all relevant project documents, including the contract, PMP, assessments, quarterly and annual reports, work plans, newsletters, success stories, and case studies (as they relate to the policy cluster approach and the evaluation questions stated here). 39  In-briefing with USAID/Indonesia detailing USAID’s objectives and specific issues the team should address, soon after the team arrives in Jakarta.  Key informant interviews with USAID, Chemonics Home Office staff involved with ProRep, ProRep project office staff, project counterparts (i.e., CSOs, research organizations, local- and national-level policymakers from legislative and executive branches of government), and others to gather information on ProRep, its development, and its results.  Focus groups with civil society organizations, journalists, public intellectuals, etc., informed of ProRep’s program. Data collection, including key informant interviews and focus group discussions, are expected to take place mostly in Jakarta with approximately five (5) days in one or two locations outside of Jakarta to meet with project partners and beneficiaries. These locations will be determined by USAID by December 31, 2015. To minimize potential bias in data collection, the evaluation team will select all respondents with the objective of ensuring that samples of interview sources are sufficient in number, scope, and diversity to qualitatively support evaluation findings. ProRep will provide the evaluation team with a list of recommended sources, along with their level and nature of interaction with the project. ProRep will also be available to support the team in arranging for interviews of selected sources. Prior to travel, the evaluation team will review the proposed methodology and alternative methods in light of the evaluation questions, timeframe, budget, data collection requirements, quality of existing data sources, and potential biases. The team will build on the proposed methodology and provide more specific details on the evaluation methodology in the evaluation work plan (see Deliverables below). They will incorporate draft data collection guides into the work plan. Interview tools or checklists of about 10-12 items will be prepared for each type of interview or discussion. The tools will be shared with USAID at the in- brief and as part of the evaluation report. The evaluation team will complete a preliminary findings presentation and present it to USAID prior to departing from Indonesia. The team members will then collaborate in drafting a detailed report to USAID over the following two weeks. VII. Deliverables 1. A draft work plan for the evaluation including design and time frame 2. A final work plan for the evaluation including design, time frame, and data collection tools 3. Weekly updates (from Team Leader and Program Manager) during fieldwork 4. Presentation of preliminary findings to USAID/Indonesia 5. Presentation of preliminary findings to implementer (Chemonics) 6. A draft report on the evaluation, as described below 40 7. Final Report (within six (6) business days of feedback from USAID on draft report, which is expected within five (5) business days of receipt of the draft report) The final (and draft) report should meet the following criteria to ensure the quality of the report:  The final evaluation report should represent a thoughtful, well-researched and well- organized effort to objectively evaluate what has worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the SOW.  The evaluation report should include the SOW as an Annex. All modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by USAID.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an Annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The format of the final evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, lessons learned and recommendations. The report should include, in the Annex, any substantially dissenting views by any team member or by USAID on any of the findings or recommendations. The report should not exceed 30 pages, excluding Annexes. The report will be submitted electronically in English. The report will be disseminated within USAID. A second version of this report excluding any potentially sensitive information will be submitted (also electronically, in English) by Chemonics to USAID’s Development Experience Clearinghouse (DEC) for dissemination among implementing partners and stakeholders. VIII. Team Composition 
 Team Leader Qualifications – a senior level specialist with experience designing and evaluating similar 41 democracy and governance programs, a proven track record supervising teams in the field and producing reports, and technical knowledge and expertise in at least one of ProRep’s three main component areas (i.e., CSOs, think tanks/research organizations, and parliaments) in addition to in-depth understanding of policy-making processes. The Team Leader will:  Finalize and negotiate with USAID/Indonesia the evaluation work plan; Establish evaluation team roles, responsibilities and tasks;  Facilitate all necessary meetings;  Ensure that logistical arrangements in the field are complete; Coordinate schedules to ensure timely production of deliverables;  Coordinate the process of assembling individual input/findings for the evaluation report;  Lead the oral and written preparation and presentation of key evaluation findings and recommendations to USAID/Indonesia Local Team Member Qualifications - Two Indonesian team members will work alongside the Team Leader to complete the evaluation. One team member will bring substantial expertise in Democracy and Governance while the second will have qualifications focused in monitoring and evaluation. One will be considered mid-level and the other a mid or junior- level staff person. Both should have a background working on USAID-funded projects (or similar in the development sphere), experience with report writing and excellent English language skills. IX. Logistical Support The Evaluation Team is responsible for arranging all logistical support for this exercise. SI staff, in Jakarta and DC, will be available to support the team in all logistical matters, including arranging transportation and lodging. The ProRep Project and USAID will be available to provide logistical support, as needed, including scheduling requested interviews. The ProRep Chief Of Party (COP) and staff will make themselves available to the team to answer questions and provide other support to the team, as needed. In addition, the ProRep Project team will make available working spaces, phones, and Internet access to the team members, should they wish to use them. X. Oversight and Management The Team Leader will liaise closely with USAID, and will be requested to provide input and feedback at periodic intervals during the evaluation process. All deliverables will be submitted by the evaluation team to Social Impact for final quality assurance review, formatting, and editing prior to submission to USAID. To ensure independence of the evaluation team, all recommended changes during the quality assurance review must be approved by the evaluation Team Leader. In the event of disagreement, the Team Leader will submit as an Annex to the report any differences of opinion. 42 XI. Timeline and Level Of Effort (LOE) Description LOE in Days Team Leader (Expatriate) Team Member 2 (Indonesian) Team Member 3 (Indonesian) Review of background documents and development of work plan 3 2 2 Team Leader travel USA to Indonesia 2 0 0 Team planning meeting with USAID and others in Jakarta 1/2 1/2 1/2 Data collection, meeting with key informants, site visits, observations (Jakarta) 6 6 6 Data collection, meeting with key informants, site visits, observations (outside Jakarta, locations TBD) 5 5 5 Production of report findings/recommendations/begin drafting report 5 3 3 Presentation of findings/recommendations to USAID 1/2 1/2 1/2 Continue production of draft report in￾country and include additional interview as needed 3 2 2 Team Leader travel back to USA 1 0 0 Continue to draft report 3 2 2 Final report, incorporate comments 1 1 1 Total 30 22 22 Dates: The expatriate Team Leader will work 19 days in Indonesia, beginning o/a January 11, 2016. Approval for a six-day `workweek will be requested to USAID. While in Indonesia, the team will agree with USAID on a date to submit the draft report for comments. They will submit their final report within seven (7) business days of receiving feedback from USAID. 43 ANNEX II: DATA COLLECTION INSTRUMENTS Key Informant Interview Protocols ProRep Final Evaluation Coversheet for all Key Informant Interviews Date of Interview: Interviewee Name, Title: Team: Time Start: Time End: Interviewer(s): Location: Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are working to conduct a final evaluation of the ProRep Project. The evaluation is intended to provide an informed evaluation of the policy cluster approach to generate key lessons learned and identify promising practices for replication in future USAID programs. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to discuss with you today to hear about your experience, in your own words, in order to help us better understand how this project looks and functions “on the ground.” The interview will last around 45 minutes. You can decide to stop the interview at any time. We will take notes of the interview and, with your consent, a recording will also be made using a digital voice recorder to aid us in fully completing our notes after the interview. Confidentiality Protocol • We will collect information on individuals’ names, organizations and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purpose of this evaluation, and will not be shared with other audiences or used for any other purpose. • Your participation in this interview is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? 44 ProRep Final Evaluation (Est. Time: 45 min) KII, USAID Staff  How are you connected to the ProRep program?  To what extant where the first three years of ProRep necessary to the success of the cluster approach?  Were there policy expectations built into the sector activities before engaging the ProRep cluster approach? Key Evaluation Question 1:  How do you see the current quality of representation and policymaking process in Indonesia?  In what ways is the policy cluster approach as implemented through ProRep bringing about better informed and more representative legislative and policymaking processes? Please provide specific examples.  In what ways is the policy cluster approach as implemented through ProRep promoting the passage of policy improvements or better policy implementation? Please provide specific examples.  How are the initiatives within the cluster approach relevant to USAID strategies and programming priorities at national and global level?  What is the importance of promoting better information and inclusion of public participation in legislative representation and policymaking processes?  What do you think about how the partners (CSOs, Think Tanks, and policy makers) were identified at the launch of the policy cluster approach?  What is the strength and weakness of the overall program activities? Key Evaluation Question 2: • What have been the main challenges that might have affected achievement of program objectives? • What are the supporting factors regarding Indonesian legal, political, or social frameworks to the program and its objectives? • Are there any constraining factors regarding Indonesian legal, political, or social framework in relation to the program and its objectives? • Are there other contributing or constraining factors you have identified? If there are some constraints, please describe (e.g. program design challenges, implementation challenges, or other). • What could be done to increase the effectiveness of the clusters in achieving each of these objectives? Key Evaluation Question 3: What are the most significant changes in policy making process at the national level that you observed? At the local level that you observed? o What differences do you see in legislative representation or policymaking at the national level since the program began? o What differences do you see in legislative representation and policymaking at the local level since the program began? What (or whom) do you attribute these changes to? What aspects of the ProRep program promoted the gains you just described at the national and local level? 45 What key practices have the policy clusters implemented/utilized to make the gains at the national and/or local level that you just described? Other:  Please describe ProRep’s gender inclusion efforts and the policy cluster’s gender inclusion efforts. How do project activities/cluster activities ensure participation by both men and women?  Do you have any other comments, suggestions or concerns about the policy cluster approach as implemented by ProRep that you would like to share with us? 46 ProRep Final Evaluation (Est. Tim: 45 min) KII, ProRep Staff • Describe your role in the ProRep project. How long have you been with the project? • Have there been any changes to the scope or program activities (related to the policy cluster approach only)? If yes, please describe. • To what extant where the first three years of ProRep necessary to the success of the cluster approach? Key Evaluation Question 1: • How do you see the current quality of representation and policymaking process in Indonesia? • In what ways is the policy cluster approach as implemented through ProRep bringing about better informed and more representative legislative and policymaking processes? Please provide specific examples. • In what ways is the policy cluster approach as implemented through ProRep promoting the passage of policy improvements or better policy implementation? Please provide specific examples. • To what extent are the initiatives / approaches relevant to the country context in terms of those mentioned objectives? • To what extent are the initiatives undertaken through the cluster approach both relevant to and a priority of the grantee that participated in year 4 and/or 5? • What do you think about how the partner (CSOs) were identified? • What is the strength and weakness of the overall program activities? • What are the advantages and disadvantages of the approach? Key Evaluation Question 2: • What have been the main challenges that might have affected achievement of program objectives? • What are the supporting factors regarding Indonesian legal, political, or social frameworks to the program and its objectives? • Are there any constraining factors regarding Indonesian legal, political, or social framework in relation to the program and its objectives? • Are there other contributing or constraining factors you have identified? If there are some constraints, please describe (e.g. program design challenges, implementation challenges, or other). • What could be done to increase the effectiveness of the clusters in achieving each of these objectives? • What are the difficulties of working with Indonesian policymakers? • What difficulties did you find in identifying and building partnership with implementing agencies (NGOs/think tanks)? • How did you deal with different characteristics of the national and local legislative representation and policy making? Key Evaluation Question 3: • What are the most significant changes in policy making process at the national level that you observed? At the local level that you observed? o What differences do you see in legislative representation or policymaking at the national level since the program began? 47 o What differences do you see in legislative representation and policymaking at the local level since the program began?  What (or whom) do you attribute these changes to? What aspects of the ProRep program promoted the gains you just described at the national and local level?  What key practices have the policy clusters implemented/utilized to make the gains at the national and/or local level that you just described? Key Evaluation Question 4: • How does the policy cluster in this sector [education, health, or environment) look different than in the other sectors? • What explains these differences? • What are the strengths and weaknesses of how this policy cluster is operationalized? • What changes could be made to promote more policy gains in the future? • In your opinion, which cluster has been the most “successful” in the last two years, and why? How do you define “success”? Key Evaluation Question 5: • How many grantees have been involved in this policy cluster in Year 4 and 5? Please provide details. • In your opinion, did the use of grants and other mechanisms allow for flexible and effective relationships between CSOs, NGOs, etc and Chemonics? • What were the most effective and impactful activities implemented by ProRep (mentoring, training, etc…)? Why? • Can you explain the reporting processes for grants (programmatic and financial), detailing ProRep’s support provided to the grantee to reach each step in the process? What worked and what didn’t work in this process? Why? o What were the biggest challenges ProRep faced in managing grantees/partners? o What worked well in the way ProRep managed partners? Key Evaluation Question 6: • How did ProRep support sustainability measures within each policy cluster? • Are there exit strategies identified and proposed by ProRep to grantees that promote sustainability? Explain. • In your opinion, were these measures accepted by grantees, or were grantees resistant to continuing in the policy cluster? Why? • Please describe the challenges to sustainability you see for this cluster [education, health, or environment]? • Apart from USAID/ProRep’s support of program sustainability, what are your strategies to make sure that program advantages for beneficiaries are sustained? Other:  Please describe the programs gender inclusion efforts. How do project activities ensure participation by both men and women?  What could be some the key lessons drawn from the program approach in this cluster?  Do you have any other comments, suggestions or concerns about the policy cluster you work with that you would like to share with us? 48 ProRep Final Evaluation (Est. Time ~45 min) KII, CSO/Think-tank • Describe your role in your organization. How long have you been in your position? • Please describe your organization’s vision and mission. • Could you explain the involvement of your organization with ProRep (type of engagement, length of time, grant amount, activities conducted)? Key Evaluation Question 1: • How do you see the quality of representation in the policymaking process in Indonesia? Does civil society have a voice/a seat at the table when policies are designed and passed? • Please explain your engagement with the [education, health, environment] policy cluster, as promoted by ProRep. • In your own words, what is the objective or goal of the policy cluster you are engaged with? • What progress has this cluster made toward the achievement of those goals since you become involved? What policies does the cluster that you are engaged with work on? • How were these policies identified/selected? Are they in-line with your organizations mission and vision? Are these policies priorities in Indonesia currently? • To what extent are the initiatives undertaken through ProRep relevant to your organization’s overall goals, strategies and programming priorities in achieving those objectives • What is the importance of promoting better information and inclusion of public participation in legislative representation and policymaking processes? • How does your policy cluster engage policy makers? Please explain. • Were there other achievements of this policy cluster that you want to mention? Key Evaluation Question 2:  In your work in the cluster, what were the challenges you faced as an organization? How did you deal with these difficulties?  In your work with the cluster, what were the challenges the policy cluster participants faced while promoting better policy? How did the cluster/group deal with these difficulties? o What are the constraining factors regarding Indonesia’s legal framework in relation to the cluster approach? o What are the difficulties of working with Indonesian policymakers at national level? o What are the difficulties of working with Indonesian policymakers at local level  What did you like about how the cluster operated? What did you not like? What would you change to improve the policy cluster/community?  Regarding the successes of the cluster that you mentioned above, what led to this success? What hampered the success?  How did you deal with sensitive issues e.g. in the environment cluster which may involve various actors (i.e. national and multinational companies, local business man, local and national politicians)? Key Evaluation Question 3: • What are the most significant changes in policy making process, at the national and/or level in the cluster you are engaged with? • Have any practices have been introduced to the policy making process in this sector as result of the program/cluster approach? 49 • How did your cluster/organization build partnerships with policymakers at national and local level? • Is this approach different than how you engaged with policy makers in previous initiatives/programs? • What aspects made this approach more or less effective? At the national/local level? Key Evaluation Question 5: • Do you think that the grant you received from ProRep was sufficient for the activities you were asked to complete/impact you were asked to have? • Were there any items in the contract that burden the completion of the program? • What were the most beneficial institutional support activities received by your organization from ProRep? • What type of support did you want more of? Key Evaluation Question 6:  Do you think the work you have completed with the policy cluster is sustainable? Why or why not?  Will your organization continue working on this issue/in this sector? Will your organization continue to coordinate with the policy cluster? Why or why not?  What strategies for sustainability have you learned from ProRep?  In your opinion, what is the most crucial step in ensuring sustainability of your organizations work in this sector? Of the gains made in policy through the cluster approach? Apart from USAID/ProRep’s support of program sustainability, what are your strategies to make sure gains/achievements are sustained? Other:  Please describe the programs gender inclusion efforts. How do project activities ensure participation by both men and women?  What could be some the key lessons drawn from the program approach in this cluster?  Do you have any other comments, suggestions or concerns about the policy cluster you work with that you would like to share with us? 50 ProRep Final Evaluation (Est. Time ~45 min) KII, Policy maker  What is your title/position? How long have you been in this position?  Could you explain your involvement with this initiative (ProRep, CSOs, policy cluster)? Key Evaluation Question 1: • How do see the quality of representation in the policymaking process in Indonesia? Do you frequently observe civil society participating in policy making? How? • Do you think that greater participation by civil society in policy making promotes better policy/better policy implementation? Why or why not? • What is the importance of promoting better information and inclusion of public participation in legislative representation and policymaking processes? • In your understanding, what is the goal of the policy cluster? • Are the issues the ProRep-supported policy cluster is working on relevant to the context in Indonesia/in this district right now? o What are the key issues in [education, health, environment] in this district/region/country? • What policy gains were made in the last two years that you think resulted from work the policy cluster/ProRep completed? Key Evaluation Question 2:  What contributed to the policy gains you just described?  If no policy gains were made as a result of the cluster, why do you think that was?  In your opinion, how could the clusters operate differently to increase their effectiveness? Key Evaluation Question 6:  Did you gain new knowledge about legislative representation and policymaking after participating in ProRep events/policy cluster? How will this influence your future work in this position?  Have you developed new relationships with civil society groups/individuals that you will maintain?  Are the policy clusters sustainable? Why or why not? Will you remain engaged with the cluster/community? Other:  Please describe the programs gender inclusion efforts. How do project activities ensure participation by both men and women?  What could be some the key lessons drawn from the program approach in this cluster?  Do you have any other comments, suggestions or concerns about the policy cluster you work with that you would like to share with us? 51 Focus Group Discussion Protocols ProRep Evaluation Coversheet for All Focus Group Discussions Date of Interview: Interviewee Name, Title: Interviewer(s): Time Start: Location: Time End: Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our group interview request, we are working to conduct a final evaluation of the ProRep Project. The evaluation is intended to provide an informed evaluation of the policy cluster approach to generate key lessons learned and identify promising practices for replication in future USAID programs. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to discuss with you all today to hear about your experience, in your own words, in order to help us better understand how this project looks and functions “on the ground.” The discussion will last around 60 minutes. You can decide to stop the interview at any time. We will take notes of the group discussion and, with your consent, a recording will also be made using a digital voice recorder to aid us in fully completing our notes after the discussion. Confidentiality Protocol • We will collect information on individuals’ names, organizations and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purpose of this evaluation, and will not be shared with other audiences or used for any other purpose. 52 • Your participation in this group discussion is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? Inform participants we may follow-up with brief email survey at the end of fieldwork. ADDITIONAL GUIDELINES FOR FGD GUIDES Be sure to: • Set up the room to facilitate a participatory discussion • Introduce yourselves and also the note taker • Introduce SI and explain why you are conducting the focus group session • Ask the participants to do the same • Give a verbal agenda and length of the meeting • Set goals • Clarify your role as a facilitator FOCUS GROUP FACILITATOR’S GUIDELINES FOR EFFECTIVE DISCUSSIONS: • Everyone is clear on the topic • Everyone participates; no one dominates the discussion - No speeches! • One person talks at a time • Comments and discussion stay on the topic • Comments should be to the whole group - no side discussions • Respect time limits • Write down unanswered questions • No divisive or confrontational language or tone • Ask participant’s consent to use the digital audio recorder • Note taker should write the dynamic of the discussion and write down the question and key answers 1. Introduction (10 min) See above  Reaffirm points of the meeting  Welcome participants 53  Set 60- 90 minute timeline  Introduce the evaluation, give verbal agenda, objectives (as stated above) 2. Setting rules (5 min) Before the participants begin dealing with issues and ideas, the participants should agree on a set of rules that define how a group will function and how the participants will interact. Sample rules • Each group member has the right to participate. • The opinion of each group member is important and should be respected. • Group members should be tolerant of different ideas. • Each group member is important. 3. Questions (45 minutes) KII, CSO/Think-tank • Describe your role in your organization. How long have you been in your position? • Please describe your organization’s vision and mission. • Could you explain the involvement of your organization with ProRep (type of engagement, length of time, grant amount, activities conducted)? Key Evaluation Question 1: • How do you see the quality of representation in the policymaking process in Indonesia? Does civil society have a voice/a seat at the table when policies are designed and passed? • Please explain your engagement with the [education, health, environment] policy cluster, as promoted by ProRep. • In your own words, what is the objective or goal of the policy cluster you are engaged with? • What progress has this cluster made toward the achievement of those goals since you become involved? What policies does the cluster that you are engaged with work on? • How were these policies identified/selected? Are they in-line with your organizations mission and vision? Are these policies priorities in Indonesia currently? • To what extent are the initiatives undertaken through ProRep relevant to your organization’s overall goals, strategies and programming priorities in achieving those objectives • What is the importance of promoting better information and inclusion of public participation in legislative representation and policymaking processes? • How does your policy cluster engage policy makers? Please explain. • Were there other achievements of this policy cluster that you want to mention? Key Evaluation Question 2:  In your work in the cluster, what were the challenges you faced as an organization? How did you deal with these difficulties?  In your work with the cluster, what were the challenges the policy cluster participants faced while promoting better policy? How did the cluster/group deal with these difficulties? o What are the constraining factors regarding Indonesia’s legal framework in relation to the cluster approach? o What are the difficulties of working with Indonesian policymakers at national level? o What are the difficulties of working with Indonesian policymakers at local level 54  What did you like about how the cluster operated? What did you not like? What would you change to improve the policy cluster/community?  Regarding the successes of the cluster that you mentioned above, what led to this success? What hampered the success?  How did you deal with sensitive issues e.g. in the environment cluster which may involve various actors (i.e. national and multinational companies, local business man, local and national politicians)? Key Evaluation Question 3: • What are the most significant changes in policy making process, at the national and/or level in the cluster you are engaged with? • Have any practices have been introduced to the policy making process in this sector as result of the program/cluster approach? • How did your cluster/organization build partnerships with policymakers at national and local level? • Is this approach different than how you engaged with policy makers in previous initiatives/programs? • What aspects made this approach more or less effective? At the national/local level? Key Evaluation Question 5: • Do you think that the grant you received from ProRep was sufficient for the activities you were asked to complete/impact you were asked to have? • Were there any items in the contract that burden the completion of the program? • What were the most beneficial institutional support activities received by your organization from ProRep? • What type of support did you want more of? Key Evaluation Question 6:  Do you think the work you have completed with the policy cluster is sustainable? Why or why not?  Will your organization continue working on this issue/in this sector? Will your organization continue to coordinate with the policy cluster? Why or why not?  What strategies for sustainability have you learned from ProRep?  In your opinion, what is the most crucial step in ensuring sustainability of your organizations work in this sector? Of the gains made in policy through the cluster approach? Apart from USAID/ProRep’s support of program sustainability, what are your strategies to make sure gains/achievements are sustained? Other:  Please describe the programs gender inclusion efforts. How do project activities ensure participation by both men and women?  What could be some the key lessons drawn from the program approach in this cluster?  Do you have any other comments, suggestions or concerns about the policy cluster you work with that you would like to share with us? 55 ANNEX III: EVALUATION SOURCES OF INFORMATION Documents Reviewed The following are the documents reviewed for this evaluation. • ProRep Final Approved Project Management Plan (PMP) • Annual Program Statement (APS) NO.02 • Indonesia CDCS Final Version • ProRep 4th Annual Report, 2014 • ProRep 5th Annual Report, 2015 • ProRep Quarterly Report (Oct-Dec,2014) • List of Targeted Policy • ProRep Impact Statement, September 2015 • USAID & Chemonics ProRep Meeting January 17, 2014 • Political Economy of Policy Making in Indonesia, Overseas Development Institute and SMERU Reseach Institute, 2011 • ProRep Accomplishment FY 2013-2014 (October 2013-September 2014) Q1 and Q2 FY 2014- 2015 (October 2014-March 2015) • ProRep Environment Success Story, 2015 • ProRep: Mid-Term Evaluation Report • Prorep Newsletter 2014 • ProRep Newsletter 2015 Respondents by Institution Key Informant Interview (KII) or Focus Group Discussion (FGD) Location Position Institution KII Jakarta Chief of Party ProRep/Chemonics KII Jakarta Former Chief of Party ProRep/Chemonics KII Jakarta Deputy Chief of Party ProRep/Chemonics KII Jakarta Public Policy Specialist/Team Leader 1 ProRep/Chemonics KII Jakarta Civil Society Specialist/Team Leader 2 ProRep/Chemonics KII Jakarta Operations Director ProRep/Chemonics KII Jakarta Public Policy Advisor ProRep/Chemonics KII Jakarta M&E Advisor ProRep/Social Impact KII Jakarta ProRep Staff ProRep/Chemonics KII Jakarta Deputy Director of Environment USAID KII Jakarta MCH/Health System Strengthening Specialist USAID KII Jakarta COR for ProRep USAID KII Jakarta Deputy Director of DRG Office USAID 56 KII Jakarta Deputy Director of Education Office USAID KII Washington USAID FGD Surakarta Head of Legislation Unit Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Commission 4 Municipality Parliament/DPRD FGD Surakarta Director YSKK FGD Surakarta Staff YSKK FGD Surakarta Staff YSKK FGD Surakarta Staff YSKK FGD Surakarta Staff YSKK FGD Surakarta Staff YSKK KII Surakarta Head of Education Office Education Office Surakarta KII Surakarta Education Office Staff Education Office Surakarta FGD Surakarta Director Pattiro Surakarta FGD Surakarta Program Manager Pattiro Surakarta FGD Surakarta Staff Pattiro Surakarta FGD Surakarta Staff Pattiro Surakarta FGD Surakarta Staff Pattiro Surakarta FGD Surakarta Staff Pattiro Surakarta FGD Surakarta Coordinator MP3S FGD Surakarta Member MP3S FGD Surakarta Secretary MP3S FGD Surakarta Member MP3S FGD Surakarta Member MP3S KII Surakarta Head of Information Department Municipality Health Office, Surakarta 57 KII Jeneponto (Phone) Director Pattiro Jeneponto KII Bengkulu Head of Development, Provincial Forestry Office Provincial Forestry Office, Bengkulu KII Bengkulu Staff District Planning Agecy, Sungai Ketaun KII Bengkulu Head of Resolution Management Land Use Office, Bengkulu KII Bengkulu Director AKAR Bengkulu KII Sulawesi Tengah (Phone) Formar Program Manager Sulawesi Institute KII (Phone) Executive Director LepMIL FGD Jakarta KMSTP member YAPPIKA FGD Jakarta KMSTP member ICW FGD Jakarta KMSTP member ICW FGD Jakarta KMSTP member Expert FGD Jakarta KMSTP member Coalition of Teacher, Banten FGD Jakarta KMSTP member Coalition of Teacher, Banten KII Jakarta Executive Director RMI KII Jakarta Advocacy Coordinator RMI KII Bandung (Phone) Board F2H KII Bandung (Phone) Program Leader F2H KII Jakarta Director of Social Forestry Ministry of Environment and Forestry KII Jakarta Executive Director Epistema FGD Jakarta Researcher Paramadina Pubic Policy Institute FGD Jakarta Researcher LPEM-UI FGD Jakarta Researcher LPEM UI KII Jakarta Chief of Party Prioritas-USAID KII Jakarta Governance Specialist Prioritas-USAID KII Jakarta Basic Education Specialist Prioritas-USAID KII Jakarta Chief of Party Kinerja-USAID FGD Jakarta Staff Article 33 58 FGD Jakarta Staf Article 33 FGD Jakarta Staff Article 33 FGD Jakarta Staff Article 33 KII Jakarta Director New Indonesia KII Jakarta Researcher Women Research Institute KII Jakarta Researcher Women Research Institute KII Jakarta Senior Program Manager Kemitraan KII Jakarta Staff Kemitraan KII Jakarta Chief of Party KSI KII Jakarta Staff United Nations Development Programme Respondent Data Disaggregated by Gender Male Female Total KIIs 26 16 42 FGDs 28 13 41 Total 54 29 83 59 ANNEX IV. DATA COLLECTION SCHEDULE Evaluation Process Schedule Weeks Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13 December Week of 1/4 1/11 1/18 1/25 2/1 2/8 2/15 2/22 2/29 3/7 3/14 3/21 3/28 Kick Off Meeting with USAID/Indonesia Consultant on-boarding Document Review Team Planning Meeting (Webex) Development of data collection tools and evaluation work plan (EWP) Logistics (with ProRep assistance) Submit Draft EWP + tools 15 Travel to Indonesia (TL) Team Meeting in Jakarta 19 In--brief with USAID/DG and Chemonics 20 Submit Final EWP 21 Data Collection Jakarta Data Collection outside Jakarta Data Analysis and report writing Out-brief with USAID; Presentation to implementers 9 Travel from Indonesia (TL) 10 Data Analysis and report writing Draft Report Submission 3 10 working days for USAID review Revise/Finalize Evaluation Report and submit all records from the evaluation (10 working days) 25 Key Completion of tasks Fieldwork Red outline Submission of Deliverable x Performance Evaluation of Program Representasi: January 4 - March 25, 2016 January Analysis and Reporting February March Field Work Start-Up and Coordination 60 *Note that the above schedule was adjusted, as the draft report on the 4th of March and the final report was submitted on the 1st of April. Fieldwork Schedule January 20 – February 5, 2016 EVENT PURPOSE LOCATION January 21, 2016 Interview with ProRep staff: Team Leader 1 KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Interview with ProRep staff: Policy Advisor KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Interview with ProRep staff Deputy COP KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Interview with ProRep staff Team Leader 2 KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta January 22, 2016 Interview with ProRep staff Operation Manager KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Interview with USAID: Deputy Director of Environment KII Discuss Environment clsuter Jakarta Interview with ProRep: Former COP KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Interview with ProRep: COP KII Discuss ProRep Year 4 and 5 progress, per cluster Jakarta Skype interview with USAID Washington Discuss policy cluster approach Skype January 25, 2016 Interview with Watershed Management Office Bengkulu KII: Discuss environment sector Bengkulu Interview with Local Planning Agency(Bappeda) Bengkulu KII: Discuss environment sector Bengkulu Interview with AKAR Foundation KII: Discuss environment cluster Bengkulu FGD with Local Parliament FGD: Discuss education cluster Surakarta FGD with YSKK FGD: Discuss education cluster Surakarta Phone Interview with Executive Director Pattiro Jenenponto KII: Discuss Health Cluster Surakarta (Phone) January 26,2016 KII with Head of Education Office KII: Discuss education cluster Surakarta KII with staff of Education Office KII: Discuss education cluster Surakarta Phone interview with Former Governor of Bengkulu KII:Discuss environment cluster Bengkulu (Phone) January 27, 2016 FGD with Pattiro staff FGD: Discuss health cluster Surakarta FGD with MPSS FGD: Discuss health cluster Surakarta KII with Staff of Health Office KII: Discuss health cluster Surakarta Phone Interview with YIPD KII: Discuss environment cluster Jakarta (Phone) Phone interview with LepMil KII: Discuss environment cluster Jakarta (Phone) Phone Interview with Sulawesi Institute KII: Discuss environment cluster Jakarta (Phone) January 28, 2016 61 KII with Staff of Health Office KII: Discuss health cluster Surakarta January 29, 2016 FGD with KMSTP FGD: Discuss education cluster Jakarta KII with ProRep staff: M&E Advisor KII: Discuss Policy Cluster progress Jakarta February 1, 2016 KII with RMI KII: Discuss environment cluster Jakarta KII with F2H Bandung KII: Discuss health cluster Jakarta (Skype) KII with F2H Bandung KII: Discuss health cluster Jakarta (Skype) KII with Ministry of Environment and Forestry KII: Discuss environment cluster and sector in general Jakarta KII with Epistema KII: Discuss environment cluster Jakarta February 2, 2016 FGD with PRN FGD: Discuss PRN Jakarta KII with USAID: Deputy Director of DRG KII: Discuss policy cluster Jakarta KII with UDAID: Deputy Director Education Office KII: Discuss education cluster Jakarta KII with USAID Health Office KII: Discuss health cluster Jakarta KII with USAID DRG Office (COR) KII: Discuss policy cluster approach Jakarta February 3, 2016 KII with Prioritas COP KII: Discuss education cluster Jakarta KII with Prioritas advisor KII: Discuss education cluster Jakarta KII with Prioritas advisor KII: Discuss education cluster Jakarta February 4, 2016 KII with ProRep staff KII: Discuss health cluster Jakarta KII with Article 33 Director KII: Discuss environment cluster Jakarta KII with Article 33 (Staff) KII: Discuss environment cluster Jakarta KII with Article 33 (Staff) KII: Discuss environment cluster Jakarta KII with Article 33 (Staff) KII: Discuss environment cluster Jakarta KII with The New Indonesia KII: Discuss education cluster Jakarta February 5, 2016 KII with WRI (Researcher) KII: Discuss health cluster Jakarta KII with WRI (Researcher) KII: Discuss health cluster Jakarta KII with KSI COP KII: Discuss policy cluster in general Jakarta KII with HR Kemitraan (Phone) KII: Discuss environment cluster Jakarta (Phone) KII with HR Kemitraan (Phone) KII: Discuss environment cluster Jakarta (Phone) 62 ANNEX V: LIST OF GRANTEES ProRep Grantees Year 2014 Grantee Issue Original/Modified Grant Value IDR USD WRI-Women Research Institute Health 316,7555,000 28,416.17 F2H-Frontiers For Health Health 435,065,000 39,029.78 PATTIRO-Pusat Telaah dan Informasi Regional (Center for Regional Studies and Information) Health 584,5444,700 52,439,64 LPEM-FEUI – Institute for Economic and Social Research, University of Indonesia Environment 223,900,000 20,086.12 Perkumpulan Article 33 Indonesia Environment 360,175,000 32,311.38 ICEL-Indonesia Center for Environmental Law Environment 377,605,000 33,426.48 Epistema Insitute Environment 518,825,000 46,274.78 YSKK-Yayasan Satu Karsa Karya (Foundation of Integrated Efforts) Education 1,128,220.000 101,212.88 PPPI- Paramadina Public Policy Institute Education 404,150,000 36,256.39 ICW – Indonesia Corruption Watch Education 428,5000,000 38,440.84 Perkumpulan Article 33 Indonesia Education 849,639,321 76,221.34 LPEM-FEUI – Institute for Economic and Social Research, University of Indonesia PRN 469,315.000 42,102.36 IRE – Institute for Research and Empowerment PRN 388,965,000 34,894.14 ProRep Grantees Year 2015 Grantee Issue Original/Modified Grant Value IDR USD YSKK-Yayasan Satu Karsa Karya (Foundation of Integrated Efforts) Education 771,840,000 58,798 ICW – Indonesia Corruption Watch Education 811,575,000 61,825 ICW – Indonesia Corruption Education 459,100,000 34,974 63 Watch PPPI- Paramadina Public Policy Institute Education 543,696,140 41,208 Epistema Institute Environment 639,150,000 48,690 Sulawesi Institute Environment 448,220,000 34,145 AKAR Environment 402,220,000 31,163 RMI-Rimbawan Muda Indonesia Environment 522,550,000 38,872 IBC Environment 398, 862,500 30,385 LePMIL Environment 394,910,000 30,084 ICEL-Indonesia Center for Environmental Law Environment 389,550,000 29,675 YKMI Environment 361,880,000 24,976 Perkumpulan Article 33 Indonesia Environment 239,850,000 16,557 HuMa Environment 362,937,500 25,054 PATTIRO-Pusat Telaah dan Informasi Regional (Center for Regional Studies and Information) Health 1,145,400,000 88,605 F2H-Frontiers For Health Health 484,112,500 36,879 Prakarsa Health 548,375,000 42,421 64 ANNEX VI. GRANT MECHANISMS The following represent grant mechanisms utilized during the implementation of the ProRep Project. Mechanism Description Use on ProRep Standard Grants Cost reimbursable grants. Monthly payment are made based on costs incurred and verified by ProRep with receipts and other back up Primary grant mechanism used for early CSOs and research grants (2011- 2013) Fixed Amount Awards (FAA)- formerly known as Fixed Obligation Grants (FOG) Grants outputs are divided into several milestones. A budget is agreed upon at the outset and payments are made upon successful completion and milestones. Primary grant mechanism from 2013-2016 In-Kind Costs incurred directly by ProRep for activities under a grant. May include equipment or activities managed by ProRep In-Kind elements incorporated into grants throughout Letters of Collaboration (LoC) A non-legally-binding agreement between ProRep and a partner establishing specific inputs provided by specific inputs provided by each party towards an activity. Used for the following: • Formalizing ProRep support to partner-led activities • Priority activities of opportunity that require rapid implementation • Involving potential future cluster partners prior to awarding a formal grant. Developed and used since 2014 Direct activities ProRep-organized and funded events and activities Used throughout ProRep. Often trainings, hiring of expert consultants, hosting consolidation meetings and communication support. 65 ANNEX VII: LIST OF REGULATIONS ENACTED The following achievements are documented in the ProRep PMP under indicators POa and POb. They are listed here according to national and local level. These represent the achievements as of January 2016. Achievements documented after this date are not included in this list.15 National Level 1. Presidential Decree on Ministry of Environment and Forestry Structure. ICEL-Environment cluster 2. Priority of Social Forestry in National Mid-Term Development Plan (RPJMN) 2015-2019 Epistema Institute – Environment cluster Local Level 1. Head of District Health Office Semarang District Decree (Surat Keputusan/SK) on the use of SMS Gateway as the Complaint Handling Mechanism in Semarang District Pattiro – Health Cluster 2. Head of Municipality Health Office Surakarta Municipality (Surat Keputusan/SK) on the use of SMS Gateway and formalization of Multi-stakeholder forum in the Complaint Handling Mechanism in Surakarta District – Health Cluster 3. Head of District Decree (Surat Keputusan/SK) on the use of SMS Gateway as the Complaint Handling Mechanism in Jeneponto District. – Health Cluster 4. Local Regulation Bombana District on Indigenous People of Moronene Hukaeya Lae. Sulawesi Institute – Environment Cluster 5. Local Legislation (Perda) on Recognition of Kasepuhan Indigenous Community in Lebak District Epsitema Institute- Environment Cluster 6. Regent Decree (SK Bupati) on recognition of to Kaili and to Kulawi Indigenous Communities in Sigi District. Epistema Institute – Environment Cluster 7. Governor Decree (SK Gubernur) on IUPHKm (Community Forestry Utilization Permit) for 8 Gapoktan in Rejang Lebong District AKAR Bengkulu – Environment Cluster 8. Governor Decree on Forest and Land Related Conflict Resolution in Bengkulu Province AKAR Bengkulu – Environment Cluster 15 See the ProRep Project Final Report for a complete list of all policy/regulatory achievements as achieved through April 2016. 66 ANNEX VIII. POLICY CLUSTERS The graphics below were created by ProRep in Year four and five and included in the ProRep Project Newsletter (2014 and 2015). Health Policy Cluster 2014 67 Environment Policy Cluster, Year 2014 68 Health Policy Cluster 2015 69 Education Policy Cluster 2015 70 Environment Policy Cluster 2015 71 ANNEX IX. EVALUATION MATRIX Evaluation Question Preliminary/Interim Findings Preliminary Analysis and Conclusions Preliminary Recommendations Q1. In what ways is the policy cluster approach, as implemented through ProRep, affecting the following objectives: 1) bringing about better informed and more representative legislative and policy making processes; 2) promoting the passage of policy improvements or better policy implementation  Though new to many ProRep Cluster Members, the majority of respondents reported that focusing on elements and needs with regard to “rulemaking” helped them be more effective at collaborating with and enhancing local government capacity. This also increased demand by government officials, particularly at the local level, for evidenced-based research that advanced key local priorities.  The ProRep cluster members agreed that the cluster approach isolated and advanced common agendas (even advocacy goals that were not immediate to their organizations’ mission) among cluster members and brought additional (non-ProRep) members from their respective communities into the political/governmental rule-making process beyond those they had engaged with in the past.  Cluster members believed that more interdependent (national and local level) relationships among stakeholders had been created through ProRep which advanced CSOs  ProRep was successful at creating many functional and complementary relationships among cluster members to advance targeted local level policy implementation objectives which in turn supported policy reforms at the national level.  The policy cluster approach increased CSO and Think Tank tactical skills in interacting with government to influence policy objectives.  Synchronizing cluster members’ capacity for community outreach and research at national and local levels was a key inducement for government officials to participate in cluster activities.  The Cluster Approach served as both a means i.e. a set of coordinated actions and an end i.e. formalized working groups that energized citizen participation.  While the ProRep’s Cluster approach was cost effective, more may have been  Selection of policy objectives and strategic action plans by cluster members should be guided by or through an internal policy negotiation/mediation process to ensure that critical needs and objectives are met.  Additional technical training should be provided to Cluster Members on better understanding and analyzing operating constraints and opportunities of respective rule-making bodies.  On-going assessment of the political environment at local levels, local level CSO engagement with their respective communities, and the alignment of practical and logistical requirements of policymakers will help ensure that policy objectives are in the manageable interest of all parties. 72 and Think Tanks’ missions, while positioning them as more equal partners to government.  The Cluster Members reported that ProRep’s reorientation of their organizational and communications skills helped them “critically engage” with the government on issues they believed were achievable.  ProRep Cluster members agreed that Indonesia is experiencing new momentum in government efforts to institutionalize transparency and accountability in government operations, and this has given them more incentives to work with government. Moreover, the Cluster’s national/Local-level cooperation model was an effective tool that reinforces decentralized governance. Two decades of decentralization has impacted how policy-making and policy implementation are carried out in Indonesia.  ProRep’s Cluster approach included 30 grants (averaging $60,000) among the three clusters that contributed to 10 enacted policy objectives (2 at the national level and 8 at the local level)  A great deal of time was invested by ProRep in synchronizing schedules of government rulemaking protocols e.g. accomplished had they had better understanding of the rulemaking processes. 73 ministerial meetings or legislative committee hearings schedules (at the national or local level) with CSO/Think Tank work plans. Q2. According to stakeholders, what are contributing and inhibiting factors toward the achievement of these objectives? What could be done to increase the effectiveness of the clusters in achieving each of these objectives? Contributing factors  Coordinated national-local plans among CSOs magnified the operational capacity of individual organizations to achieve policy goals.  The recent elections seemed to have deepened government commitment to accountability reform  The limited research capacity of subnational units of government provided an entry point for cluster.  Time-sensitive policy objectives brought more discipline to Cluster members  Development of negotiated policy objectives among stakeholders was a key mobilizing element for success.  Flexible and deliverable-oriented donor support were important for CSO bottom lines.  Increased fluency of CSOs on rulemaking processes e.g. how policy considerations are introduced to responsible committees, budget and legal requirements, oversight and comments by other government officials, etc were important new skills  As government continues to support more transparency and accountability in its operations and local governments exercise more authority the cluster approach may gain more utility.  Rulemaking calendars drove Cluster members’ actions plans; intense and strategic forward planning was key to achieving milestones.  The cluster approach seems to have engendered additional concentric circles of participation among CSO cluster members that required additional organizational support.  When policy objectives and well￾coordinated strategies were not shared by all Cluster members, impact was diffused.  Increase the use of LOCs to provide added institutional support to Cluster members to accommodate complexity and to manage changing circumstances.  The clusters suggest greater attention to managing risk and going with the “grain” of existing institutions. 74 Inhibiting Factors  A main concern for CSO and Think Tank cluster members involved the ability to synchronize their primary mission with negotiated policy objectives  CSOs and think tank members were also concerned if their annual work plans did not allow enough time to work on targeted policy objectives, particularly when there were competing donor priorities or ideological differences among cluster members. Q3. In what ways did the policy clusters under ProRep Influence national vs local policy? What are the differences as similarities in how clusters operate at these levels?  Rather than promoting the development of clusters at different levels of government, ProRep linked cluster members at national and local levels to achieve complementary policy objectives.  Whether Clusters were organized at national or local levels, they agreed to core elements that required 1:) All cluster would have government, Think Tank and CSOs working together; 2) to develop negotiated policy objectives; 3) develop strategic national-local action plans directed at reinforcing strategic policy objectives ; and 4)to  The cluster approach was effective at identifying achievable policy objectives (particularly at the local level) and utilizing the respective strengths of cluster members at both the national and local level.  The policy cluster approach built confidence among members and responded well to the emerging patterns of policy implementation at the local level and policymaking at the national level, by working through a National/Local organizations that  Standardize training and continuous monitoring of rulemaking procedures and protocols at national and subnational for cluster members.  Policy Clusters should be better linked to their counterparts at both national and local levels to ensure that a critical path in policy making/reform at the national level and policy implementation at the local level are not missed. 75 provide evidenced based recommendations that responded to policymakers needs.  ProRep’s focus on tactical support to government rulemaking processes was instrumental in assisting cluster members select national level policy reforms that would be supported policy implementation efforts by their local level affiliates  Use of more government-friendly policy briefs and continuous feedback among Cluster Members were vital to ensuring momentum on policy policy objectives among Cluster Members empowered, local level policy implementation while building momentem for national level policy reform. Q4. How has the policy cluster/community approach been operationalized differently (or similarly) in the sectors of environment, health, and education through ProRep? Have different methods of operationalization affected achievement of cluster objectives?  ProRep’s experience forming and utilizing the cluster approach in the first sector cluster, Environment, influenced succeeding Education and Health cluster formation. For example, the Education cluster more readily formed a new formal identity called KMSTP which now has over 40 members.  For some clusters, International funding disparities/ interest (Environment CSOs seemed to have more interest among donors) created different incentives for cluster memberships, resulting in less formal  Fundamentally, there were no significant structural differences among sectors that would require different modalities for operationalizing  Producing manageable common policy objectives may be easier to achieve when the public service delivery by government affects more directly broader or more immediate needs of citizens (e.g. in education and health).  Willingness among cluster members to negotiate policy objectives and the relative operational reach ( e.g. country 76 relationships among environment CSOs at the national level.  Additionally, members of the environment cluster noted the level of interest by significant actors such as private sector interest were more pronounced and fractious.  The Health and Education clusters CSOs have more formal relationships at the national level. membership of beneficiaries) of CSOs were key to successful cluster functioning. Q5. How did ProRep’s management of partnerships (through grants, letters of collaboration, and other mechanisms) contribute to or inhibit the achievement of policy cluster objectives?  The cluster approach, particularly at the beginning was seen by many cluster members as cumbersome, requiring them to participate in multiple meetings to set policy agendas and orient themselves to ProRep’s cluster approach.  FOG and LOCs were instrumental in driving the pace of work (grantees had to become “Master’s of Milestones”).  ProReps pre-award audit, that identified organizational weak points in grantees, was an important tool utilized during project activities to further support grantee performance.  ProRep identified appropriate CSOs committed to exploring new approaches to utilizing political/governance processes.  The FOG and LOCs are appropriate mechanism for supporting grantees that permit more room for achieving milestones that are concurrent to time-sensitive rulemaking process.  Institutional strengthening support of cluster members was required to maintain full compliance of grant activities  Continue the utilization of FAAs that provide cluster members more time and resources to focus on deliverable while alleviating some of the administrative burdens and increase use of LOCs (for timely tactical grantee assistance).  Further development of a grants management database system beyond M&E functions would enhance overall program operations. Q6. What aspects of sustainability supported or promoted by ProRep are present in the policy clusters? What are the challenges to sustainability for each cluster?  Despite hard ProRep deadlines and exact deliverables, many CSOs and Think Tanks interviewed have internalized grant operating  ProRep’s heuristic approach and to cluster formation have resulted in multiple examples of formal, semi-formal and non￾binding cooperation among  Encourage the formation of cluster secretariats as legal entities to replace ProRep as lead convener and provider of technical support to clusters. 77 procedures into their own organizations  The cluster approach which emphasized “achievable and pragmatic” policy objectives have been confidence boosters for cluster members.  Cluster members at the local level reported that they are receiving request for additional cluster-like technical assistance from their respective government officials.  Several ProRep cluster members reported that the policy objectives achieved through the cluster approach has engendered follow-up implementation responsibilities/activities for cluster CSOs that has helped their standing in the communities they serve.  Environmental Cluster members were keenly aware of increasing international donor interest in supporting the sector in Indonesia sector players. The durability of each model has not been determined given that all are still in the very early stages.  The cost effectiveness of the Cluster approach may be attractive to donors  As secretariats are formed, protocols should be established to ensure that the secretariat does not become a competitor to its members  Encourage Cluster Joint solicitation of new funding by cluster members, particularly in the environment sector. 78 ANNEX X. CONFLICT OF INTEREST Name Patrick J. Fn’Piere Title Team Leader Organization Social Impact Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (or RFTOP or other appropriate instrument number) AID-497-C-11-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ProRep, Chemonics I have real or potential conflict of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 1-5-2016 79 Name Ashari Cahyo Edi Title Policy Expert Team Member Organization Social Impact Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (or RFTOP or other appropriate instrument number) AID-497-C-11-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ProRep, Chemonics I have real or potential conflict of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I have previous work experience with Institute for Research and Empowerment (IRE), which is one of ProRep grantees. However, I have not bene involved in the ProRep project don by IRE. Currently, I am not employed as a full time employee and not receiving salary from IRE. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 4 January 2016 80 Name Isma Novitasari Yusadiredja Title Monitoring and Evaluation Team Member Organization Social Impact Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (or RFTOP or other appropriate instrument number) AID-497-C-11-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ProRep, Chemonics I have real or potential conflict of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08 January 2016 81 ANNEX XI. EVALUATION TEAM BIOGRAPHIES Patrick J. Fn’Piere, Team Leader Patrick Fn’Piere is an international development and public policy executive with over 25 years of experience building democratic institutions of government. Currently, he is a senior associate at Democracy International (DI), a Washington-based firm that provides analytical services to Bi and Multi￾lateral international development institutions. An expert in working with legislative bodies, Mr. Fn’Piere has conducted formal impact evaluations of USAID legislative strengthening programs in the Philippines, Mexico, and Afghanistan; program and project reviews in: Guatemala, Bolivia, and Egypt; project designs for USAID programs in: Pakistan, Zimbabwe, and Haiti. As the Head of the Governance Office within the Center for Democracy & Governance, he convened the first international conference sponsored by USAID on Legislative Strengthening in 1996 and was a contributing editor/writer on the first USAID handbook on strategies for providing assistance to legislative bodies worldwide. Isma Novitasari Yusdiredja, Monitoring and Evaluation Specialist Isma Novitasari Yusadiredja is a socio-development professional with over fourteen years of experience. She has experience in monitoring and evaluation, qualitative and quantitative research, and report development. Most recently, Ms. Yusdiredja was an Endline Evaluation Quality Assurance Technical Advisor for the USAID funded KINERJA Project. Currently, Ms. Yusdiredja is a Researcher for the Health Research Unit at the Padjajaran University Bandung evaluating a community-based HIV/AIDS prevention and care program for most at risk population. She holds a Masters in International Health from Ruprecht-Karls Universität Heidelberg. Ashari Cahyo Edi, Policy Expert Team Member Ashari Edi, has over seven years working in democracy, governance, and policy in the Indonesian context. Mr. Edi holds a Master’s in Public Administration from the University of Delaware. He is currently a faculty member in the Department of Politics and Government at the Gadjah Mada University. As a policy expert, he has served as project coordinator and a research for a number of democracy and governance projects and grants in Indonesia. Mr. Edi has worked on grants or projects funded by USAID, UNDEF, The Ford Foundations, Mercy Corps, and National Democratic Institute for International Affairs, and DfID. He has several published papers on the topics of governance and democracy in Indonesia and was the recipient of USAIDS’s Program to Extend Scholarships and Training to Achieve Sustainable Impacts (PRESTASI), Democracy and Governance Sector from 2010-2013. 82 ANNEX XII. PROREP ACTIVITY MAP 83 ANNEX XIII. SOCIAL IMPACT RESPONSE TO USAID COMMENTS ON DRAFT REPORT Comments were submitted to the Social Impact Evaluation Team both in the body of the report and in an Evaluation Report Checklist. All comments are noted in the table below, followed by the Evaluation Team’s corresponding response and/or revision. Comments submitted to the team within the body of the report are titled “in document” comments. Comments submitted to the team within the Evaluation Report Checklist are titled “in checklist” comments. 84 COMMENT NUMBER USAID COMMENT SI RESPONSE/DISCUSSION TITLE PAGE USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in checklist) The title doesn’t clearly indicate that this was a performance evaluation. This has been updated. Thank you for the note. Comment 2 (in checklist) A title block in USAID light blue background color with the word “Evaluation” at the top and the report title (also including the word “evaluation”) underneath? This has been updated. Comment 3 (in checklist) Suggest adding (ProRep) in the title This has been added to the title page. Comment 4 (in checklist) A caption explaining the “who, what, when, where, and why” of the cover photo, with photographer credit? There is no photographer credit and the connection between the photo and ProRep is not clear. The cover photograph caption has been updated with the requested details. CONTENTS USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) Now we called this the Ministry of Education and Culture (MOEC) instead. Indonesian: Kementrian Pendidikan dan Kebudayaan. Please make sure this term is consistency used throughout the document. Thank you for the note. This has been updated in the final report (in table of contents and on related pages). ACRONYMS USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in checklist) Some acronyms were not translated into English. All acronyms in Bahasa Indonesia are now followed by English translations. EXEC SUMMARY USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in checklist) The executive summary doesn’t contain methods. Also, it’d be helpful to start this section with the purpose of the evaluation, not background, which initially focused on the portion of ProRep that was not evaluated. It would have been easier to follow this section if it was organized by evaluation questions. A short description of the performance evaluation methods has been added to the Executive Summary. Additionally, the Executive Summary now begins with a clear description of the purpose of the evaluation, followed by relevant program background information. The Evaluation Team removed information about Phase I of ProRep, as it is not the focus of the evaluation. Comment 2 (in checklist) The executive summary does not include methods and limitations. Also, it’s hard to see the connection between specific questions, findings/conclusions and recommendations. Methods and limitations have been added to the Executive Summary. The team has clarified what findings and conclusions included in the Executive Summary relate to what evaluation question(s). BACKGROUND USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) Missing page 10 This has been corrected. Comment 2 (in checklist) Missing information on award number, award dates, and funding level. This information has been added in footnote #1. 85 Comment 3 (in checklist) The focus sectors were listed, but no context for them was presented. The evaluation team believes that the important context to establish in the background section for ProRep is regarding governance, not the specific policy sectors covered by ProRep (health, education, and environment). Providing further details on the status of each of these sectors does not appear contribute to or strengthen the report findings and conclusions about governance￾strengthening approaches. Comment 4 (in checklist) The background section provides minimal sector and country context for the policy cluster activities. It is not clear if there are sector differences in the policy-making environment in Indonesia or how the broad decentralization process is affecting policy/law￾making. The evaluation report does not include a map of ProRep activity sites so it’s harder to understand the activity’s geographic scope. The theory of change underlying ProRep’s cluster approach is not adequately presented. The Evaluation Team has added a ProRep Activity Site map and clear description of the program’s results framework (as detailed in the PMP) to the background section. The team also added an Annex that includes another ProRep map in more detail (Annex XII). The theory of change for ProRep has been added to this section. Regarding the sector and country context, see Comment 3 response above. ASSESSEMENT METHODS AND LIMITATIONS USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) Please indicate which ones. Either here or under table 1. This information has been clarified in Table 1 (with additional respondents denoted with an asterisk). The respondents have also been added to the sentence indicated in the comment. Comment 2 (in checklist) Although the methods section describe the specific data collection approach, it is not clear how different data collection methods relate to each evaluation question. Also, as no background information was provided on the geographic scope of ProRep, it was hard to understand how representative Solo and Bengkulu sites were of the local policy cluster efforts. In the description of the FGD approach, it was unclear whether facilitators were ET members or other individuals. Did members of the health and education clusters participate in separate or joint FGDs in Solo? Instead of noting that the ET met with “99 percent of respondents on the USAID list” (p. 17), it’d be helpful if there was a total number of respondents that the ET met and the number of respondents that it didn’t meet.  The Evaluation Team has added details regarding how each type of data contributed to the answering of evaluation questions (see Annex IX).  Geographic scope information is now included in the background section, as noted above. Under “Site Visits”, the Evaluation Team added the following description: “Two site visits were selected based on evaluation budget and input from USAID and the IP. The site visits provided the ET an opportunity to collect in-depth information regarding two specific policy objectives pursued by clusters during ProRep.” The site visits were not selected in order to represent all local-level policy initiatives. They were selected to provide in-depth details about 86 two policy objective efforts (as all are unique within ProRep). A detailed map is included in Annex XII.  Regarding FGDs, these were facilitated by the ET. A note has been added in the text to clarify this point.  Health and education cluster respondents participated in separate FGDs in Solo. A sentence has been added to clarify this.  The ET met with a total of 42 respondents for KIIs, and conducted three FGDs and two site visits. These are included in the methodology section. The ET was only unable to speak with/contact three individuals from the USAID￾provided list. See footnote 7. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS General I (in checklist) The recommendations did not consistently and clearly follow from findings and conclusions. Also, they did not consistently designate the executor and seem to primarily focus on potential improvements to the next version of ProRep (i.e. a donor￾supported policy cluster approach that raises sustainability concerns). The evaluation team has added bolded text to identify the executer of each recommendation more clearly. The evaluation team believes that the recommendations do clearly follow from the findings and conclusions presented in the report. General 2 (in checklist) Although gender considerations are likely to play a role in education and health policies, the evaluation team did not address these issues. The gender discussion in the report appeared to focus on ensuring a more equal gender representation among respondents, but not policy considerations of gender. Consequently, recommendations do not take gender issues into account. During the in-brief with USAID Indonesia, the evaluation team asked if USAID had any gender-specific concerns or questions they wanted to see addressed in the evaluation. There were not any issues mentioned, and so SI followed internal protocols by including gender questions in the questionnaires and disaggregating information by gender when possible (for example, regarding respondents). Additionally, the team analyzed the data for any unanticipated outcomes that related to gender from ProRep – none were identified. While gender does play a role in sectors in Indonesia, the evaluation team was not tasked to evaluate the policies or the sectors. The task of the evaluation was to evaluate the policy cluster approach as a governance-approach. 87 Nonetheless, the evaluation team did find from its field investigations that the cluster approach had a high rate of women’s participation in cluster activities within all three clusters; that women leadership in clusters were pronounced; and gender-focused /led issues were supported, such as the Health Cluster’s work on reforming insurance policies related to the inclusion of mid-wife services in health provision. The evaluation team projects that the cluster approach can be an effective tool for gender inclusion. Gender-related information has been added to Question 2 in the report. QUESTION 1 USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) We noticed that both terms of policy-making and rule making are used in this document. Do they refer to two different things? If so, please clarify. The following text has been added to clarify this point: “Policy-making is primarily undertaken by the executive branch through national ministerial or presidential decrees. These policies require rule-making for implementation. That occurs primarily at sub-national levels of government. Rule-making at sub-national levels rather than at the national level provides opportunities for adaptation to different sub-national characteristics. The variety of sub-national experiences can then be aggregated to inform national policy improvements.” Comment 2 (in document) These two paragraphs are confusing. We recommend a revision. The first paragraph has been removed. Comment 3 (in document) Please clarify what meant by deconcentrated national offices in this context. Otherwise, please consider rephrasing, We have re-phrased the point as follows: “Secondly, ProRep CSOs, think tanks, and government officials interviewed relayed that the cluster approach was effective at taking advantage of the opportunity gained through decentralization, i.e., moving substantial authority, resources, and responsibility for policy implementation to local governments.” Comment 4 (in document) The statement that says cluster approach did not deliberately reinforce the strategic value of research capacity is not entirely accurate. Some of the policy cluster partners are members of the Policy Research Network (PRN) that was established with ProRep support. Support to PRN members include research capacity for members. We suggest this sentence be revised. We have revised this sentence as follows: “Additionally, while the cluster approach did not deliberately reinforce the strategic values of community outreach, it did seem to utilize local communities in carrying out grant activities.” 88 Comment 5 (in checklist) For conclusions for question 1, the distinction between “pure advocacy” and “more effective engagement in participatory governing” was not clear. Also, did the ET consider think tanks as non-CSOs? If so, a definition of a CSO would have been useful. On p. 22, the report appears to reference an incorrect DO (DO3 as “Democratic governance strengthened”).  This sentence has been edited, as follows: “The ProRep cluster was designed as a new tool for CSOs to move from advocacy to more effective engagement in governing at both national and sub-national levels, requiring them to more systematically work with think tanks to better understand and subsequently produce materials relevant to government officials to achieve a successful policy outcome.”  The evaluation team has added program definitions of think tanks and CSOs, to provide clarity. These footnotes can be found in the “Program Background” section. The definitions referenced are those from the implementing partner’s contract with USAID for ProRep.  The DO reference has been corrected to DO 1. QUESTION 3 USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) Please consider using different term. And please make it consistent throughout the document. The sentence has been revised as follows: “The national level policy contributions from the Environment Cluster illustrate this cycle.” QUESTION 4 USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in document) We don’t think that the findings adequately answer the second question on “Have different methods of operationalization affected achievement of cluster objectives?” despite the conclusion “The ET did not find any significant structural or fundamental differences among sectors that would require different modalities for operationalizing sector clusters in the future”. Please consider a revision. The evaluation team did not find different methods for cluster operationalization; therefore, different methods of operationalization did not affect achievement of cluster objectives. Question 3 findings and conclusions established that all clusters shared core elements in how they operated. The evaluation team has provided additional clarity on this point in the first sentence of the conclusions section for Question 4: “The ET did not find any significant structural or fundamental differences among sectors that would require different modalities for operationalizing sector clusters in the future; therefore, the cluster approach in each sector did not impact achievement of objectives differently.” ANNEXES USAID COMMENT SI RESPONSE/DISCUSSION Comment 1 (in checklist) Signed disclosure of conflict of interest forms of evaluation team members? (These are required to be signed, but only recommended to be included in the report annex.) Conflict of interest forms for all evaluation team members have been included as Annex X. 89 Comment 2 (in checklist) Abridged bios of the evaluation team members, including qualifications, experience, role on the team, and ethical certifications (optional, required for PEPFAR evaluations)? Short biographies of each team member have been included in Annex XI. 90 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523