MADANI CIVIL SOCIETY SUPPORT INITIATIVE Midterm Performance Evaluation JANUARY 9, 2023 Contracted under GS-00F-210DA / 72049720M00001 Indonesia Monitoring, Evaluation, and Learning Platform (MEL-P) Authors Peter Bauman, Team Leader Saut Sagala Dianty Ayu Basyirah Alwi DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. i | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ABSTRACT This midterm evaluation report of the USAID/Indonesia MADANI Civil Society Support Initiative assesses MADANI’s progress in (1) improving CSO organizational capacity, (2) strengthening LG accountability and promoting tolerance, (3) improving CSO financial sustainability, and (4) improving the enabling environment for CSO engagement at the local level. The evaluation employs a mixed￾methods approach consisting of an extensive document review, 110 interviews with 273 key stakeholders, and quantitative data from MADANI and secondary sources. The evaluation finds evidence of organizational improvement in CSOs and their staff as well as improved CSO ability to advocate for government accountability and promote tolerance. MADANI has also helped CSOs become more eligible for donor funding and has strengthened the enabling environment for CSO engagement in the local development process. The CSOs face challenges in implementing collaborative governance processes on their own because of technical, resource, and time-related issues. They also still require support in financial management, budgeting, and resource mobilization to ensure their sustainability. The political environment, MADANI’s short activity lifetime, and CSOs’ nascent level of capacity are remaining challenges for sustainability. Cover photo caption: Members of the health cadres at one of three revitalized health care centers for teenagers (Puskesmas Remaja) in Sukabumi (West Java). For the last two years, USAID MADANI and local civil society organization Saba Desa have advocated for reopening and revitalizing these health care centers throughout the District of Sukabumi. The revitalized centers will provide teenagers and youth with better access to health services and the opportunity to participate in reducing stunting and maternal and infant mortality rates. Photo credit: Ade Sonyville for USAID/MADANI USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | ii CONTENTS ABSTRACT ............................................................................................................................... I CONTENTS ............................................................................................................................ II LIST OF TABLES........................................................................................................................................................III LIST OF FIGURES......................................................................................................................................................III ACRONYMS........................................................................................................................... IV EXECUTIVE SUMMARY.......................................................................................................VI EVALUATION PURPOSE ...................................................................................................................................... VI BACKGROUND ...................................................................................................................................................... VI ACTIVITY DESCRIPTION..................................................................................................................................... VI EVALUATION QUESTIONS................................................................................................................................. VI EVALUATION METHODOLOGY..................................................................................................................... VII KEY FINDINGS AND CONCLUSIONS........................................................................................................... VII RECOMMENDATIONS..........................................................................................................................................IX INTRODUCTION.................................................................................................................... 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS ......................................... 1 EVALUATION PURPOSE ........................................................................................................................................ 1 EVALUATION QUESTIONS................................................................................................................................... 1 ACTIVITY DESCRIPTION ..................................................................................................... 3 DEVELOPMENT CONTEXT.................................................................................................................................. 3 ACTIVITY OVERVIEW............................................................................................................................................. 3 MADANI THEORY OF CHANGE AND ASSUMPTIONS.............................................................................. 4 EVALUATION METHODS AND LIMITATIONS................................................................ 5 LIMITATIONS ............................................................................................................................................................. 7 FINDINGS AND CONCLUSIONS ........................................................................................ 8 EQ1: HAS MADANI IMPROVED THE ORGANIZATIONAL CAPACITY OF ITS CSO PARTNERS? IF SO, HOW? IF NOT, WHY NOT? .................................................................................................................... 8 FINDINGS........................................................................................................................................................... 8 CONCLUSIONS .............................................................................................................................................17 EQ2: HAS MADANI STRENGTHENED THE LOCAL GOVERNMENTS’ ACCOUNTABILITY AND PROMOTED TOLERANCE? IF SO, HOW? IF NOT, WHY NOT?............................................................18 FINDINGS.........................................................................................................................................................18 CONCLUSIONS .............................................................................................................................................26 EQ3: HAS MADANI IMPROVED THE SUSTAINABILITY OF TARGETED CSOS IN BRINGING ABOUT CHANGE IN POLICY AND PRACTICE? IF SO, HOW? IF NOT, WHY NOT?....................26 FINDINGS.........................................................................................................................................................26 CONCLUSIONS .............................................................................................................................................30 EQ4: HAS MADANI IMPROVED THE ENABLING ENVIRONMENT FOR CSO ENGAGEMENT? IF SO, HOW? IF NOT, WHY NOT?.......................................................................................................................31 iii | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV FINDINGS.........................................................................................................................................................31 CONCLUSIONS .............................................................................................................................................34 RECOMMENDATIONS......................................................................................................... 35 RECOMMENDATIONS FOR CSO ORGANIZATIONAL CAPACITY.....................................................35 RECOMMENDATIONS FOR LG ACCOUNTABILITY AND PROMOTING TOLERANCE..............36 RECOMMENDATIONS FOR CSO FINANCIAL SUSTAINABILITY..........................................................37 RECOMMENDATIONS FOR ENABLING ENVIRONMENT FOR CSO ENGAGEMENT...................37 ANNEXES............................................................................................................................... 38 ANNEX I: TIMELINE...............................................................................................................................................38 ANNEX II: EVALUATION STATEMENT OF WORK....................................................................................40 ANNEX III: EVALUATION DESIGN MATRIX................................................................................................. 51 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS.....................................................................54 ANNEX V: SOURCES OF INFORMATION.....................................................................................................65 DOCUMENTS REVIEWED ..........................................................................................................................65 LIST OF RESPONDENTS..............................................................................................................................67 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST .............................................................72 ANNEX VII: EVALUATION TEAM MEMBERS................................................................................................. 76 LIST OF TABLES Table 1. Interview Distribution ............................................................................................................................... 6 LIST OF FIGURES Figure 1. MADANI Theory of Change and Results Framework ..................................................................... 4 Figure 2. MADANI Districts per Province ........................................................................................................... 5 Figure 3. Year 1 and Year 2 OPI Level of Organizational Maturity................................................................ 9 Figure 4. Year 1 and Year 2 OPI Average Scores by Domain ......................................................................... 9 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | iv ACRONYMS ADS Automated Directive System AMELP activity, monitoring, evaluation, and learning plan AOR agreement officer’s representative CLA collaborating, learning, and adapting COP chief of party CSO civil society organization CSR corporate social responsibility DPA budget execution document DRG democracy, human rights, and governance DSP district support partner EQ evaluation question FC field coordinator FGD focus group discussion FHI Family Health International FPI Forum Performance Index FY fiscal year GESI gender equality and social inclusion GOI Government of Indonesia iHub Innovation and Knowledge Hub IP implementing partner IR intermediate result Kemenag Ministry of Religious Affairs Kemendagri Ministry of Home Affairs Kemenkes Ministry of Health Kemenkumham Ministry of Law and Human Rights KemenPAN-RB Ministry of State Administrative and Bureaucratic Reform KMIS Knowledge Management and Information System KII key informant interview LF Learning Forum LP lead CSO partner LG local government MCH maternal and child health MEL monitoring, evaluation, and learning MEL-P Monitoring, Evaluation, and Learning Platform MNH maternal and newborn health MOU memorandum of understanding NSP national service provider OCA organizational capacity assessment v | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ONA organizational network analysis OPI Organizational Performance Index PE performance evaluation PITT performance indicator tracking table PII personally identifiable information PSP provincial support partner RKPD RPJMD SEKDA Regional Working Plan Regional Medium-Term Development Plan regional secretary SFC senior field coordinator SOP standard operating procedure SOW statement of work TOC theory of change TKPKD Coordinating Team on Regional Poverty Alleviation USAID United States Agency for International Development USG United States Government USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | vi EXECUTIVE SUMMARY This report presents the findings, conclusions, and recommendations from the midterm performance evaluation (PE) of the MADANI Civil Society Support Initiative (MADANI) in Indonesia, implemented by FHI360. The evaluation was commissioned by the USAID Indonesia Mission and designed and implemented by the Monitoring, Evaluation, and Learning Platform (MEL-P). EVALUATION PURPOSE This midterm performance evaluation aims to inform Mission staff and the MADANI team (FHI360) on progress toward activity outcomes and objectives. The evaluation results will inform program adjustments and future strategy for work with civil society organizations (CSOs), particularly at the subnational level. The evaluation’s primary audience is USAID leadership and the MADANI implementing partner. The secondary audience includes the Government of Indonesia (GOI) at the national and subnational levels, development partners, other USAID operating units, and other stakeholders who may benefit from the findings and conclusions. BACKGROUND Indonesia has seen significant socioeconomic and political progress since the fall of Suharto in 1998, yet challenges remain at the subnational and district levels. Beyond corruption and graft, the quality￾of-service provision remains low, citizen needs are often excluded in government plans and priorities, and access to frontline services in disadvantaged areas is limited. In addition to low levels of local government accountability, Indonesia is experiencing a rise in intolerance. Furthermore, women and persons with disabilities often receive unequal treatment. Although Indonesia boasts a vibrant democracy, civil society is hindered by inefficient organizational management, lack of technical expertise, dependence on charismatic leaders, low accountability, and weak financial management. Dependence on donor support, the current legal environment, and the legacy of mutual distrust between GOI and civil society also impede CSOs’ effectiveness and sustainability. ACTIVITY DESCRIPTION To overcome some of these challenges, the $19.8 million USAID MADANI Civil Society Support Initiative was designed to strengthen government accountability and community tolerance at the local level by improving and sustaining local civil society’s capacity, legitimacy, and sustainability. To achieve its goal and objectives, MADANI was designed to support targeted “lead” CSOs in 32 districts across six provinces, including Banten, West Java, Central Java, East Java, West Kalimantan, and South Sulawesi. Commencing in March 2019 and scheduled to end in April 2024, MADANI is at the midpoint of its five-year period of performance. To complement MADANI’s internal evaluations and provide an independent assessment, USAID commissioned MEL-P, led by the Panagora Group, to conduct an external midterm performance evaluation. EVALUATION QUESTIONS The evaluation answers the following four evaluation questions (EQs): 1. Has MADANI improved the organizational capacity of its CSO partners? If so, how? If not, why not? 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? vii | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV 3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 4. Has MADANI improved the enabling environment for the CSOs’ engagement? If so, how? If not, why not? EVALUATION METHODOLOGY This performance evaluation employed a mixed-methods approach that included primary and secondary data collection. A four-person evaluation team conducted a thorough document review and 110 interviews (45 KIIs and 65 small group interviews) with 273 individuals (135 male and 138 female). Semi-structured interviews were conducted with all relevant stakeholders remotely and in person in Washington DC, Jakarta, and 16 districts in all six provinces in July 2022. The evaluation team split into two teams and spent approximately one week in each province (two days per district). The evaluation team also reviewed and analyzed qualitative and quantitative data from MADANI and other secondary sources. To ensure that the findings were evidence based and objective, the evaluation team analyzed primary and secondary data; identified patterns; and applied source, method, and evaluator triangulation techniques. Although the evaluation triangulates evidence across multiple data sources and stakeholder categories to strengthen the findings’ validity and reliability, it has some limitations, including potential bias due to self-reported data from interviews, inability to provide generalizable district￾level analysis, problems answering some EQs due to limited data availability, and insufficient resources to quantify qualitative findings. KEY FINDINGS AND CONCLUSIONS EQ1: Has MADANI improved the organizational capacity of its CSO partners? If so, how? If not, why not? Evidence shows organizational improvement. CSOs have completed or revised SOPs and CSO staff have gained knowledge, skills, and experience using organizational development tools. Although most CSOs facilitated by MADANI were at a nascent level, they had completed or revised their SOPs in financial management, procurement, and human resource management; five-year strategic plans; and MEL plans in one year. According to Year 2 Organizational Performance Index (OPI) results, 31 of the 32 CSOs had improved their overall scores and almost all nascent CSOs had moved into the emergent level. Scores also rose across the five domains, with the greatest improvements in effectiveness and efficiency. Tapping into the CSOs’ ability to apply learning to their own organizational results and core issues could ensure that the CSOs can replicate what they have learned to sustain outcomes post-MADANI. EQ2: Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? MADANI has provided CSOs with processes and tools to promote its approach to collaborative governance and has created opportunities for CSOs, local governments (LGs), and communities to come together, exchange information, build relationships, and achieve tangible changes in service delivery, albeit not across all districts. However, CSOs continue to need technical support, resources, and time to adopt the collaborative governance process on their own, and it is unclear whether they could replicate the approaches or sustain the advocacy efforts without the backing of USAID or MADANI. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | viii Pressure to show results against indicators may circumvent genuine CSO capacity building in the three intermediate results (IRs), which are the foundation of MADANI’s theory of change. It also oversimplifies advocacy for LG accountability and community tolerance to the drafting of policy briefs and the use of certain tools. CSOs and Learning Forums may not understand the real challenges, time, and resources associated with strengthening LG accountability and communal tolerance. It is too early to determine whether MADANI’s approach will strengthen LG accountability and promote tolerance in communities. EQ3: Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? MADANI has helped CSOs become more eligible for donor funding, and some CSOs have received funding from various sources. However, they still need significant organizational capacity development and support, including financial management, budgeting, and resource mobilization planning and implementation. Other factors involved in financial sustainability may be beyond MADANI’s purview, including human and financial resources, leadership, motivation, past performance, proposal writing skills, marketing and presentation skills, creativity, relationships with donors, entrepreneurial skills, professional and social networks, time management, and investing skills, among others. Therefore, through national service providers, provincial support partners, and other instruments, MADANI expects to extend the amount of TA and mentoring, introducing lead CSO partners (LPs) to strategies to enhance organizational capability, followed by a year of learning￾by-doing in which LPs build their understanding and skills in fundraising. MADANI’s financial sustainability efforts have yielded positive results in ensuring CSOs’ ability to sustain and secure their finances through established connections with donor and government￾funded projects, although this could become an impending challenge on CSOs’ ability to independently advocate LG accountability and hamper CSOs’ self-sufficiency as explained in MADANI Working Paper I. Additionally, complacency on contractual affairs might affect the CSOs' authentic stance of accountability, particularly on upholding the rule of law in governance implementation. Improving CSOs’ capacity to sustainably affect change in policy and practice goes beyond supporting the CSOs in acquiring financial resources. If the goal is to encourage CSOs to continue to advocate for LG accountability using MADANI’s collaborative governance approach, the Learning Forums will also need support in resource mobilization and financial sustainability. EQ4: Has MADANI improved the enabling environment for the CSOs’ engagement? If so, how? If not, why not? MADANI has strengthened the enabling environment for CSO engagement through collaborative governance, pilot projects, public consultation forums, exploratory networks, and capacity building in financial management and resource mobilization. This endeavor has increased engagements between LG and CSOs. LGs who engaged in collaborative governance procedures have also issued a decree encouraging CSO involvement and participation. Some CSO policy recommendations have been incorporated into regulations or adopted into related programs. MADANI's preselected themes, prescribed collaborative governance approach, milestone-focus, and active USAID support have been the driving factors of involvement in activities so far. However, obtaining an issue-specific dialogue, trust building, and alliance building oriented to the local communities have proved challenging. The political atmosphere has influenced the enabling environment between CSOs, government officials, and communities. If the supportive LGs were ix | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV replaced, concerns would arise about whether the CSOs can effectively manage and implement projects for donors and other public and private institutions. RECOMMENDATIONS The following are key recommendations based on the evaluation findings and conclusions. Additional details are included in the body of the report. CSO Organizational Capacity ● The IP and MADANI need to shift the organizational development strategy to build organizational capacity. ● USAID and MADANI need to upgrade current training mechanism to expand effective mentoring approaches, such as the district support partner model. ● IP needs to reinvigorate gender equality and social inclusion (GESI); diversity and tolerance; and anti-corruption issues through integrated activities. ● USAID and MADANI need to make CSO work plans more flexible so that they can better adapt GESI, diversity, and inclusion topics or design and implement new activities in collaboration with other partners. ● The IP needs to disseminate best practices as reference resources and for peer-to-peer learning. ● USAID needs to review and strengthen MADANI’s administrative and operational processes to identify inefficiencies that may hinder program implementation and overburden the staff. LG Accountability and Promoting Tolerance ● Strengthen CSO capacities and self-confidence in collaborative governance processes by designing and tailoring tools to individual CSO contexts. ● Test experiential learning approaches in selected districts by taking lessons learned and applying the process in another location/theme with less intensive support from MADANI. ● The IP needs to provide more training and practice on conducting sociopolitical, economic, and stakeholder analysis so that CSOs understand the barriers and opportunities in advocacy work. ● The IP should incorporate a mid-course pause-and-reflect exercise into quarterly "Learning Weeks" within MADANI’s M&E system for CSOs on the theory of change, underlying assumptions, contextual changes, and the extent to which activities are contributing to the achievement of program objectives. ● MADANI needs to develop and adjust a mechanism that assesses the process of engagement among stakeholders. CSO Financial Sustainability ● USAID and MADANI need to set realistic expectations of what can be achieved in building CSO capacity to mobilize resources. ● The IP needs to assess the degree to which sustainability initiatives result in CSOs becoming donor-driven, or government contractors could limit their independence to advocate for LG accountability. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | x ● The IP needs to define MADANI’s goal for capacity building on resource mobilization and ensure that the training curriculum supports it, builds on CSO capacities, and provides relevant approaches/tools. ● The IP should develop a sustainability plan for the Learning Forums to continue after the program ends and include LF members in capacity-building activities for financial resource mobilization. ● The IP needs to start preparing and developing exit strategies with CSOs to minimize programming and staffing disruptions when MADANI’s financial support ends. Enabling Environment for CSO Engagement: ● The IP needs to strengthen Learning Forums to provide sustainable space for CSO engagement in the local development process. ● The IP must assess MADANI’s efforts around unmet community expectations and apply the Do No Harm principles as part of the exit strategy. ● Consider locally led actions, innovative ideas, and transformative approaches that could institutionalize improvements in the enabling environment. 1 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV INTRODUCTION This report presents the midterm performance evaluation for the five-year MADANI Civil Society Support Initiative (MADANI) in Indonesia. The evaluation was commissioned by the USAID/Indonesia Mission and was designed and implemented by the Monitoring, Evaluation, and Learning Platform (MEL-P). Annex A provides USAID’s Statement of Work for evaluation. This evaluation complements MADANI’s internal evaluations through an independent assessment of current program achievements and progress toward program objectives. As stated in the Statement of Work (SOW), this midterm performance evaluation covers the following tasks: fieldwork and site visits, implementing partner (IP) data collection, coordination meetings, review of data monitoring performance, implementation documentation, MADANI’s midline survey, and fidelity to the agreed-upon implementation and performance evaluation designs, as well as communication and outreach to local stakeholders about the implementation of MADANI’s evaluations. To ensure that this evaluation is carried out properly, MEL-P is responsible for: ● Working with USAID and MADANI to determine the sample frame for the evaluation ● Developing and testing data collection instruments ● Collecting and analyzing quantitative and qualitative data, including reviewing and assessing ● Producing an evaluation design report and an evaluation report ● Facilitating a learning event and presentation of the evaluation findings This evaluation is also in accordance with Automated Directive System (ADS) 201.3.5.8 guidance; all data used for this evaluation will be reviewed for validity, reliability, timeliness, precision, and integrity. EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE This evaluation assesses MADANI’s progress toward its outcomes and objective. The evaluation results will provide Mission staff and the MADANI team (FHI360) with findings, conclusions, and recommendations to make necessary project adjustments and inform future strategy on working effectively with civil society organizations (CSOs), particularly at the subnational level. The evaluation’s primary audience includes USAID staff and leadership and the MADANI IP. The secondary audience includes the Government of Indonesia (GOI) at the national and subnational levels, development partners, other USAID operating units, and other stakeholders who may benefit from the findings and conclusions. EVALUATION QUESTIONS In accordance with USAID’s SOW, with revisions approved by USAID on June 10, 2022, and on September 27, 2022,1 this evaluation answers the following four evaluation questions (EQs): 1 USAID approved the evaluation design with changes to the EQs. Subsequently, USAID requested that EQ4 be modified back to that of the original SOW; however, since data collection and analysis had already concluded, the evaluation team and USAID agreed to change EQ4 to only the parts that could be answered with the data collected. Additional details on the implications of this change can be found in the Limitations sections. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 2 1. Has MADANI improved the organizational capacity of its CSO partners? If so, how? If not, why not? 1.1. How has MADANI improved the organizational effectiveness of its CSO partners to achieve their organizational results? 1.2. How has MADANI improved the organizational efficiency of its CSO partners' capacity to manage core issues? 1.3. Which approaches or tools are more effective for building organizational capacity? Why? What approaches or tools are less effective? Why? 1.4. What enabling factors (conditions) are most conducive for improving organizational capacity? 1.5. Did MADANI mainstream GESI into CSO organizational capacity development? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 1.6. How have MADANI and its CSO partners coped with the pandemic? How has the pandemic affected MADANI’s effectiveness in delivering technical assistance? 1.7. What are the key lessons learned from MADANI in terms of building the organizational capacity of its CSO partners? 1.8. What are the barriers to and opportunities for improving CSO organizational capacity? 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1. Has MADANI enabled its CSO partners to advocate for government accountability in their communities? If so, how? If not, why not? 2.2. Has MADANI enabled its CSO partners to advocate for tolerance in their communities? If so, how? If not, why not? 2.3. Which social accountability tools or mechanisms were most effective and why? 2.4. What factors (i.e., size and type of CSO, level of preexisting capacity, funding sources, etc.) were most conducive for success? 2.5. Did MADANI mainstream GESI into its work on accountability and tolerance? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 2.6. What are the key lessons learned from MADANI in terms of working with subnational CSOs to strengthen accountability and promote tolerance? 2.7. What are the barriers and opportunities to improve CSO capacity to strengthen accountability and promote tolerance? 3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1. How has capacity improvement led to financial sustainability for the different types of CSOs? 3.2. What is the likelihood that this improvement will be sustained after MADANI support ends? 3.3. What factors contribute to facilitating sustainability? (i.e., size of CSO; experience; reputation; popularity of staff; relationships; history with LG or other stakeholders; endorsement from reputable party; religious or political affiliation; etc.) 3.4. Which approaches or tools are most effective for improving the financial sustainability of CSOs? Why? 3.5. Did MADANI mainstream GESI into the design and implementation of activities related to resource mobilization and improving the enabling environment to improve the sustainability of the CSO partners? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 3.6. What are the key lessons learned from MADANI in terms of building the organizational capacity of CSOs to mobilize resources? 3 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV 3.7. What are the barriers and opportunities to improve CSO capacity to mobilize resources? 4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1. In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? ACTIVITY DESCRIPTION DEVELOPMENT CONTEXT Indonesia has seen significant socioeconomic and political progress since the fall of Suharto in 1998. For instance, with support from USAID and other donors, Indonesia has made wide-ranging political and social reforms that “have led to greater freedoms of the press, limitations on presidential powers and terms, a stronger enabling environment for civil society, and the decentralization of power and resources to the regions.”2 Indonesia also boasts one of the world’s fastest-growing economies. However, significant challenges remain at the subnational and district levels. Beyond corruption and graft, quality-of-service provision remains low; citizen needs are often excluded in government plans and priorities; and access to frontline services, including maternal and newborn health (MNH) in disadvantaged areas, is limited. In addition to low levels of local government (LG) accountability, Indonesia is experiencing a rise in intolerance, including attacks against religious minorities, destruction of property, and discrimination in public services such as education and health and in granting official certificates.3 Furthermore, women and people with disabilities often receive unequal treatment in employment and access to services and facilities. As stated in the MADANI Cooperative Agreement, “civil society must be at the frontlines of encouraging greater accountability and tolerance.” However, although Indonesia boasts a vibrant democracy, civil society is hindered by inefficient organizational management, lack of technical expertise, dependence on individual charismatic leaders, low accountability, and weak financial management. Dependence on donor support and the current legal environment also impede CSO effectiveness and sustainability. The legacy of mutual distrust between GOI and civil society often compounds these challenges. The COVID-19 pandemic also had a negative impact on CSO effectiveness and sustainability. For instance, many donor and government funds were diverted to the response, and social distancing limited in-person interaction and forced CSOs to conduct their activities virtually. ACTIVITY OVERVIEW The USAID/Indonesia MADANI Civil Society Support Initiative is a $19.8 million activity focused on strengthening LG accountability and community tolerance by improving and sustaining local civil society’s capacity, legitimacy, and sustainability in 32 districts across six provinces. Commencing in March 2019 and scheduled to end in April 2024, MADANI is implemented by FHI360 and is at the midpoint of its five-year period of performance. 2 See Associate Cooperative Agreement No. 72049719LA00001. 3 Ibid. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 4 MADANI THEORY OF CHANGE AND ASSUMPTIONS MADANI aims to increase government accountability and communal tolerance in Indonesia by improving local civil society’s capacity, legitimacy, and sustainability. To accomplish this goal, MADANI provides direct TA and grants to target CSOs to improve their operational capacities and systems, build stronger ties with communities and LGs, and support an improved environment and more sustainable means of mobilizing financial resources, including from the government and the private sector. MADANI aims to improve local civil society’s ability to advocate for government accountability and communal tolerance at the local level in the following thematic areas: anti￾corruption, frontline service delivery, natural resource governance, maternal and newborn health, village fund transparency, communal tolerance, and diversity among youth. The MADANI results framework includes three intermediate results (IRs), each with two sub-IRs (Figure 1). Figure 1. MADANI Theory of Change and Results Framework To achieve its goal and objectives, MADANI supports targeted “lead” CSOs (LPs) in 32 districts across six provinces: Banten, West Java, Central Java, East Java, West Kalimantan, and South Sulawesi (Figure 2). 5 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Figure 2. MADANI Districts per Province Key Assumptions: According to MADANI program documentation,4 successful implementation is contingent on the continued validity of the following assumptions: ● MADANI activities will improve government responsiveness to CSO advocacy. ● CSOs will build their capacity and collaborate more effectively to safeguard democratic gains. ● GOI will allow MADANI to operate without interference. ● The political landscape continues to enable dialogue on governance reform. ● Local government agencies in targeted districts will be interested in replicating MADANI packets. MADANI is set to have internal baseline, midterm, and endline evaluations5 to monitor progress and expected results and to improve program design, management, and operational decision making. To complement these internal evaluations and provide an independent assessment, USAID commissioned the MEL-P, led by the Panagora Group, to conduct an external midterm performance evaluation. EVALUATION METHODS AND LIMITATIONS In alignment with the EQs, this evaluation will focus on MADANI’s progress in (1) improving CSO organizational capacity, (2) strengthening LG accountability and promoting tolerance, (3) improving CSO financial sustainability, and (4) improving the enabling environment for CSO engagement at the local level. This evaluation employed a mixed-methods approach that included primary and secondary data collection. A four-person evaluation team conducted a thorough document review and 110 interviews (45 key informant interviews [KIIs] and 65 small group interviews) with 273 individuals (135 male and 138 female). The sampling frames take into account stakeholders and components that contribute to MADANI’s implementation, including some with firsthand knowledge and 4 See AMELP December 2021. 5 Baseline data were collected in February 2020. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 6 experience of implementation approaches at either national or subnational level and some community members to provide multiple viewpoints on MADANI activities. Semi-structured interviews were conducted with all relevant stakeholders remotely and in person in Washington DC, Jakarta, and 16 districts in all six provinces in July 2022. The districts visited for in-person data collection included the following: ● East Java: Malang, Jombang, Jember ● Central Java: Boyolali, Surakarta, Wonosobo ● West Java: Bandung, Bogor, Garut ● Banten: Kab Tangerang, Serang ● West Kalimantan: Singkawang, Mempawah ● South Sulawesi: Makassar, Pangkep, Bulukumba The evaluation team split into two teams and spent approximately one week in each province (two days per district) conducting KIIs and small group discussions with a broad range of stakeholders (Table 1). Table 1. Interview Distribution STAKEHOLDER CATEGORY NUMBER OF RESPONDENTS Lead CSO partner 60 Learning forum 43 Community members 40 National government 6 Local government 51 National service provider 21 District support partner 11 Provincial support partner 11 MADANI team 22 Private sector 6 USAID 2 Total 273 Key stakeholders from each stakeholder category were selected based on the following criteria: ● The stakeholder has firsthand knowledge of and/or experience with MADANI activities and implementation approaches. ● The stakeholder holds a primary role in MADANI development and implementation. For example, the Ministry of Law and Human Rights (Kemenkumham) was selected as the main government stakeholder at the national level because of its determination to assist CSOs in Indonesia. At the district level, Bappeda and Kesbangpol from the six selected MADANI provinces are also critical stakeholders because of their roles in MADANI’s success; moreover, in some districts they have committed to providing resources, including office space and training materials, to support CSO development. ● The stakeholders as a group depict MADANI’s diverse thematic issues from various perspectives. CSOs were selected across the seven thematic areas selected by MADANI. ● Stakeholders include MADANI staff at the national and regional levels, particularly field coordinators, to provide perspective on program implementation and track CSO development within the region. ● Stakeholders include community leaders and members involved in MADANI activities—for instance, Mawar Desa Community in Serang, which contributed by filling out the community scorecard, and several thematically related institutions, such as the Community Health 7 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Center (Puskesmas) in Jember, that have intertwined established health programs with MADANI. The evaluation team also reviewed and analyzed qualitative and quantitative data from MADANI and secondary sources. To ensure that the findings were evidence based and objective, the evaluation team analyzed primary and secondary data; identified patterns; and applied source, method, and evaluator triangulation techniques. This evaluation report's methodology is predominantly qualitative and involves content analysis of qualitative data obtained from the Organizational Performance Index (OPI), a standardized self￾assessment conducted by the CSOs and facilitated by MADANI, KIIs, and focus group discussions (FGDs). The evaluation team interviewed a range of stakeholders in each district/province to provide a representative sample to analyze themes, patterns, lessons learned, and challenges for the overall project area. The evaluation team employed a theory of change (TOC) approach (contribution analysis). The team assessed MADANI’s overarching intervention logic and results framework to determine whether and how each part of the activity contributed to the larger objective. Throughout data collection and analysis, the team examined relationships among the various aspects of the TOC and tested the TOC assumptions’ validity. This process included examining the logical pathways between inputs, outputs, outcomes, sub-IRs, IRs, and MADANI’s overarching goal to compare what was planned and hypothesized with the change that had occurred. This approach helped the team assess the relevance of actions and results and identify potential gaps that need to be filled for MADANI to achieve its desired results. LIMITATIONS The evaluation’s design and implementation had several limitations that the evaluation team sought to mitigate. ● Potential respondent bias in self-reported data from OPI, KIIs, and group interviews. This evaluation relies on data collected from key informants and self-reported data from the OPI. Interview data are subject to cognitive biases, including recall bias, and self-reported OPI results may depend on participants’ interpretation of the questions. The evaluation team conducted systematic triangulation of interview and document sources and an appropriate selection of a range of MADANI stakeholders at the national, district, and local levels to strengthen the findings’ validity and reliability. These measures reduced the potential for bias. ● Inability to provide generalizable district-level analysis. The evaluation team visited half of the 32 districts where MADANI works. Given the variance in context and stakeholders, as well as in activity outputs and outcomes, the evaluation cannot provide a detailed assessment at the district level. However, the evaluation team interviewed a range of stakeholders in 16 districts across all six provinces to obtain a representative sample. ● The small number of CSOs advocating for tolerance limited the ability to answer the corresponding part of EQ2. Only two of the LPs focused on promoting tolerance, and each had a different focus (one on youth and the other on religious tolerance). Although the evaluation team conducted interviews in both districts with the CSOs and relevant stakeholders, the sample size was too small to draw reliable conclusions on whether MADANI enabled CSO partners to advocate for tolerance in their communities. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 8 ● The expanded data collection scope limited resources to quantify qualitative findings. To get a representative sample, the evaluation’s data collection scope was expanded to 16 districts in all six provinces, but the allocation of time and resources for completing the data collection, analysis, and drafting process did not change. As a result, the evaluation team had to consolidate, organize, and analyze an enormous amount of data in a short time, which limited its ability to quantify qualitative findings by number of observations or percentage of respondents. Instead, the evaluation team analyzed data within and across districts and stakeholder categories to develop a thorough understanding of responses, address contradictory findings, and highlight common themes and narratives. ● The activity’s implementation timeline was too limited to expect measurable changes in certain program outcomes and objectives. This midterm evaluation collected data and assessed change after three years of program implementation, which had a slow start because of the COVID-19 pandemic. Thus, it is too early to determine the sustainability of certain outcomes—for example, improvement of the enabling environment for CSOs at the local level and strengthening of LG accountability and promotion of tolerance. These outcomes require more time for program implementation to continue and shift into maturity. The evaluation team assessed the approaches, processes, results to date, and participant perspectives along the TOC to obtain a sense of the progress toward achieving the outcomes and objectives. ● Late-stage change of EQ4 constrained data availability and limited triangulation of data sources on this question. The EQ design underwent modifications to align with the original SOW and overall evaluation objectives. This process delayed USAID’s approval of the EQ. The evaluation team reviewed the data collected and determined that more information would need to be gathered from secondary sources and additional interviews with MADANI staff to answer the question. Unlike those of the other EQs, EQ4 findings and conclusions are not derived from the same broad range of stakeholders. Instead, they rely primarily on information gathered from MADANI staff and reported by MADANI. FINDINGS AND CONCLUSIONS EQ1: HAS MADANI IMPROVED THE ORGANIZATIONAL CAPACITY OF ITS CSO PARTNERS? IF SO, HOW? IF NOT, WHY NOT? FINDINGS Improved CSO partner organizational effectiveness and efficiency MADANI provides TA to 32 target CSOs to improve their skills in accountability and tolerance programming (effectiveness) and build their capacities in planning and budgeting activities (efficiency). MADANI geared its Y1 TA in accordance with the Y1 OPI results (Figures 3 and 4), which identified 75 percent of the CSOs as nascent. Organizations at this low capacity level need foundational organizational capacity development to strengthen their internal systems and their capacity to maintain the systems. MADANI’s TA therefore aimed to help CSOs meet minimum reporting and accountability standards and provide basic training on organizational development, including standard topics such as financial management, human resource management, and strategic planning. By Y2, 94 percent of target CSOs had reached the emergent level, and most CSOs had completed or revised their SOPs in financial management, procurement, human resource management, five-year 9 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV strategic plans, and MEL plans (all of which are USAID requirements for receiving a grant). OPI score improvements varied across CSOs based on several factors, including current capacity, experience, motivation and leadership, and familiarity with thematic issues. MADANI’s TA in FY 2022 focused on continued implementation of SOPs; strategic and performance management; CSO advocacy, stakeholder engagement, and community mobilization; communication strategy; resource mobilization; and support in implementing local solution modeling. Figure 3. Year 1 and Year 2 OPI Level of Organizational Maturity 24 1 6 30 2 1 Year 1 Year 2 Nascent Emergent Mature Source: MADANI Year 1 OPI Results Report and MADANI Year 2 OPI Results Report. Figure 4. Year 1 and Year 2 OPI Average Scores by Domain Source: MADANI Year 1 OPI Results Report and MADANI Year 2 OPI Results Report. 1.36 1.33 1.64 1.42 1.36 2.29 2.18 2.23 2.10 2.16 Effectiveness Efficiency Relevance Sustainability Resilience Year 1 Year 2 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 10 The evaluation team interviewed CSOs and MADANI staff to gain insights on the improvements in organizational capacity captured by the OPI. CSOs valued the OPI process, stating that it had led to good discussions among staff on the importance of learning and organizational development. CSOs also valued the TA provided by MADANI to develop and/or update their SOPs and stated that it is unusual for donors to invest in their organizational development. Instead, most CSOs have had to rely on ad hoc projects from donors with varying modalities and targets, leaving them with no clear vision or mission and without well-defined targets of their own. Nearly every CSO interviewed expressed that this was one of the most useful TA activities and that completing or improving these documents may make them more compliant and eligible for potential funding from various sources. Organizational capacity has also improved at the outcome level. Some CSOs claimed that because of the TA on SOPs, their organizational structure is better defined, the number of permanent staff has risen, and working hours and leave permits are more regulated. Two CSOs stated that having an annual work plan, SOPs, and a strategic plan made them eligible to receive funding from a new source. Also, along with training in administration, MADANI gave training on procuring goods and services, a good practice in fostering a reciprocal environment between government and CSOs as well as building sustainability. Individual CSO staff reported positive results from the overall TA on organizational capacity development, including increased knowledge in a range of topics (e.g., budgeting, financial management, reporting, marketing content development, MEL) and experience using organizational development tools, collaborative governance approaches, and advocacy tools. They also reported exposure to new themes (e.g., gender equality and social inclusion [GESI, interfaith tolerance, maternal and newborn health, waste management); expansion of their networks; and greater confidence in facilitation, community organizing, and engagement with stakeholders. Mainstreaming cross-sectoral collaborations among CSOs has facilitated peer group learning to boost experience and knowledge in advocacy practices (such as evidence-based policy formation and policy briefs). Also, the gap analysis conducted by the national training providers has helped map out CSO-related issues or goals, optimizing performance by providing teaching and learning materials that the LP CSOs needed. For instance, one of the providers (ICW) implemented the gap analysis based on LP specializations and interests and found that some LPs already had advocacy programs and that the LPs were interested mostly in public service issues. ICW aimed to facilitate the LPs’ introduction to anti-corruption issues and develop the most suitable curriculum based on LP levels and interests. The mapping process also helped identify strategic partners. Findings also indicate several areas for improvement and challenges to overcome in MADANI’s remaining time. For example, the standardized basic training program may not have been appropriate for all CSOs given their varying levels of organizational capacity. Some CSOs have been around for a long time and are part of larger networks and national institutions that have established administrative processes and requirements. Other CSOs were established as grassroot networks and movements, so the OPI could not measure their organizational capacity and the TA did not always match their needs. One interviewee from a grassroots organization explained, “Some aspects of the TA are useful such as capturing lessons learned from previous experiences to inform five-year strategic plans; however, MADANI’s administrative requirements and complicated processes can become a burden and may diminish our existing voluntary base.” In addition, most CSOs’ thematic issues differ from those selected and driven by MADANI, and the CSOs may not have chosen the targets or had experience with MADANI’s programs. For example, CSOs focused on women’s rights may lack experience in religious tolerance. Another CSO lacking 11 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV experience in environmental issues needed to learn to address waste management. Although the program approach thus exposed many CSOs to new themes, a shift to a more customized and iterative process could help identify CSOs’ specific capacity-building and organizational needs. Interviewees were concerned that without a customized and specific understanding of each CSO’s capacity-building needs, it would be hard to prepare CSOs for when MADANI ends, a challenge that might be exacerbated by the CSOs’ dependence on MADANI for financial resources. Training delivery was also hampered by coordination difficulty and corrigible challenges surrounding the quality among a multitude of service providers. In addition, the COVID-19 pandemic shifted all trainings online, which resulted in less interactive and engaging sessions. The online trainings were not recorded for viewing at alternate times and could not be shared. Nearly every interviewee raised concerns about the irregular training schedule and impersonal online format. Interviewees also expressed feeling challenged by the competing time requirements to complete deliverables from their MADANI work plans while working on their own organizational activities and professional responsibilities. These factors made it difficult for CSO staff to learn any topics in￾depth, practice and apply the knowledge to their work, and transfer the knowledge to their colleagues. One interviewee stated, “The CSOs are participating in the training, but the training is not necessarily resulting in improved capacity.” Excerpts from interviews with a range of stakeholders reinforce this finding: “Although the TA is needed for organization, the TA was too intense, and overlapping so we were not able to absorb all the training and considered it ineffective since it was conducted online. The knowledge was also on the surface and none of the TA could be optimized due to conflict schedules. There is a clash between our other activities besides MADANI. “The goal is clear to strengthen the local CSOs. But, in practice They may receive two trainings or support on the same general topic, but with different processes. The CSOs are like a glass of water that is overflowing.” In addition, CSOs interviewed stated they needed more financial resources to cover staff, overhead, and activity costs than what MADANI expected and provided. For instance, for many CSOs, MADANI’s budget covers only 30 percent of three staff salaries. But because of the workload, the CSOs are dedicating more than 100 percent of their time on MADANI work. Also, the activity budgets cover transport costs and snacks for only a small number of participants, so if more people attend, the CSOs must cover the costs themselves. Thus, CSOs often had to bring in external support, including volunteers and family members, to carry out the required work. They also relied heavily on Learning Forum members, who are expected to support the target CSOs in implementing their MADANI work plans but do not receive any direct funding and receive only limited TA to build their own capacity. Although many of the Learning Forum members valued the networking opportunity this provided, some Learning Forum members felt they were there to help the targeted CSOs achieve their own objectives and not necessarily those shared by the Learning Forum. One interviewee described this one-way collaboration as follows: “If a Learning Forum member or the entire Learning Forum wants to focus on a non-MADANI theme or activity, then the target CSO does not have resources to support…we have to think how we can all benefit.” Furthermore, the findings suggest that more time is required to determine whether CSOs could draft or revise their administrative documents without external support. MADANI staff, consultants, and volunteers spent significant time reviewing and revising documents for the CSOs and performing USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 12 technical and operational functions. Many CSOs stated that they lack adequate human and financial resources to implement their SOPs, strategic plans, and MEL plans on their own and that they lack a budget to hire the needed staff. One interviewee stated, “They have the documents, but they don’t have a budget to conduct the activities based on their strategic plan and work plan.” Also, the field coordinators played a key role in open communication with the LG because of their strong networks in the region. As stated by a Learning Forum interviewee, "The selection of the field coordinator really impacted MADANI's project implementation, because he already has a strong network and mutual trust with LG and local CSOs.” However, as another interviewee stated: "When the time for this MADANI Program has ended for this CSO, what will the plan be for the team and what is the management going to do? That’s what I’m worried about. Making sure that after 2024 they will be able to survive is tough homework.” Most interviewees stated that the CSOs needed more support and practice in applying and transferring knowledge and skills. One interviewee reported, “We want to transfer our knowledge that we gain from MADANI… but we don’t have a budget to transfer it.” Some CSOs believe that knowledge transfer can proceed through socialization and dissemination, but MADANI’s program has not allocated budget for such events, and CSOs do not have an internal organizational budget set aside expressly for knowledge transfer initiatives. Also, the large time and staff commitment needed to complete MADANI milestones and deliverables has made it more difficult for CSOs to apply the learning from capacity-building activities to their own needs. Many CSOs stated that they have not had the time or financial resources to focus on their core competencies and activities because of MADANI work plan requirements. Even though MADANI offers TA from technical service providers to LPs in the form of coaching and mentorship, some LPs still find it difficult to follow MADANI's standards, organized reports, and SOPs because of lack of available human resources. Approaches and tools for building organizational capacity MADANI implements a multipronged approach and various tools to improve CSO organizational capacity. The evaluation assessed these approaches’ and tools’ effectiveness. District support partners (DSPs): One effective training approach was pairing CSOs with a DSP that has contextual knowledge and provides more tailored, consistent, and intensive support. As MADANI expands this approach, the challenge will be to find qualified and committed DSPs who are willing and able to help build the capacity of emergent CSOs. OPI: Most CSOs appreciated the OPI as a joint organizational self-reflection tool. However, given the nascent level of most CSOs, MADANI did not apply the Organizational Capacity Development Action Plans. Using this tool now would enable MADANI to create a customized learning plan for each CSO and shift its organizational capacity-building activities in a more tailored direction. MADANI direct TA: The provincial and district field coordinators play an enormous role in MADANI’s approach to capacity building and TA provision. The position’s effectiveness varies by district. Many interviewees stated that it is hard to imagine MADANI working without the field coordinators. At the same time, interviewees expressed concern that field coordinators’ intensive role in managing the CSOs and Learning Forums might create CSO dependence on MADANI and inhibit the sustainability of networks and capacity development. One interviewee said, “The field coordinator helps us with everything, including our own internal program. The field coordinator has been going above and beyond…because [the field coordinator] is doing so much, we will not be able to solve our own problems….we are solely dependent on the field coordinator.” Collaborative governance approach: Stakeholders considered MADANI’s collaborative governance approach and the application of various tools beneficial to many CSOs and Learning Forums. 13 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV However, interviewees stated that the CSOs and Learning Forums need more than one or two opportunities to practice, reflect, and learn before they can be expected to duplicate the process on their own. Also, the collaborative governance approach did not directly reflect many of the CSO and Learning Forum members’ missions, goals, and objectives, so it is unclear whether the capacities built through these experiences will increase their organizational efficiency and effectiveness beyond implementation of their MADANI work plans. Trainings: The online TA from a broad range of service providers was not seen as an efficient or effective way to build CSO capacity. In addition, the cascade approach was not effective because the trainings and workshops were not designed as training-of-trainers courses. Many CSO staff who attended the trainings stated that they did not understand the material enough to effectively transfer the knowledge to their colleagues and the Learning Forum members. They also did not receive recordings or worksheets as resources to help train their colleagues. However, the previous MADANI’s assessments and feedback from the first year of TA show that classroom training (online or in person) is most effective when combined with personal mentoring and coaching. Therefore, it is necessary to adjust the upcoming TA based on these combination methods. Given the dependence on online TA due to COVID-19 and other budget and logistical constraints, stakeholders did not understand why the trainings were not taped and shared and why additional learning tools were not provided online. According to the FY 2019 Q2 Progress Report, the Innovation and Knowledge Hub (iHub) was supposed to be “an online, dynamic application and knowledge management system that will enable users to use and download tools in the form of gamified capacity assessment, tip sheets, training modules, simple reading materials, and performance data, as well as a directory of technical assistance resources and consultants and a tool for interactive question-and-answer sessions and a chat room. The iHub will be the core application used by field coordinators and national office staff in delivering technical assistance to local partners starting in early 2020.” However, according to many interviewees, the iHub was not effective in serving this function. One interviewee stated, iHUB “is a complete failure….MADANI spent a lot of time and money on iHUB…but it didn’t work.” In addition, as reported in the FY 2021 Annual Report, the iHub SimpulMADANI has not developed as quickly as expected and utilization is lagging. By the end of Q1, the iHub had 479 registered users and 331 participants for four webinars on various civil society issues, and the web-based application had garnered 14,144 page views from October to December 2021. However, only 41 active users accessed more than one iHub feature in Q1, far below the target of 175. To boost the IHub’s usage rate, MADANI started transferring the iHub grant to ATMA-Connect by hiring a consultant.6 Mainstreaming GESI, anti-corruption, and tolerance in organizational capacity development Gender equality and social inclusion (GESI): MADANI has a GESI task force within the MADANI national office that meets every two weeks, three national consultants to help mainstream GESI within CSOs, and a national service provider contracted to conduct a five-module workshop for CSOs on women’s leadership and empowerment. During MADANI’s first and second year, the consultants provided online trainings on a range of GESI-related topics. The consultants were also tasked with reviewing the CSOs’ SOPs, strategic plans, and MEL plans to ensure GESI was integrated into the documents. According to interviewees, after many revisions, all the SOPs are now GESI sensitive. Interviewees were concerned that although the GESI training provided the CSOs with information, it was not enough to equip them with the knowledge and skills to integrate GESI into their SOPs on their own or to apply what is written in them. 6 MADANI Quarterly Performance Report Q1 FY 2022, p. 25. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 14 Interviewees also pointed out that even though several women-led CSOs were involved in MADANI, only one of the CSOs was a disability organization and very few of the CSOs knew anything about disabilities. According to interviewees, some CSOs have made facilities more GESI sensitive by providing ramps, handrails, and lactation rooms in their offices. However, many of the CSOs lack the financial resources to buy equipment and make such renovations on their own. According to the SOPs, 30 percent of staff should be women. Several CSOs have promoted and hired female staff and volunteers. Some CSOs have incorporated diverse new members (e.g., people with disabilities, women, and youth) into learning forums. In addition, some CSOs are starting to incorporate GESI considerations when selecting partners or during a procurement process. According to interviewees, many of the CSOs are enthusiastic to learn about GESI and to unlearn old ideas. For example, the older generation did not often see youth as effective actors for advocacy. But through the training, the older generation is starting to see a real role for young people in their CSOs. One interviewee summarized as follows: “We had to integrate GESI into the five documents….But is this something that the CSOs are internalizing? Some have provided space (ramp, handrail, lactation room) in the CSOs’ offices. Also, 30 percent of staff should be women….However, all the CSOs can’t change this. It depends on their financial situation. But, in terms of activities, it is just mainstreaming the outputs. It is just fulfilling the indicator. It is not yet being fully implemented. It is a requirement from MADANI.” Anti-Corruption: Since none of the LGs chose anti-corruption as a thematic area, MADANI contracted a national service provider to provide TA on this theme. The service provider was also responsible for reviewing and identifying gaps in each CSO’s SOPs and ethics codes. The TA included a combination of online training in the form of classes, meetings, and discussion forums with some limited off-line coaching and mentoring. In general, TA in anti-corruption has yielded positive, albeit still far from optimal, results. On a post-training evaluation conducted by the service provider, participants scored 68 percent on understanding of anti-corruption issues. However, even though the training has enriched the participants’ knowledge, respondents stated that there is a long way to go for the CSOs to understand and internalize anti-corruption issues properly and apply it to their organizations and advocacy activities. Tolerance: Since only two LGs chose the thematic area of tolerance, MADANI contracted another national service provider to help build CSO capacity on this theme and mainstream tolerance throughout the program. The national service provider also reviewed each CSO’s SOPs, strategic plan, and MEL plan to ensure that tolerance was integrated throughout the documents. The TA included three workshops—one online and two in person—and mentoring on topics such as strengthening diversity, strengthening the narrative in social media campaigns, and advocacy. Many participants appreciated the TA. Interviewees also reported it as challenging—many CSOs lacked knowledge of the subject and did not see its relevance in relation to their MADANI work plans. Integrating diversity and tolerance into their organizations also requires an in-depth understanding of the context. One interviewee stated, “Mainstreaming tolerance requires identifying the main actors at the local level. This is difficult because it requires a lot of networking and connecting with all the stakeholders.” Interviewees shared several instances of positive change in diversity and inclusion. For example, several CSOs and Learning Forums have involved participants with diverse backgrounds (women, youth, persons with disabilities) in their activities and meetings. In one district, the CSO supported 15 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV an emergency humanitarian response for flood victims from a minority group. In some districts the target CSOs have started networks or joined established networks of CSOs focused on diversity. In another district, an interfaith forum was invited to participate in discussions on diversity and inclusion. The national service providers, consultants, CSOs, and Learning Forums identified challenges and constraints that limited the effectiveness of mainstreaming GESI, anti-corruption, and tolerance throughout the program, including the following: ● Budget constraints limited national service providers’ ability to provide adequate coaching, mentoring, and post-training follow-up plans. As a result, it was difficult for them and for consultants to know whether participants had applied the knowledge and principles after the training and what additional support was needed. ● CSOs’ human and financial resource constraints hindered their ability to fully engage in the TA and make the necessary adaptations to their CSOs according to their revised SOPs. ● The CSOs drafted their SOPs and other documents before receiving the specialized TA. Also, many CSO staff who attended the TA did not fully understand or internalize the information, so it was difficult for the CSOs to revise the documents themselves. As a result, it took a lot of time for the national service providers and consultants to review and provide input into all 32 CSOs’ SOPs, strategic plans, and MEL plans. An interviewee stated, “I had to really concentrate on editing all the CSOs’ documents. Sometimes, I would say… I need a break!... and the field coordinator would demand the consultant’s support and endorsement. There was a lot of pressure…to complete the documents.” As a result, interviewees were concerned that the CSOs were not aware of or did not understand what was in the documents enough to apply the information consistently. As stated by an interviewee, “It cannot just be written into a document to complete a deliverable.” ● MADANI’s approach to GESI, anti-corruption, and tolerance mainstreaming did not adequately integrate the themes into other capacity-building activities, which made it difficult for the national service providers and consultants to situate the specialized TA into the bigger picture of organizational capacity development. This also made it difficult for the CSO staff to see the direct relevance and importance of the specialized TA. Several respondents stated that some participants opted out of the TA because it was not directly related to the milestones in their work plans. As stated by an interviewee, “The CSOs only do the activities according to their performance report based on milestones per their contract. It is more of a checklist.” ● The TA on tolerance included only one or two individuals from each CSO. The hope is that they will transfer the knowledge, awareness-raising process, and individual sensitivity to their colleagues; however, it was hard for the national service provider to determine the extent to which this is being done. Coping with the COVID-19 pandemic The COVID-19 pandemic worsened an already challenging environment for building CSO capacity to advocate for LG accountability and community tolerance. MADANI and its CSO partners found ways to cope with the pandemic, but some challenges were difficult to overcome. For example, the MADANI program design and initial setup were completed before the pandemic, so MADANI had to adapt to remote and online engagement at nearly every level of communication and programming. Although some tasks were already designed for online implementation, others, including on-the-job training, peer-to-peer exchanges, and personal mentoring, were not. MADANI adapted some activities and reallocated some resources to help respond to the pandemic. For example, some CSOs helped the LG distribute personal protective equipment and basic supplies, others supported health campaigning and awareness-raising efforts, and several became active USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 16 members of district COVID-19 task forces. MADANI also provided two grants to CSOs to address the pandemic: one to AtmaGo on community awareness raising and one to INFID to help strengthen government accountability in COVID-19 services.7 Interviewees saw all these initiatives as positive. However, despite the delays and challenges, USAID and MADANI did not adapt their expectations in several key areas, including the program’s timeframe, spending, and target indicators. As stated in the FY 2020 Q3 Report, “In the current situation (COVID-19), MADANI needs to ensure that the program for all partners can be implemented according to the schedule and spending plan.” Barriers and opportunities for improving CSO organizational capacity By applying a systems thinking approach anchored in MADANI’s TOC and assumptions, the evaluation found that several interrelated factors have inhibited MADANI’s ability to fully achieve its capacity-building objective. Some of these barriers were within USAID’s and MADANI’s control and sphere of influence; others were not. COVID-19 pandemic: As described above, the COVID-19 pandemic worsened an already challenging environment. The pandemic also caused delays in implementation, hindered relationship building between CSOs and communities, challenged the national service providers to deliver tailored and engaging online TA and coaching, obstructed internal MADANI communication between Jakarta and the field, and placed a lot of responsibility and pressure on the field coordinators. It also made it difficult for MADANI and USAID staff in Jakarta to observe the reality on the ground. “The most challenging part has been shifting to online technical assistance and training due to COVID. It also made it difficult to tailor the training as much as we had hoped. We have some older and stronger CSOs and some that are new, but it is hard to design online trainings that are not generic. It is also hard to maintain focus and absorption capacity is a challenge.” “When we designed the technical proposal, we did not expect COVID. We had to transition from in-person to remote TA. Also, the effectiveness of using online training and TA is less than 40 percent in terms of absorbing the input and translating it into action. It’s very hard to improve the capacity, advocacy, strategic communication—you really need in-person engagement.” “The field staff didn’t have the culture of using virtual meetings. They just learned how to use the technology. Many of the participants may not have taken the online training seriously. It is also hard for participants to express their ideas using the virtual whiteboards. Everything took more time.” Unexpected majority of nascent CSOs: MADANI is one of the first and only programs that focus on building CSO capacity at the very grassroots level in Indonesia. Nearly every interviewee applauded MADANI’s ambition, courage, and vision to take on this challenging task. An interviewee stated, “As far as I know, this is the first international program focused specifically on CSO development. There are so many donor agencies that hope that all CSOs and NGOs should be independent and sustainable. But most of them do not care about the development of the organization.” 7 See FY2020 Q3 Progress Report. 17 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV However, the MADANI program design and technical approaches assumed a higher level of local CSO capacity and national and provincial-level TA service providers and underestimated CSOs’ resource constraints (time, human, and financial). “We assumed that [CSOs] had a minimum quality requirement [and we] assumed that they had the capacity to organize other CSOs and stakeholders. But when we started, we realized that more than half did not have the minimum capacity, so it took more time and resources. “We worked with national or provincial subcontractors to provide the [TA] services. Over 60 percent were not ready to deliver the TA as expected in all 32 districts. They did not have the human resources.” As a result, it was challenging for MADANI to find qualified CSOs that met the minimum standards to receive a USAID grant or service providers with the appropriate capacity and expertise to provide remote training and TA to 32 CSOs. This challenge delayed implementation and changed several aspects of the program. Also, it put significant pressure on the CSOs and MADANI field coordinators to achieve the contractual milestones in the established timeframe so that MADANI could in turn achieve its own indicator targets. “A challenge for capacity-building programs is to prove increased capacity especially with a RF with 26 indicators that require results….This means that the CSOs must do a lot of activities at the same time rather than building capacity first and adding on activities. The RF may drive the activities rather than the activities being owned by the CSO and LG.” After two years, MADANI recognized that its expectations were too high and decided to reduce the number of districts that it would cover from 64 to 32 districts, as stated in the FY 2021 Q3 Progress Report MADANI continues to shift and adapt its organizational capacity development strategy for the remainder of the project lifetime. The evaluation identified several operational challenges for staff, including a lack of objective monitoring and measurement of organization effectiveness and efficiency beyond the OPI self￾assessment, administrative burdens on MADANI staff, and heavy reliance on observations of field coordinators, who have a vested interest in sharing positive results. CONCLUSIONS Evidence shows organizational improvement. CSOs have completed or revised SOPs and staff have gained knowledge, skills, and experience using organizational development tools. MADANI’s design to incorporate field coordinators as mentors in YI, facilitators in Y2, and support in Y3 has improved CSOs’ adaptation to learning and organizational capacities, including delivering objectives and building relationships. Online training delivery and time and resource constraints have presented challenges. It is still too early to determine the sustainability and reach of MADANI’s capacity-building efforts as the CSOs are new and capacity building takes time. To properly gauge results, more time should be allocated for MADANI to incorporate adaptive strategies. However, the CSOs’ internal systems are now operating efficiently, and MADANI can concentrate its capacity-building efforts on improving scale and quality of service delivery and on outcomes related to targeted beneficiaries, an important step toward improved self-reliance. MADANI’s TA may prove a helpful resource for long-term sustainability. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 18 EQ2: HAS MADANI STRENGTHENED THE LOCAL GOVERNMENTS’ ACCOUNTABILITY AND PROMOTED TOLERANCE? IF SO, HOW? IF NOT, WHY NOT? FINDINGS Achievements in CSO advocacy for LG accountability MADANI focused on five accountability-related thematic areas: anti-corruption, frontline service delivery, natural resource governance, maternal and neonatal health, and village fund transparency. MADANI has enabled target CSOs to advocate for LG accountability. In each district, MADANI facilitates a multistakeholder Learning Forum that implements an action plan based on a selected thematic area related to accountability or communal tolerance. Through a co-creation process, target CSOs and Learning Forum members develop local solutions for advocacy and accountability that address community-identified challenges and prepare the action plan with demand-side interventions to ensure supply-side reforms and provide concrete solutions that the LG can consider and adopt. Thematic action plans also include pilot projects in which the CSO interacts with various parties on the enabling environment and on social accountability. Through coordination and joint workshops with key LG representatives, facilitated by MADANI field coordinators, target CSOs have become better connected with LGs. LGs have demonstrated responsiveness to CSO advocacy and are involving CSOs in policymaking processes. MADANI has also established Learning Forums to provide a multistakeholder space where CSOs, LGs, the private sector, and other community leaders can come together and implement MADANI’s collaborative governance eight-step approach to identify problems, map collaborative partnerships, develop action plans for local solutions, use accountability tools, advocate for accountability and improvement in thematic service delivery, and share the learning for replication and/or scale-up. However, as reported by MADANI, LG participation in Learning Forum meetings remains low, averaging between 10 and 15 percent.8 Another MADANI activity, the acceleration grant program, provides additional funding to high￾performing CSOs to test or implement an innovative approach or activity related to the district’s thematic area to promote better accountability or improve communal tolerance. Under these grants, CSOs and LGs work together more frequently to strengthen their capability, legitimacy, and sustainability. For instance, Kelurahan Mandiri ("Self-Reliant Neighborhood") enhanced CSO technical and advocacy capacity as part of local economic development. To date, as stated in the Q2FY2022 report, seven CSOs have received an acceleration grant. However, time and financial constraints and heavy workloads made it difficult for CSOs to implement these grants. One interviewee stated, “When we were about to be given the acceleration grant, we almost declined. We told MADANI that we weren’t ready. If it is only for one year it is too much for us in relation to the budget.” Collaboration and relationship building between LGs and CSOs are growing, boosting CSO advocacy for government accountability, as evidenced by the following: Changes in knowledge and experience: Some CSO staff now have experience conducting basic socioeconomic and stakeholder analysis and applying social accountability tools. As a result, they are more aware of the local issues related to their advocacy themes. Some staff have also used and interpreted data to support evidence-based collaborative governance and advocacy work and 8 MADANI FY2022 Q1 Progress Report. 19 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV facilitated dialogue with a range of stakeholders to promote mutually beneficial changes. Some staff also have experience taking on leadership roles in multistakeholder forums. “I think [the target CSOs and Learning Forums] have been performing well compared to the other CSOs [not working with MADANI] ... I see that they were able to explore problems, collect valid data, coordinate with the LG, and they were active in involving the community.” Changes in relationships and processes: Some CSO staff have had the opportunity to build relationships and collaborate with other CSOs, LGs, and communities. In addition, some LG officials stated having a better understanding of working with CSOs and finding common solutions on specific themes. “The challenge was the lack of CSO knowledge and the minimum role that they played... Now, I see them differently… they have been found very effective in helping us [the LG] in our work.” “Before MADANI the LG did not have an imagination regarding collaboration with CSOs. Also, LG can’t identify what a CSO ideally looks like. After MADANI, the LG knows the meaning of collaborative governance and how to involve and work with civil society.” In some districts, MADANI’s collaborative governance approach resulted in the establishment or improvement of processes whereby information and demand generated by communities (e.g., citizen feedback surveys) was communicated up to LGs for action. LG stakeholders attribute this to MADANI empowering target CSOs to “work hand-in-hand with the LGs to find solutions instead of simply criticizing [the LG] as other CSOs tend to do.” As a result, some CSOs were included in village planning and budgeting, some CSOs provided input into annual development plans for stunting, and in several districts the CSOs have become the LG’s partner in handling community maternal and child health issues. In addition, many of the collaborative governance processes have resulted in policy briefs, decrees, regulations, agreements, and other outputs. For instance, some of the tools of collaborative government, such as community scorecards, have fostered community participation on programs implemented in the region. Additionally, support for CSO involvement in advocacy through MADANI deliverables such as policy briefs could trigger a sociopolitical environment of accountability among LGs, CSOs, and their communities. According to the FY 2022 Q2 Progress Report, 44 policy recommendations have been formally adapted or adopted. Changes in thematic service delivery: In some districts, the collaborative governance process has resulted in tangible outcomes at the village level, including increased wages for cadres, availability of medicine and equipment, improved cleanliness of health centers, shorter wait times, friendlier staff, increased security, more health workers, improved access to female contraception, increased access to national health insurance, distribution of fresh vegetables to high-risk women, and improved accessibility (e.g., ramps and handrails) to public facilities. Most interviewees stated that these changes in knowledge and experience, relationships and processes, and thematic service delivery are crucial for translating advocacy into government accountability. They are helpful starting points for addressing larger, more complex, and longer-term sociopolitical, cultural, and economic issues that are, however, often limited by LG funding constraints. “Some of the problems [at the health center] were resolved but it depends on the funding from the LG. MADANI facilitated this discussion, but the LG does not have the budget to make the changes.” USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 20 “When we talk about inclusive education, this is a long journey. It cannot be resolved in a year. The cause of the challenge needs to involve many sectors. This is a huge effort. Achievements in CSO advocacy for tolerance in their communities MADANI focused on two tolerance-related thematic areas: community tolerance and diversity among youth. Two LGs chose this thematic area and each implemented it differently. Both CSOs involved in the tolerance activities reported that their knowledge and capacity grew, and they shared several anecdotal results. For example, in one district the youth worked together to protect religious sites and to respond to the bombing of a church. In another district, the CSO was introduced to topics related to interfaith diversity and tolerance, which has expanded their knowledge and relationships with other stakeholders more familiar with the topic. Social accountability tools and mechanisms MADANI has facilitated the use of several social accountability tools and mechanisms to enable target CSOs to increase the demand for public service improvements and hold LGs accountable. The evaluation assessed the effectiveness of the various tools and mechanisms. Community scorecards: The most used tool (25 districts) is the community scorecard to collect feedback from community members on selected public services and to engage more effectively with the LG for improvements. These scorecards provide information on community concerns to collaboratively gather solutions and generate policy recommendations that are then presented to the LG. For example, in one district the LG had never conducted a survey on service satisfaction. The target CSO and MADANI supported the LG in launching a community scorecard targeting consumers of village funds and solicited feedback. The community scorecards are more effective than other traditional approaches such as musrenbang, which tend to be less representative. As a result, the LG has allotted a self-management budget for the CSO to survey villages and gather feedback on village fund transparency. In another district, the LG will fund replication of the community scorecards in 25 wards. Complaint handling system: In one district, the LG health office piloted a complaint handling system, where through a social media platform, Atma Connect, complaints related to basic service delivery at a community health center (Puskesmas) are forwarded directly to the district’s health office and Puskesmas for immediate response through the new system. This mechanism has received positive attention from the LG, which is proposing to scale it up to all 38 Puskesmas in the district. If successful, the process will be replicated in other MADANI districts. Social audits: The social audits were conducted as a follow-up to the thematic survey to track the development process, commencing with planning and continuing with monitoring and evaluation at the urban village level. The social audit forms one of the eight stages in the collaborative governance process, which also includes the formation of the team, debriefing, analysis, exploring analysis results, recommendations, and drafting a policy brief. In two districts, public information disclosures through social audits are increasing transparency. CSOs then applied a community approach to oversee and monitor the LG budget. For instance, CSOs in Solo used a community approach to oversee and monitor the LG budget to investigate poverty in two urban villages. The LP was invited to musrembang in the second year to participate in the strategic plan discussion and team formation come from the representatives of LPMK (Urban Village Community Empowerment Institution), Faskel (Urban Village Facilitator), Karang Taruna, and PKK (Family Welfare Empowerment). In Solo, the target CSO includes people with disabilities in the social audit process as part of youth social inclusion. Several shortcomings, such as the relatively small budget and actors’ capacity, have hindered the social audit process. 21 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Community-based forums and task forces: Stakeholders reported renewed energy and high engagement from service delivery workers and community members through community-based forums and newly created task forces. The challenge is to maintain the continued collaboration and participation of multiple stakeholders without the direct facilitation and support from MADANI’s field coordinators and service providers. Mainstreaming GESI into accountability and tolerance work MADANI aimed to mainstream GESI into accountability and tolerance work in various ways. For example, MADANI hired consultants to provide online trainings and review all five of the LPs’ primary documents to ensure their GESI sensitivity. The consultants stated that after many revisions all the SOPs are GESI sensitive. However, it is not possible to know whether the LPs are internalizing or applying what is in the documents. For instance, an interviewee stated that “sometimes the LPs just want us to help us write a paragraph about GESI so they can complete the deliverable on time. But, we say, no. You must integrate it. It takes time to absorb and integrate GESI. It can’t just be written in a document.” Another interviewee stated: “Some of the LPs were beginning to know or understand GESI. But only one out of the 32 LPs were disability organizations. Very few even knew about disabilities. So before completing the SOPs, they need enough knowledge and skills. MADANI collaborates with USAID HARMONI and a local partner to provide training and learning tools to CSOs on strengthening tolerance and social inclusion, including addressing unconscious biases and micro-aggressions. MADANI also planned various events with CSOs to strengthen their ability to network and form alliances; promote policies; and advocate for GESI, anti￾corruption, and tolerance. However, as most CSOs did not focus on these thematic areas, their commitment to follow through with MADANI’s mainstreaming expectations often did not go beyond completing the TA or milestone.” Challenges to improving CSO capacity to strengthen accountability and promote tolerance Several challenges hinder MADANI’s ability to improve CSO capacity to strengthen accountability and promote tolerance, only some of which are within USAID’s and MADANI’s control and sphere of influence. Uncertainty over continued collaboration after MADANI ends: The achievements and progress described above have not been equal across all districts. Some CSOs lack adequate capacity, resources, time, and commitment to fully participate or benefit from MADANI’s collaborative governance approach. In addition, some of the achievements have occurred with significant support from volunteers, MADANI staff, DSPs, service providers, and consultants. For example, as an interviewee stated: “The timeframe is too short for the CSOs to learn how to do [LG] accountability. So we are just doing it for them. But will they be able to do it without us?….In reality, some CSOs drafted the policy brief and then they did the social accountability. It should have been reversed.” Also, although interviewees stated that the CSOs’ affiliation with MADANI gave them access to various public and private stakeholders, most of them doubted that the LG would respond to them without the backing of MADANI, the Ministry of Law and Human Rights, and the provisional or district secretary. Many interviewees also doubted that the LG would respond to CSOs if it had not chosen the advocacy theme and did not consider it in its interest. Further, selected thematic areas may not have directly related to CSOs' advocacy focus. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 22 In addition, many target CSOs are service providers and not advocacy organizations, so they are not staffed or resourced to replicate this advocacy approach. One interviewee stated, “There are two types of CSOs—some are advocacy, and some are service providers. Usually, you don’t provide both. If you do service provision it can dilute the advocacy.” In some cases, this has led to inadequate programming implementation and a heavy reliance on MADANI and external support. It is thus too early to determine the sustainability of these advocacy efforts without MADANI directive and support. Underlying assumptions in the theory of change: MADANI’s development hypothesis assumes that the LG (supply side) is willing and able to respond to the demand created by the target CSOs and Learning Forum’s collaborative governance work. As stated by an interviewee: “MADANI is a demand-side project. If there is no response from the LG, then nothing ….But MADANI is not an LG project, also not a sectoral project. We work on the demand side with the hope that through the demand and pressure, the LG will act.” However, the causal pathways necessary for linking MADANI’s three IRs (capacity, legitimacy, and sustainability) with the program goal and objective may require additional factors (e.g., leadership, stronger networks, longer timeframe, and enabling environment), many of which are outside MADANI’s scope and sphere of influence. For example, the FY 2021 Q2 Report states: “Through its SCS Global program, FHI360 has found that increased organizational and technical capacity for local CSOs does not necessarily translate into CSOs’ ability to advocate for accountability….MADANI’s theory of change—that CSO capacity, legitimacy, and sustainability will lead to improved advocacy—might need to be supplemented with other preconditions for change, such as high-quality leadership.” Although this shows recognition from USAID and MADANI that assumptions may not hold and that gaps in the theory of change have been identified, no significant adjustments have been made to the TOC, results framework, or indicators. LG resource constraints: LG response, which is an underlying assumption in MADANI’s TOC, is also affected by resource constraints. Several LG officials stated that they lacked the human and financial resources to implement the agreements and policies developed through the collaborative governance process. LG officials also stated that the requested changes did not fit into their planning cycles. One government official stated, “Our budget does not include our office working with the communities or CSOs. The budget is already locked so maybe in 2024….There are more urgent priorities that we need to address.” Most interviewees were skeptical of the LG’s responsiveness. Interviewees stated that they were doubtful that a public forum would be a strong enough incentive to ensure that the feedback provided to the LG is acted on, especially without MADANI’s backing. For instance, in some districts, even after the collaborative governance process was complete, the LG did not respond to the results despite their inclusion in the process. Several interviewees stated that whenever they advocated with the LG, the response would repeatedly about budget.” Several government officials were also apprehensive about the LG’s willingness and ability to follow through. As stated by an interviewee, “One event or survey (community scorecard) is not enough. It will not have any impact on the 10,000 citizens in the community. Also, the LG does not have a budget or the technical capacity to continue doing the community scorecard.” 23 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Several interviewees and government officials stated that the LG also has limited human resources and needs capacity building. One official stated, “We need an expert to give us knowledge. If we speak about budget, it will be limited….We need an expert from USAID to improve our competency.” Reinforcing several of these points, another interviewee stated: “The level of expectation regarding government accountability must be managed. Many contributing factors such as substantial investment in building the capacity of the LG, including investing in their human and financial resources, must be provided to push supply side engagement and to enable them to fulfill the promises made in the various agreements, policies, etc.” Term limits and staff turnover within the various government offices also pose a challenge to the implementation and sustainability of MADANI’s collaborative governance approach. For example, when a mayor’s office term ends, revised regulations might have implications for the budget allocation and for agreements, and the new mayor may not continue the work. Commitments and policies/decrees also depend for implementation on the person who passed them, so high turnover and rotations within the LG offices put implementation at risk. Political economy issues: CSOs have gained more knowledge of and exposure to accountability and political issues in their respective regions. However, several CSOs feel that LG accountability extends beyond advocating tangible areas of service delivery. Some small CSOs are not confident in their ability to address complicated reforms and LG accountability, considering LG politics, culture, and lack of human capital. This evaluation’s findings suggest that "supply-side" reforms in service delivery may result in some observable improvements but that efforts to promote such reforms may not have a measurable impact on the standard of governance, particularly at the highest levels. MADANI’s Working Paper 1 on collaborative governance reinforces this view, citing examples of CSO practices in several cities: “For example, in one city, the head of the local health office who championed service accountability reform was eventually charged with corruption. In another city, the previous mayor, who was also a strong supporter of CSO engagement with the local government, was similarly implicated in a graft case. Understanding this “missing link” between CSO empowerment and actual responsive and accountable government is an important part of future civil society activities.”9 Collaborative governance approach driven by outputs rather than process: Most stakeholders appreciate MADANI’s collaborative governance approach. At the same time, many interviewees reported that the strict interpretation of the approach and the need to complete milestones forced CSOs to circumvent the experiential learning process and resulted in siloed activities that did not always complement other steps as planned. Some interviewees reported that the CSOs, with support from MADANI staff, would have to do a collaborative governance activity before receiving the TA or that they drafted policy briefs before completing the collaborative governance process with the various stakeholders. Interviewees stated: “The CSOs have a specific work plan. In terms of CSO collaboration and growth they need more flexibility and creative methods rather than such a strict methodology and 9 Prio Sambodho – Collaborative Governance in Strengthening Accountability and Tolerance in Decentralized Indonesia – A MADANI Program Assessment. July 2020 (Working Paper 1). USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 24 work plan… The CSOs work with one track, they do not open up to doing work outside of their workplan.” “We use collaborative governance as our main approach, [but] each part is not coming together to complement each other….[For example] the Learning Forum is not feeding into the collaborative governance approach [as planned because in several districts] the LG and private sector do not participate in the Learning Forum.” “Everything is already arranged in the milestones. There is no flexibility….If the CSO does not learn from a TA how to complete a milestone, then there is no time for them to learn…the field coordinator comes in and does it for them, so they meet the milestone. The model of MADANI is for CSOs to learn and then encourage and train other CSOs to advocate for accountability.” Other interviewees reported that because most CSOs do not typically focus on advocating for government accountability or on their selected thematic areas, CSOs often needed a better understanding of the thematic area and the sociopolitical dynamics before engaging in advocacy work. This step was often rushed or skipped. MADANI’s Working Paper I states: “CSOs should be able to identify what type of political work is strategic and appropriate for them. CSOs can be politically informed and/or politically active. They can be politically informed and work on short-term pragmatic objectives based on the prevailing context. On the other hand, they can be politically active and focused on actual institutional changes that may lead to a more transformative outcome—such as policy reform….[A] decision such as this about the basic approach should be based on a full reading of the local context and the decisions of the local beneficiaries—who become the champions.” Because of inflexible timeframes, preselected themes, and limited capacity building, many CSOs and Learning Forums could not “identify what type of political work is strategic and appropriate for them” or do a “full reading of the local context and the decisions of the local beneficiaries.” Furthermore, pressure to show results against indicators may circumvent genuine CSO capacity building in the three IRs—the foundation of MADANI’s TOC. The emphasis on showing programmatic results by drafting policy briefs and using certain tools may oversimplify advocacy for LG accountability and community tolerance. Additionally, several CSOs and Learning Forums may not understand the associated challenges, time, and resources as capacity building in human resources is sometimes attended only by representatives from CSOs/LPs. Thus, not all CSO members benefit from technical assistance. The effectiveness of knowledge transfer has also not been tested. It is thus too early to determine whether MADANI’s approach will sustainably strengthen LG accountability and promote community tolerance. Simplification of strengthening government accountability: Given MADANI’s demand-side scope, CSO capacity to strengthen LG accountability is measured by the number of policy recommendations, through CSO policy briefs that LGs adopt or adapt. Policy implementation is beyond the control of MADANI and its CSO partners; it is a supply-side issue. As summarized by interviewees, “If the LG doesn’t do the changes what are you going to do? The process that we have taken thus far is limited to the implementation of an action plan.”; “We are confused, is the goal to complete the policy brief, or to build the relationship between the community, CSO, and LG?” 25 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV In addition, MADANI uses the terms collaborative governance and advocacy interchangeably without clearly defining either. For instance, the FY 2019 Q3 Progress Report stated that “provincial visits largely confirmed the MADANI development hypothesis on the importance of supporting partnerships between CSOs and local governments on shared priorities. Most provincial governments with whom the project met strongly support the MADANI objective of collaborative governance.” It is important to differentiate between collaborative governance and MADANI’s stated goal of strengthening local civil society’s capacity to advocate for government accountability and tolerance in their communities. For instance, the skills and confidence necessary for local CSOs to facilitate collaborative governance activities on topics of shared interest and priority may differ from the skills and confidence needed to advocate for LG accountability on more contentious themes that are not in the LG’s shared interest. This issue was reinforced by many interviewees: “We worry that if in the future we raise a different thematic issue, the LG may not be responsive. We also worry about the LG’s financial commitment to make the changes and continue working on other issues.”; “We must limit the expectation to a very specific area. We want to prove that CSOs are capable of advocating in a specific area. So, it may not necessarily transfer to other issues.” In addition, the focus on achieving an output or milestone meant that many LPs did not recognize that MADANI was preparing the CSOs to hold LGs accountable to their promises and very few LPs thought they were prepared to do so, even with more time and ongoing human and financial support from USAID and MADANI. This ambiguity in expectations has spilled over to the local level; communities and LGs in some districts now have expectations that the LPs and Learning Forums cannot meet. For example, the inclusive school pilot locations were very skeptical and concerned about holding the LG accountable to provide the support and services promised in the signed decrees. They were also concerned that the LG may provide small infrastructure improvements such as ramps for handicapped students and specially equipped bathrooms even though the major challenges for the headmasters and teachers were large classroom sizes, lack of qualified teachers trained to work with students with disabilities, lack of school counselors and psychologists who could diagnose and treat students with disabilities, educating the parents, changing social stigmas, and so forth. Additional excerpts from many interviewees highlight this dilemma: “For the thematic issues, this becomes a problem because there are budget constraints. The LPs want to work with the community, but they don’t have any financial resources to do this work. So we have to find a solution, so we try to get funding from the LG and the private sector.” “The LG said they don’t have money for a trash can. The LG said to approach the private sector. But the private sector does not have this on their agenda. So we faced a brick wall. The trouble is to fulfill all of these expectations.” “We want the success of this program to be felt by the community. This program…should not just be a program on paper. This is like planting a tree without any fruit.” “When there is an international program funded by a donor, the LG assumes that we will pay for everything. When we seek budget allocation from the LG, they always say that they can provide a resource person or a room for meetings. But it is hard to get them to assume their responsibility. It should be the LG’s responsibility to lead, and we are USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 26 supporting. But they expect the CSOs to take the lead role. It seems like the role has been reversed.” Working Paper I illuminates this fundamental challenge: “Given the current pattern of engagement there is a risk CSOs may become a Band-Aid for government ineffectiveness—rather than helping transform the local government bureaucracy. There is an implicit understanding in some cases that the CSO provides all the ideas and skills, while the government just provides space. Moreover, the prevalence of contractual relations between local government and CSOs pose a challenge to realizing effective and sustainable collaborative governance. This would require that the CSO be able to embed its values in the planning and decision-making process—rather than be just a partner or a contracted party for implementation.” CONCLUSIONS MADANI has provided CSOs with processes and tools to promote its approach to collaborative governance and has created opportunities for CSOs, LGs, and communities to come together, exchange information, build relationships, and achieve tangible changes in service delivery, albeit not across all districts. CSOs continue to need technical support, resources, and time to adopt and apply the collaborative governance process on their own, and it is too early to determine whether they could replicate the approaches or sustain the advocacy efforts without USAID or MADANI backing. Also, pressure to show results against indicators may hinder sustainability of CSO capacity building in the three IRs. EQ3: HAS MADANI IMPROVED THE SUSTAINABILITY OF TARGETED CSOS IN BRINGING ABOUT CHANGE IN POLICY AND PRACTICE? IF SO, HOW? IF NOT, WHY NOT? FINDINGS Improvements in CSO financial sustainability MADANI aims to help CSOs mobilize financial resources that allow them to become self-reliant by diversifying their funding sources, developing short- and long-term financial strategies for their organizations, and capitalizing on available local and national government funding mechanisms. Initially, MADANI postponed its TA on resource mobilization to develop a more comprehensive resource mobilization strategy; it then faced delays contracting a service provider. Thus, the TA on resource mobilization had not yet started during data collection for this evaluation. In the absence of this TA, MADANI has supported its target CSOs in drafting resource mobilization plans for their organizations, identifying local funding opportunities based on their expertise, relationships, and background. MADANI’s field coordinators also provided support to CSOs on fundraising and collaboration with the private sector for their selected thematic action plans and held five online discussions on corporate social responsibility (CSR) engagement and access to philanthropic funds. None of the CSOs had received funding through Presidential Decree (PerPres) 16/2018, which allows registered CSOs to bid for government tenders, so MADANI has identified 16 potential districts to access this government funding. Support for this activity had not yet taken place as of the writing of this evaluation. Many of the target CSOs are interested in applying for and receiving these funds and think it is a good opportunity to foster a reciprocal relationship between LGs and CSOs. However, other interviewees thought it would compromise the CSOs’ independence and ability to advocate for LG accountability, especially when the themes are not in the interest of the 27 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV government or a government official. Several interviewees were also concerned that MADANI staff were putting pressure on the field coordinators, DSPs, and service providers to encourage the CSOs to access government funding even if it did not align with their principles, vision, mission, and strategic plans. Although tailored activities for resource mobilization are still in the early implementation phase, MADANI’s organizational capacity assistance also supports CSOs’ financial sustainability. For instance, all CSOs have developed or revised their core administrative documents (e.g., SOPs, strategic plans, and MEL plans), which makes them compliant and eligible to receive USAID and other donor funding. MADANI has also provided additional organizational capacities, including USAID grant management, knowledge, and skills; expanded networks; and improved relationships with LGs and other stakeholders, including the private sector. As a result of MADANI’s support, in some districts, CSOs have received funding from new sources (i.e., donor, international NGO, LG, CSR) primarily as service providers. Ten MOUs have been signed with private sector companies, and the LG has contracted a few CSOs as service providers. However, many of the connections with donors, international NGOs, LG offices, and the private sector were initiated by MADANI staff, service providers, and DSPs. Therefore, it is unclear whether many of the CSOs would be able to make these connections on their own or whether the various funders would be interested in supporting them without USAID/MADANI backing. Challenges to improving CSO financial sustainability Achieving long-term sustainability remains an overarching challenge for nearly all CSOs. Most current resource streams are not sustainable or sufficient for the LPs to support long-term strategic planning and capacity building. Instead, CSOs are limited to time-bound, project-based funding focused on small-scale service provision, hindering them from reaching a sufficient level of capacity to effectively advocate for greater LG accountability. The evaluation identified additional challenges that may hamper MADANI’s efforts to improve CSOs’ financial sustainability: ⮚ There is a lack of clarity on whether MADANI’s goal for capacity building on resource mobilization and financial sustainability is for CSOs to become service providers for LGs, advocate for LG accountability through local development processes, or access funding from any source. The first two goals may conflict with each other—advocating for LG accountability could inhibit CSOs’ ability to access funding, and if the CSOs receive the funding, they may be deterred from advocating for LG accountability—thereby trapping CSOs in a pattern of trying to secure funding from the LG rather than holding the LG accountable. As explained in MADANI Working Paper I: “With the marked scaling down of donor-funded projects in Indonesia, especially in the field of civil society support and empowerment, the space for civil society is increasingly limited (USAID 2019; Mietzner 2012). This makes competition among CSOs to secure longer-term funding greater, while at the same time makes many CSOs increasingly dependent on government contracts as a source of funding….This pattern of engagement, while providing an important avenue for “penetrating the state,” risks undermining the capacity for civil society political mobilization and programmatic, evidence-based engagement—because negotiation is mainly driven by exchanges of favor with the powerholder.” USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 28 MADANI does not force CSOs to choose government funding over other sources and plans to provide targeted TA to each CSO based on needs and potential funding sources. CSOs that cannot take government funding are encouraged to develop their abilities in networking with the private sector, adopting crowdfunding platforms, and optimizing the use of social media. Accessing funding from diverse local sources has proven to be challenging, however. It is difficult for most small CSOs to access the private sector without USAID/MADANI support. An interviewee stated, “The private sector companies are local, so it is challenging to obtain resources because these are only small-scale companies. We have mapped several businesses….If we want to establish this cooperation with the private sector, the monetary support is very small, so this is a challenge.” During interviews, several private sector companies, including banks, stated that they are for-profit businesses so there would have to be a benefit for them to contribute to a CSO. An interviewee stated, “We usually do not give money to CSOs. We may contribute in-kind to the LG….CSOs could have a facilitating role to make suggestions, but multiple stakeholders would need to be involved. It cannot just be one CSO.” Interviewees also thought it was unlikely that the private sector would fund CSOs to advocate for LG accountability. ⮚ Large gaps between MADANI’s expectations and the allocated budget, timeframe, and CSO capacity to achieve objectives have hindered the program’s progress, discouraged some CSOs from applying for the acceleration grant, and caused some national service providers to either end their contracts or provide inadequate services. A broad range of stakeholders stated that most CSOs had limited capacity to absorb and apply the information provided to them without significant external support from volunteers, MADANI staff, and service providers. For example, several interviewees stated that MADANI’s support on resource mobilization and financial sustainability was too advanced and that most CSOs would not have adequate human resources to execute their resource mobilization plans. Excerpts from interviewees reinforce this point: “MADANI is working with very small and nascent CSOs. Two years is not enough time for capacity building. They need ongoing support.” “We are introducing more tools like financial resilience. For the next 1.5 years we are going to help them develop good proposals to submit for funding. We are focused on three types of funding (government, private, and income generation model and crowdfunding). We first conduct an online training and then we provide onsite technical assistance….During this process we are providing some advice and mentoring to each partner. We start at the individual level and move to organizational level and then collectively among the CSO partners. For the next 15 months we are going to try to see the improvement of their capacity in terms of the resource mobilization. First, we use the diversification of their funding; effective system to mobilize and manage funds; capacity to make an endowment fund for long-term sustainability. This is very advanced. This can’t happen in 15 months…MADANI’s partners are small and young.” National service providers have also faced difficulties meeting MADANI expectations given their constrained human resources and inadequate budgets for travel or for hiring expert consultants. An interviewee stated, “The unit cost is very small. This is the main issue, but we don’t share this with MADANI.” But when they talk with the other service providers and the CSOs, they all have an issue with the budget allocation in relation to the expectations and tasks. As another interviewee stated, “MADANI did not allocate enough budget for specific staff to conduct resource mobilization training. We gather experts on resource mobilization to provide materials to the CSOs. But, in the 29 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV implementation, it will be difficult for the CSOs without the human resources to execute their resource mobilization plans.” MADANI offers a variety of top-notch national service providers, such as ReMDeC, to help CSOs increase their capacity for financial sustainability. All LPs received a basic development training program covering financial management, human resource management, and strategic planning. ReMDeC inducted the final eight LPs in Banten and West Kalimantan in October 2020, in addition to providing 76 personnel from 23 LPs who engaged in basic training. Later, in Q3 FY 2020, ReMDeC gave them the fundamental training in organizational, financial, and human resource management that they had given to LPs in the other provinces, and in Q2 and Q3 of the same year, they gave them more advanced training in organizational development. Concerns arose, however, about CSOs‘ capability to internalize the materials, partly because of ineffective training modalities. The same challenges are present in the trainings on resource mobilization and financial sustainability that were in the organizational capacity development, both offered through the same national service provider. And similar budget constraints are likely to inhibit the service provider and consultants from providing the type and amount of training that each unique district and CSO needs. Many CSOs were concerned that MADANI staff and the national service provider would help them map out a broad range of potential funding sources and develop strategies but that they would not have enough knowledge or capacity to implement the strategies. As stated by an interviewee, “Sustainability imposes a great deal of challenge….We worry that we can only just produce documents.” Another interviewee said, “The online TA was difficult….These TA cannot be absorbed….We need direct coaching and assistance…for resource mobilization….We received the theory, but we have not been able to implement the theory.” ⮚ The Learning Forums’ ability to sustain changes in policy and practice is uncertain. Local politics, LG and village budgets, and Learning Forum sustainability also affect CSOs’ ability to effect change in policy and practice. Most of the CSOs, LGs, and communities relied on the Learning Forum members to implement their pilot collaborative governance activities. And if the CSOs and Learning Forums cannot mobilize resources to continue monitoring LG compliance, their legitimacy in the communities could falter. When asked if the Learning Forums would continue working together on the current advocacy theme or work on new themes without MADANI, many Learning Forum members stated that it was their duty and they wanted to continue but that they were waiting for the CSO to tell them what was next. They also said that they had identified a lot of advocacy themes they wanted to work on but that the themes were not in the CSOs’ work plan so there was no budget. As stated by an interviewee, “We need support from USAID/MADANI to work on more issues....Everything depends on funding. We need funding, TA, guidance, etc.” Other stakeholders reinforced this perspective, stating: “Theoretically [the Learning Forums] are working. They are designed to be a strategic forum and if it works then it becomes a unity of power perspective to advance strategic issues with the LG. It is also a platform for collaborative governance. Maybe 30 to 40 percent of Learning Forum are working well. The rest are struggling. The issue is sustainability. “Learning Forums may not be sustained outside of the MADANI project. [Some Learning Forums continued to meet] even if MADANI stopped for a little while. Others did not. The leadership of the CSO is important for the sustainability of the Learning Forum. In USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 30 some cases when the CSO did not receive grants, [the CSO and Learning Forum] stopped the MADANI activities until they got the new grant funding.” Several interviewees stated that in several districts MADANI should have explored and strengthened established structures before setting up new Learning Forums to increase the likelihood of sustainability after MADANI ends. For example, an interviewee stated, “There are some existing intergovernmental groups like a women’s group consisting of LG employees, but MADANI has not connected the LP with these groups….The chairperson of this women’s group is the head of SECTA so it would be good for the LP to connect with and invite this inter-governmental women’s organization of female employees because they have a direct connection to the regional secretariat.” Most of the CSOs and Learning Forums will likely be unable to mobilize adequate financial resources to continue working in the current pilot locations or to replicate MADANI’s collaborative governance approach to advocacy in new locations without ongoing human and financial support. It does not appear that MADANI has an exit strategy for each of the CSOs and Learning Forums. Multiple stakeholders raised this concern. An interviewee stated, “I hope that MADANI does not abruptly end because based on our experience working with international donors, everything ends when the project ends. We want an exit strategy with MADANI.” CONCLUSIONS MADANI has helped CSOs become more eligible for donor funding, and some CSOs have received funding from other sources. However, they still need significant organizational capacity development and support, including in financial management, budgeting, and resource mobilization planning and implementation. Other factors involved in financial sustainability may be beyond MADANI’s purview, including established human and financial resources, motivation, past performance, proposal writing skills, dependence on individual charismatic leaders, marketing and presentation skills, creativity, relationships with donors, entrepreneurial skills, professional and social networks, time management, investing skills, and others. Therefore, it is too early to determine whether the current amount of TA and mentoring will contribute significantly to CSOs’ ability to mobilize enough financial resources to sustainably effect change in policy and practice. In addition, financial sustainability efforts risk compromising CSOs’ ability to independently advocate for LG accountability, trapping them in cycles of service delivery for donor- and government-funded projects. Explained in MADANI Working Paper I, providing services and technical support may yield revenue sources and connections with donors and LG officials, but “overreliance and complacency on the technocratic domain and contractual relations may risk stagnating CSOs, in the long run, as an independent voice of accountability and speaking truth to power.” Helping CSOs sustainably effect change in policy and practice requires more than supporting the CSOs in acquiring financial resources. If the goal is to encourage CSOs to continue advocating for LG accountability using MADANI’s collaborative governance approach, the Learning Forums will also need support in resource mobilization and financial sustainability. 31 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV EQ4: HAS MADANI IMPROVED THE ENABLING ENVIRONMENT FOR CSO ENGAGEMENT? IF SO, HOW? IF NOT, WHY NOT? FINDINGS Improvements in the enabling environment for CSO engagement in the local development process MADANI aims to improve the enabling environment for CSOs at the local level by conducting activities to support LG capacity and willingness to collaborate with them. Specifically, MADANI encourages LGs and the private sector to work with CSOs. The work revolves around increasing engagements between LGs and CSOs; increasing local policies drafted, proposed, and adopted that foster an enabling environment for CSOs to effect change; and enhancing transparency and providing citizens with access to information. As discussed under EQ2, MADANI has contributed to building relationships between CSOs and LGs and improving LG receptiveness and responsiveness to CSO advocacy. The evaluation found an increase in CSOs’ confidence and capacity to engage LGs in thematic discussions, which have led to policy change in most districts. These improvements were facilitated through the collaborative governance process, public consultation forums, pilot projects, exploratory networks, and capacity building in financial management and resource mobilization. Increased engagements between LG and CSOs: Many CSOs and Learning Forums valued the hands-on learning experience of working through the stages of MADANI's collaborative governance approach and implementing the various tools. Some CSO staff reported gaining valuable knowledge undertaking fundamental socioeconomic and stakeholder analyses. In addition, some CSO staff have established and assumed leadership roles in multistakeholder forums, used and evaluated data to support evidence-based collaborative governance work, and encouraged communication with a variety of stakeholders to generate better changes. However, the CSOs and Learning Forums require more than one or two chances to practice, reflect, and learn to be able to repeat the procedure independently. To promote an enabling environment for collaborative governance, MADANI, with provincial government support, selected districts based on the LG’s interest and capability in collaborating with CSOs. This selection process enabled MADANI to work with LGs that were ready for CSO engagement. In the initial consultations with provincial and LG stakeholders, MADANI also emphasized that they would work only on strengthening the demand side, so the LG collaboration was expected to enable dialogue and, if willing, resource allocations to address specific thematic issues. MADANI prioritized thematic areas in which a collective agreement process could thrive. Positive results were evident in several targeted cities. For example, in Serang, active involvement of the target CSO and Learning Forum in the village development planning forum (musrenbangdes) has led to a pilot project follow-up on waste management issues in the mentored villages through a village regulation-making process. Meanwhile, in Solo, the LG enabled support for all CSOs in diverse aspects, including accountability issues. The LG invited the target CSO to the Development Participation Planning Evaluation (by Bappeda), musrenbang, SOP OPD Kesbangpol discussion, and so forth. The Learning Forum also participated in formulating the Regional Medium-Term Development Plan (RPJMD), musrenbang, and Kesbangpol’s Working Plan and in SOP discussions. Bappeda’s provision of workspace for MADANI’s field coordinators also demonstrates greater LG support for CSO engagement. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 32 Interaction and engagement between the LG and CSOs have grown in Singkawang, where the Bappeda feels that the CSO now has a variety of methods for handling thematic issues. Previously, there had been a lack of CSO knowledge on the issues and the role they play. CSOs were perceived as not representing the community because they did not understand the necessary process for engaging with the LG. But with MADANI’s support, the CSO is now considered effective in helping the LG do their work. Understanding how LGs function was crucial for CSOs to collaborate and form partnerships with LGs. In addition, the pilot projects demonstrate increased engagement with LGs and communities. For example, in one district, the CSO involved youth in conducting a social analysis and mapping the community’s social conditions and applied the community scorecard to assess youth services provided by the LG. The CSO sent a policy brief to the mayor and other LG offices on how youth could be involved in community consultation forums. As a result, the LG stated that the pilot project offered a model of how to work with CSOs, demonstrated the role of substantive participation, and initiated a partnership between the LG and CSO. The pilot project also changed the LG’s skepticism about youth involvement and in turn involved youth representatives in the LG process. Government mistrust and lack of understanding of how to cooperate with CSOs present challenges for collaborative governance. Although in some districts, LG perceptions of CSOs are improving and LGs are slowly engaging CSOs in the planning and development deliberation (musrenbang), the extent of CSO participation has been simply to attend the meetings. In West Kalimantan, there is still confusion about the role division between the LG and CSOs, and it became a daunting task to get the LG to assume its supposed roles and responsibilities to lead and make decisions. In contrast, the LG expects the CSOs to take on the leading role, but the CSOs’ fundamental role is to support the LGs and not vice versa. However, CSOs can now apply collaborative governance approaches before taking any action at local level by combining their current resources with those of state authorities. This has boosted CSOs’ confidence level when networking with LGs. Additionally, CSO participation in government decision making has grown substantially—mainly through public consultation forums. For example, in Wonosobo, the target CSO felt that its legitimacy and branding improved through better collaboration with the LG. The collaborative governance task force in Pangkep has also helped foster LG recognition of CSOs through the CSOs’ work on maternal and neonatal health, which raised health care service quality. The LG has thus trusted the CSO to also address other health-related issues, such as stunting, making them an integral supporting partner for the LG. The CSO is also collaborating with Kesbangpol to measure the capacity of all other CSOs in Pangkep by overseeing the CSO registration database. Increased local policies that foster enabling environment for CSOs to effect change: Many of the collaborative governance processes on specific themes selected by the government have resulted in policy briefs, decrees, regulations, agreements, and other outputs. For example, according to the FY 2022 Q2 Progress Report, 44 policy recommendations have been formally adapted or adopted. For example, in South Sulawesi, the target CSO demonstrated skills in drafting and utilizing policy briefs. With MADANI support, the CSO advocated for the signing of an MOU to commission information from the LG. The CSO presented a policy brief, conducted activities, and sought audiences with Diskominfo. As a result, the mayor’s regulation was revised and an SOP was established. 33 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV However, the timing of the CSO’s policy briefs did not always fit into the LG’s planning cycle; in some cases, the budget had already been committed and no revisions could be made until the next budget planning period. CSOs must understand the LG's regular planning and budgeting operations because CSOs have built relationships with the LG. They cannot simply pressure the LG to implement the policy brief. In addition, the level and definition of adoption require further investigation. In many districts, policy adoption did not go deep enough. The LG’s response must go beyond a decree; how to ensure the LG’s willingness and ability to follow through is still a work in progress. For example, in Singkawang, after the LG issued a pledge to turn the pilot school into an inclusive school, no further support was given to increase teachers’ capacity to teach students with special needs, and the curriculum has not been revised for students with disabilities. Finally, some CSOs require additional capacity and support to draft policy briefs. In one district, the CSO submitted policy briefs to the LG without prior discussions or a consultative process, which led to their rejection by the LG. Some CSOs without prior experience in advocacy work face difficulty implementing it. It is not easy to develop a policy brief and it requires a staff member to be fully in charge of it. In addition, the CSO needs to maintain communication and follow-up with the LG to ensure that they adopt the recommendations in the policy brief and allocate a budget. Otherwise, advocating for LG accountability is futile. Enhancing transparency and providing citizens with access to information: In most districts visited, the CSOs reported difficulty gaining access to LG documents, especially on budgeting (APBD). Only one district (Tangerang) had access to APBD, which is currently used for analysis and draft policy briefs. In Makassar, the CSO prepared a policy brief to request public information from the LG. By law, all regional organizations should have a person in charge of public information and disclosure. The information management and documentation officer is responsible for publishing public documents but only one person is in charge, whereas the national law requires that each LG must have a division for public information. Reports of commission indicate that Makassar has never been transparent with public information. With MADANI’s help, the CSO pushed two issues through a policy brief: diversity/social inclusion and disclosure of public information. A declaration has been achieved between the CSO and the Makassar LG. Enabling policies and environment improved MADANI has increased CSOs’ legitimacy in the eyes of the LG and other stakeholders, thereby improving enabling policies and environments. In Solo, MADANI helped the Kesbangpol reverify the number of active local CSOs in the city. Both also collaborated in holding a capacity-building event targeting 80 local CSO representatives. Bappeda also benefited from MADANI’s presence as the target CSO helped Bappeda launch an OPI assessment on the Coordinating Team on Regional Poverty Alleviation in all subdistricts in Solo. Furthermore, Kesbangpol prepared IDR 75 million in 2022 to strengthen the local CSOs through OPI’s introduction and practice, including its measurement, technical assistance, and recommendations. The other governmental bodies approached by MADANI were Bank Indonesia and Baznas Jateng, both of which the CSO assessed to determine further collaboration opportunities. Local CSO involvement in LG meetings, such as with the District Government, Bappeda, Kesbangpol, and other agencies, also demonstrates the improvement in enabling policies and environment. Komunitas Belajar MADANI (Kombes), as mentioned above, also participated in formulating the RPJMD, musrenbang, and Kesbangpol’s Working Plan and in SOP discussions. LG recognition of CSOs also led to realization of CSO policy recommendations—for example, in the USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 34 Budget Execution Document (DPA) Kesbangpol FY 2022. This proposal is important for strengthening CSO capacity in Solo, including through OPI training to the 40 local CSOs, institutional strengthening activities (such as arrangement of AD/ADRT, strategic plans, employment SOPs, financial SOPs, and procurement SOPs), and development of a database of ideas. Collaborative governance between CSOs and local communities also enabled stronger ties between stakeholders in Boyolali. And in Wonosobo, the LP (KITA Institute) felt that its legitimacy and branding had improved, as shown by better relations with Bappeda, the Health Office, Puskesmas, Kesbangpol (Divisions of Organization and Governmental Affairs), and the Education Office. MADANI encouraged the CSOs to explore relationships with other stakeholders, such as the private sector, academia, and local residents. For example, a CSO signed an MOU with the most prominent university in Solo, Universitas Sebelas Maret. And a CSO program for local residents delivered through the AtmaGo platform (a MADANI-supported citizen journalism digital platform) has had a direct impact. MADANI’s efforts to improve enabling conditions for CSO engagement in local development processes yielded several lessons. For example, the team must analyze the program area’s political conditions. It must also maintain good relations between CSOs and LGs. In addition, involving external parties can influence how the government treats CSOs. The sustainability of these gains after MADANI ends in 2024 may depend partly on the results of simultaneous national and regional elections that year, as the victor could change policies, including on budget allocation, and government officials’ ranks. Also, political friction could surface during a government transition, making it difficult for CSOs to intervene in LG projects. For instance, the latest Regent of Jember does not cooperate with CSOs that do not support his election. MADANI has also fostered an enabling environment for CSOs by improving LG accountability. Some subnational CSOs have gained knowledge and experience through MADANI TA and capacity building, including conducting basic socioeconomic and stakeholder analysis, applying social accountability tools, conducting an evaluation system, and developing policy briefs. As a result, they are more aware of local issues, and some CSO staff have used data to support evidence-based collaborative governance work and facilitated dialogue with a range of stakeholders to promote mutually beneficial changes. Some staff now have experience forming and taking on leadership roles in multistakeholder forums. The CSOs have also been exposed to monitoring activities. However, the short project timeframe presents a significant barrier to improving the enabling environment; five years do not provide enough time for longer-term change. As stated in MADANI’s Working Paper 1: Instead of striving solely for overarching and transformative goals—such as passing a local law or achieving regulatory reform—CSOs should also focus on the process of engagement with the local government, rather than output and outcome alone. A mechanism is needed that specifically assesses such engagement, with the design of monitoring mechanisms that assess processes, especially in terms of building civil society networks and building trust with the local government. Evaluation mechanisms should measure outcomes for intended beneficiaries—not just program log-frames.” CONCLUSIONS MADANI has strengthened the enabling environment to foster CSO engagement in the local development process through collaborative governance, pilot projects, public consultation forums, 35 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV exploratory networks, and capacity building in financial management and resource mobilization. These approaches have increased engagement between LGs and CSOs; local policies drafted, proposed, and adopted; and, to a lesser extent, transparency and citizen access to information. MADANI has made it possible for CSOs, LGs, and communities to collaborate and discuss common interests. It is uncertain whether CSOs can continue initiatives on their own, however, especially if they engage the LG in more divisive issues. The political environment, the short activity lifetime, and CSOs’ nascent level of capacity present challenges for sustainability, however the cooperation between the CSOs and the private sector is expected to reduce CSO dependence on its donors. RECOMMENDATIONS RECOMMENDATIONS FOR CSO ORGANIZATIONAL CAPACITY ● To build organizational capacity beyond the foundational level, MADANI and the IP should adjust the strategy to focus on (1) clear and specific action plans and learning steps for each CSO based on a revised assessment of their unique needs, interests, and context; (2) tailored mentoring and coaching to apply knowledge and tools focused on achieving their own results and addressing their own issues; (3) knowledge application assessment and organizational performance measurement; and (4) additional capacity building on gaps. ● The IP should adjust training efforts to (1) ensure adequate expertise from the service provider; (2) ensure that the service provider has adequate time, human, and financial resources to provide CSOs with the appropriate level of support; (3) improve training quality, delivery, and topics; (4) diversify approaches, such as expanding the DSP model; (5) increase access to resources and opportunities for ongoing learning; (6) avoid limiting capacity development to technical troubleshooting and include other organizational aspects such as innovation, culture, leadership, networks, and partnerships; and (7) limit the number of external stakeholders involved in building each CSO’s capacity. ● MADANI has tried to incorporate GESI, tolerance, and anti-corruption into program implementation, and there are anecdotal examples of positive change. However, challenges and constraints among national service providers, consultants, LPs, and Learning Forums have limited these efforts’ efficiency and effectiveness. Therefore, it is important to integrate these themes into other capacity-building activities. ● MADANI and the IP need to improve the selection criteria for participants in the crosscutting capacity-building themes (such as GESI, tolerance, and anti-corruption), including seniority and commitment, to ensure that the participants can understand, internalize, and transfer the knowledge to their colleagues, CSOs and Learning Forums. ● The IP should invite LG representatives to some of the trainings and workshops. This may require a different strategy. For example, the first trainings may be separate, the LG training covering topics such as how to mainstream GESI, tolerance, and diversity into public policy while the CSOs focus on understanding the issues and integrating them into their organizations and lives. Then the LG and CSO representatives could come together to discuss the topics and how they can work together to mainstream them into their policies and organizations. ● The IP and MADANI need to make the CSOs’ work plans more flexible so that they can adapt and incorporate topics related to GESI, diversity, and inclusion into their activities or design and implement new activities in collaboration with other partners. The predetermined milestone to achieve USAID grants should align with the CSOs’ capability as well as refocusing strategy on the activities that the LPs prioritize. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 36 ● MADANI and the IP should disseminate best practices and processes for CSOs to use as reference resources and expand peer-to-peer learning by pairing CSOs with peers to promote horizontal learning and transition MADANI’s function to local, sustainable actors. ● The IP should review and strengthen MADANI’s administrative and operational processes to identify inefficiencies and unintended incentives that may be hindering program implementation and overburdening staff. This can include improving lines of communication and creating safe spaces and opportunities for candid dialogue, feedback, and sharing of successes and failures between USAID and MADANI staff in Jakarta and those in the field; streamlining MADANI’s administrative processes and tasks to enable programmatic staff to focus more on providing field support on programmatic issues; and developing objective monitoring and measurement of increased organizational effectiveness and efficiency beyond the OPI self-assessment. RECOMMENDATIONS FOR LG ACCOUNTABILITY AND PROMOTING TOLERANCE ● Strengthen CSOs’ capacities and confidence in collaborative governance by tailoring tools to CSOs’ individual contexts. Test experiential learning approaches in districts where CSOs, LGs, and communities have made progress on a thematic topic by taking lessons learned and applying the process in another location or theme with less intensive support from MADANI. Use this approach as an opportunity to build CSO ownership in replicating the process and to identify gaps and adjust approaches for the remainder of the program. ● MADANI needs to support more collaborative governance events with issues that can be mainstreamed to LG development plans, such as Regional Medium-Term Development Plan (RPJMD) and Regional Government Work Plan (RKPD). ● The IP needs to provide more training and practice on conducting sociopolitical, economic, and stakeholder analysis so that CSOs understand the barriers and opportunities in advocating for a certain theme or promoting community tolerance on a certain topic in a specific location. ● The IP needs to collaborate with USAID/ERAT in a few districts to experiment with simultaneously supporting both the supply and demand side in relation to a shared theme. This will entail a lot of coordination between the programs as it must be run as one program in these test districts. ● The IP needs to disseminate lessons learned consistently and transparently through knowledge products and exchange visits among the CSOs in a province. ● Incorporate a mid-course pause-and-reflect exercise into quarterly "Learning Weeks" within MADANI’s M&E system for CSOs on the TOC, underlying assumptions, contextual changes, and the extent to which activities are contributing to achievement of program objectives. Revisit the selection criteria for CSOs in future programs of this nature. Many of the LPs are not set up to be think tanks or advocacy organizations focused on LG accountability or community tolerance. ● Rather than just measuring outputs and outcomes (milestones and indicators in MADANI’s log frame), the IP should develop a mechanism that assesses the process of engagement among stakeholders. The assessment should measure “where [the stakeholders] are on the engagement ladder—from tokenism, to cooperation, to collaboration. A clear and honest process of assessment is necessary, and then a plan on how to move from there towards the next level.”10 10 Prio Sambodho – Collaborative Governance in Strengthening Accountability and Tolerance in Decentralized Indonesia – A MADANI Program Assessment. July 2020 (Working Paper 1). 37 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV RECOMMENDATIONS FOR CSO FINANCIAL SUSTAINABILITY ● MADANI must set realistic expectations of what can be achieved in building CSO capacity to mobilize resources. ● Assess the degree to which sustainability initiatives result in CSOs becoming donor driven, or government contractors could limit their independence to advocate for LG accountability. The IP and MADANI need to improve the enabling environment for LPs to reinforce their current financial sources through capacity building in resource mobilization. Adjust the training curriculum to build on CSO capacities with relevant approaches/tools. ● Apply recommendations from MADANI’s other capacity-building efforts (see EQ1 and EQ2), including (1) contracting TA service providers with adequate capacity and budget to conduct effective activities, (2) avoiding generic trainings and resources, (3) developing tailored curricula and learning steps with enough time to learn, (4) focusing on knowledge transfer beyond individual participants, (5) supporting practical and experiential learning rather than focusing on meeting deliverables and milestones, (6) providing peer-to-peer learning opportunities, and (7) serving as a facilitator instead of doing the work for participants. ● Develop a sustainability plan for the Learning Forums to continue after the program ends and include Learning Forum members in capacity-building activities for financial resource mobilization. ● Start preparing and developing exit strategies with CSOs to minimize programming and staffing disruptions when MADANI’s financial support ends. RECOMMENDATIONS FOR ENABLING ENVIRONMENT FOR CSO ENGAGEMENT ● Strengthen Learning Forums to provide a sustainable space for CSO engagement in the local development process by (1) enabling involved stakeholders to collaborate on mutually agreeable themes and local solutions using MADANI's collaborative governance approach or their own approach to advocacy and tolerance and (2) developing a sustainability plan with refined capacity-building and resource mobilization approaches. This can help empower members, minimize the hierarchical structure, strengthen commitment, improve outcomes, and increase sustainability. ● Continue to strengthen CSOs' internal and external capacities to better position them for participation in local development processes and avoid placing them in vulnerable situations that can undermine trust between parties, particularly the LG. Assess MADANI’s efforts around unmet community expectations and apply Do No Harm principles as part of the exit strategy. ● Consider locally led actions, innovative ideas, and transformative approaches that could institutionalize improvements in the enabling environment, such as encouraging regional legislation to incorporate a general mandate that explicitly fosters collaboration with CSOs. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 38 ANNEXES ANNEX I: TIMELINE No Activities Responsible 2022 Apr May Jun Jul Aug Sep Oct Nov Dec 1. Evaluation Design 1.1 Internal evaluation team discussion Evaluation team, MELP 1.2 Desk review Evaluation team, MELP 1.3 Team planning meeting with USAID Evaluation team, MELP 1.4 Team planning meeting with IP Evaluation team, MELP 1.6 Develop draft evaluation design (workplan, tools) Evaluation team, MELP 1.7 Internal review draft evaluation design MELP 1.8 Submit draft evaluation design to USAID MELP 1.9 Team presents evaluation design Evaluation team, MELP 1.10 USAID review evaluation design USAID 1.11 Update evaluation design based on USAID feedback Evaluation team 1.12 Submit final evaluation design MELP 1.13 Evaluation design approval USAID 1.14 USAID introductory letter to initiate data collection USAID 1.15 Field work arrangement (logistic, meeting arrangement) Evaluation team, MELP 2. Data Collection and Data Analysis 2.1 Data collection (interview, desk review, etc.) Evaluation team, MELP 2.2 Coordination with USAID Evaluation team, MELP 2.3 Data analysis Evaluation team 2.4 Develop out-brief presentation (preliminary findings and conclusions) Evaluation team 2.5 Review out-brief presentation (preliminary findings and conclusions) MELP 39 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV No Activities Responsible 2022 Apr May Jun Jul Aug Sep Oct Nov Dec 2.6 Address eventual comments Evaluation team, MELP 2.7 Presentation of preliminary findings & conclusions to USAID Evaluation team, MELP 2.8 Presentation of preliminary findings & conclusions to IP Evaluation team, MELP 2.9 Presentation to related GOI key partner Evaluation team, MELP 3. Evaluation Report and Exit Briefings 3.1 Submit two-pager to USAID and IP (new task) MELP 3.2 Follow-up discussion on two￾pager with USAID and IP (new task) Evaluation team, MELP 3.3 Writing of draft evaluation report Evaluation team, MELP 3.4 Internal review draft evaluation report MELP 3.5 Submit draft evaluation report MELP 3.6 USAID and IP review draft evaluation report USAID, FHI360 3.7 Update evaluation report to incorporate USAID and IP feedback Evaluation team, MELP 3.8 Internal review on final evaluation report MELP 3.9 Submit final evaluation report MELP 3.10 Final presentation to USAID, recommendation tracker Evaluation team, MELP USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 40 ANNEX II: EVALUATION STATEMENT OF WORK 1. PURPOSE Purpose of this task is to conduct an external mid-term Performance Evaluation (PE) to MADANI, a civil society capacity building activity funded by the United States Agency for International Development (USAID). The result of this evaluation will inform USAID leadership for making necessary project adjustments and future strategy to achieve the outcomes by working effectively with Civil Society Organizations (CSOs), particularly at the sub national level. 2. AUDIENCE AND INTENDED USE The primary audience of this performance evaluation is USAID leadership and the Project Implementer. The secondary audience includes the Government of Indonesia (GOI) at the national and sub national level, development partners, and USAID global that may benefit from the evaluation results and analysis. Results from the evaluation will be published in USAID’s Development Experience Clearinghouse to build the body of knowledge on the multi-sectoral approach. 3. USAID MADANI ACTIVITY DESCRIPTION USAID MADANI is a five year and $19.8 million civil society support initiative to strengthen government accountability and promote community tolerance at the local level in Indonesia. It empowers local CSOs by improving organizational capacity, expanding reach and relevance, and building up sustainable means of mobilizing financial resources. MADANI fosters a more vibrant, reputable, and self-reliant local CSOs in which decision making takes place in partnership with local governments (LGs) that are accountable to and welcoming of all citizens and capable of combating communal intolerance. MADANI started in March 2019 and will end in April 2024 and is currently almost at the mid-point of its five-year period of performance. By design, MADANI is set to have internal midpoints and end line PEs. To support this evaluation, a baseline data has been collected by MADANI at the initial project implementation (in February 2020). These internal evaluations were mainly to monitor project progress and expected results and help in program design, management and operational decision making. To ensure a more comprehensive understanding of one specific aspect of MADANI’s performance, USAID plans to launch this additional external PE. This evaluation will mainly focus on MADANI results in (i) improving organizational capacity of CSOs, (ii) strengthening accountability and promoting tolerance, and (iii) improving capacity of CSOs in mobilizing local resources. DEVELOPMENT CONTEXT The U.S. Government, through USAID, encourages citizen participation and engagement in government, society, and the private sector for greater economic growth. An active and empowered civil society is necessary for a strong democracy, promoting accountability, inclusivity, and tolerance. Civil society organizations have proven their valuable role as watchdogs, advocates, and representatives of citizen interests. However, capacity and sustainability challenges, including inefficient organizational management, lack of technical expertise, dependence on individual charismatic leaders, low accountability, and weak financial management are even more acute at LG level. Meanwhile, dependency on donor support and the current legal environment are impediments to CSOs sustainability. While Indonesia in 41 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV general is a vibrant democracy where citizens, there are signs of democratic stagnation due to growing intolerance and shrinking civic space. This situation poses a threat to the effective roles of civil society. Furthermore, the COVID-19 pandemic impacted negatively on the sustainability of CSOs. When LGs allocate most of their resources to tackle the pandemic, many CSOs depend on funding from external parties, business units that are struggling to continue their activities. The pandemic has also introduced social distancing that limits in person interaction. As a result, CSOs were forced to conduct their activities virtually. During the pandemic, USAID MADANI also was forced to conduct most of its technical assistance virtually. This virtual method has raised questions to the effectiveness of technical assistance deliverance. THEORY OF CHANGE AND RESULT FRAMEWORK The overarching MADANI goal is to strengthen LG accountability and communal tolerance in Indonesia by improving and sustaining local civil society’s capacity, legitimacy, and sustainability. The primary outcomes of MADANI work directly and through grantees are: • Strengthened resilience of local democratic practices • Improved local and village government performance and responsiveness • Better accountability of frontline service delivery • Improved and inclusive mechanisms for community-level participation and transparency • More tolerant and inclusive communities The theory of change for MADANI is: IF the organizational capacity of civil society groups at the local level increases and sustainability is enhanced, THEN Indonesian civil society can effectively strengthen government accountability and promote tolerance. Progress is tracked toward achievement of three Intermediate Results (IRs), each with two Sub-IRs, as indicated in the framework and program logic below. The figure below shows the MADANI results framework. 4. INFORMATION SOURCES USAID recommends the following MADANI’s documentation for the desk review of this mid-term evaluation: USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 42 • Activity monitoring, evaluation, and learning plan • Quarterly and Annual Progress Reports • Core list and/or copies of the awards, performance reports, relevant baseline or assessment reports, and other key IP documentation. • Activity midline survey report conducted by the IP 5. EVALUATION QUESTIONS This evaluation will assess four evaluation questions with explanatory questions for each question: 1. In what ways has MADANI improved the Organizational Capacity of its CSO partners? ● In what ways has MADANI has improved the organizational capacity of its CSO partners (focusing primarily on their improved effectiveness to achieve organizational results and Improved efficiency in managing core issues)? ● Which approaches or tools are more effective? Why? ● What enabling factors are most effective in improving capacity [defined as organizational performance index that include aspects of effectiveness, efficiency, relevance, and resiliency. Please refer to documents]? How has the experience and lesson learned of MADANI been working with sub national CSOs? What are the barriers and opportunities in improving CSOs organizational capacity? ● How do MADANI and CSOs cope with the pandemic? How does pandemic affect its effectiveness in delivering technical assistance? 2. In what ways has MADANI strengthened local governments’ accountability and promoted tolerance? ● To what extent did MADANI enable CSO partners to advocate for government accountability and tolerance in their communities? How? ● Which social accountability tools or mechanisms were most effective and why? What factors (size and type of CSO, level of pre-existing capacity, funding source, etc.) were most conducive to success? ● How has the experience and lesson learned of MADANI been working with sub national CSOs to strengthen accountability and promote tolerance? What are the barriers and opportunities in improving their capacity in these areas? 3. In what ways has MADANI improved the capacities of CSOs in mobilizing local resources? And how? ● To what extent has this capacity improvement led to financial sustainability for the different types of CSOs? What is the likelihood of this improvement to sustain post MADANI support? What factors contribute the most to facilitating this sustainability? ● Which approaches or tools are most effective? Why? ● How has the experience and lesson learned of MADANI been working with sub national CSOs? What are the barriers and opportunities in improving CSOs capacity in this area? 4. In what ways has MADANI improved the enabling environment for the CSOs engagement? ● In what ways has MADANI improved the enabling environment at local level to enable CSOs engagement in the local development process? ● What kind of enabling policies and environment have been improved? ● How has the experience and lesson learned of MADANI been working with sub national CSOs? What are the barriers and opportunities in improving CSOs' enabling environment? During the PE co-design with MADANI IP and USAID, stakeholders may agree to a modification of the evaluation questions listed above. 43 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV GENDER CONSIDERATIONS In accordance with USAID’s Automated Directive System 201 point 7, the research design for this evaluation shall consider gender-specific aspects pertinent to MADANI activity. MADANI supported gender equality, women’s empowerment, and social inclusion (GESI) in three ways: 1) advancing women’s leadership and empowerment in CSOs; 2) promoting a GESI-sensitive organizational culture in CSOs; and 3) setting a GESI-relevant advocacy agenda through CSO networks. As such the evaluation team shall explore gender aspects in line with the evaluation questions and available data sources. 6. CO-DESIGN OF MADANI PERFORMANCE EVALUATION The evaluation team will collaborate and coordinate with the MADANI implementing partner (IP) for the following evaluation activities: • Field work, including site visits for the determination of PE design • Coordination between the evaluation team and the IP data collection • Review of performance monitoring data, implementation documentation, MADANI’s midline survey and fidelity to the agreed upon implementation and PE designs • Communication and outreach to local stakeholders about the implementation of MADANI’s PEs. 7. EVALUATION DESIGN AND METHODOLOGY TARGET GEOGRAPHIC AREA AND PARTICIPANTS MADANI works in 32 districts in 6 provinces (East Java, Central Java, West Java, Banten, South Sulawesi, and West Kalimantan). Final site selection for the PE will be determined collaboratively by the MADANI IP, USAID, GOI and the evaluation team, to determine an implementation plan that allows for a rigorous PE. PERFORMANCE EVALUATION CO-DESIGN AND IMPLEMENTATION The evaluation team will determine the most robust PE design, in coordination with the MADANI IP and USAID. To ensure that the PE is carried out properly, the evaluation team will be responsible for (see more technical requirements under “Deliverables”): • Familiarizing themselves with documentation about the activity • Working with USAID and the MADANI IP to determine the sample frame for the evaluation • Developing and testing data collection instruments • Collecting and analyzing all quantitative and qualitative data associated with the evaluation process • Reviewing and assessing performance information or data from MADANI as they are shared by the IP and USAID • Producing an Evaluation Design Report and an Evaluation Report • Facilitating a learning event and presentation of the evaluation findings. • The evaluation team will ensure that USAID evaluation policy is adhered to and that rigorous evaluation standards are maintained as per the ADS 201. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 44 DATA COLLECTION AND ANALYSIS REQUIREMENTS As much as possible, USAID expects the evaluation team to use a mixed-method evaluation approach with careful attention paid to triangulating outcome measures. The methodology shall combine a review of quantitative data, including a review of the monitoring data generated through MADANI’s midline survey, and application of qualitative evaluation techniques to obtain information, opinions, and data from counterparts, implementing partners, relevant GOI entities, beneficiaries, and other stakeholders as appropriate. In choosing possible data collection methods, the evaluation design must consider the implications of an operating environment that has been altered by the COVID-19 pandemic. Depending on the prevailing environment at the time of the evaluation, the evaluation team must propose an evaluation design that considers remote monitoring or data collection methods as appropriate. This may include methods such as the use of cellphones or SMS to conduct interviews, interactive voice response, voice calls, or maximizing national data collection activities that are still active etc. The PE must clearly articulate the link between each evaluation question, the proposed data to address it and the analysis plan for these data. For example, the design may describe a regression model and variables to be used in quantitative analysis and how these methods contribute to answering the evaluation question. For qualitative approaches, the design will detail each planned analytical step (eg. coding frame, how it was developed) and how these steps are better at answering the relevant evaluation question than others. The evaluation design must demonstrate, through cited evidence, that the proposed approaches are best practice (based on evaluation and research literature), that they are intended to provide robust answers to each evaluation question, and that they are suitable to the Indonesian context. The evaluation must also examine differences by gender. While a detailed analysis by gender might not be relevant for each question, the PE must address each and explain who is relevant. The analysis may require more than simple disaggregation of quantitative data. For example, analysis of gender dynamics is more than statistics by gender. The evaluation team must refer to relevant USAID guidance on gender and inclusion, specifically ADS 205, and propose specific evaluation designs, as appropriate. Evaluation questions must be answered with data at various levels, including the effectiveness of MADANI interventions on the organizational capacity of the CSO, strengthening local government accountability and promoting tolerance and CSO capacity in mobilizing local resources. Qualitative data must also be used to complement the PE, to further understand the challenges and obstacles faced by the MADANI IP and to collect information on their performance. All instruments and protocols may be based on existing tools, they must be adapted and piloted to ensure they address the specific evaluation questions in the context MADANI was implemented in. Instruments must be translated, back translated, and the language used must be piloted to make sure that each question is understood and validated in the Indonesian context. The evaluation team must include a detailed plan for relevant piloting of such instruments. The inception or evaluation design report must be specific and name each data collection method used, the reason to use it and the motivation for data sources for each method. For example, if Focused Group Discussions are proposed, the evaluation team must specifically describe what is meant by Focused Group Discussion, what is the motivation for the selection of its participants and why would Focused Group Discussion be more appropriate than a Group Discussion or a Key Informant Interview for a particular data source and a particular evaluation question. 45 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV All quantitative data must be collected digitally, and qualitative data must be recorded and transcribed. The inception or evaluation report must demonstrate that proper data quality assurance systems will be put in place. All data (including transcripts) and analytical codes must be shared with USAID. 8. EVALUATION TASKS Given the above requirements, the evaluation team will be required to perform the below tasks per USAID ADS 201. COMPONENT 1 – DESIGN Task 1: Draft Inception or Evaluation Design Report There is no page requirement for the Inception Report, but it shall contain a Data Collection Plan, Analysis Plan, Dissemination Plan, Limitations/Risks, Quality Control Protocols and Work Plan. The Inception Report shall contain all these components as outlined below. ● Data Collection Plan o All data sources are identified and mapped against the evaluation question they are meant to answer o The method of data collection, including remote data collection, for each data source, who will perform the collection and the timing for when collection will take place o Relevant enumerator training protocols (if applicable), instrument piloting plan and quality control procedures o Roles and Responsibilities of the Evaluation Team for completing data collection tasks ● Data Analysis Plan o Analytical methods for each type of data collection method to include relevant quality controls for the method o Roles and Responsibilities of the Evaluation Team for completing data analysis tasks ● Collaboration, Learning and Adaptation: Outline of CLA objectives and tasks the evaluation will complete to improve the end use of the evaluation ● Limitations o The limitation of proposed collection and analysis methods to include bias, missing data points for triangulation, timing issues, etc. per evaluation question o Risk Management strategy for mitigating the effects of limitations, issues related to remote data collection methods as appropriate ● Dissemination o Timeline for producing the In-brief, Inception Report, Pause and Reflect Session (if applicable), Final Report and Out-brief o Roles and Responsibilities of the Evaluation Team for completing dissemination tasks ● Quality Control: For all data collection, analysis, and dissemination tasks, outline the risks to evaluation quality and the steps being taken to mitigate the risks with relevant citations of applicable evidence and best practice ● Work Plan: An outline of all deliverables, collection, analysis, and other tasks set against a timeline that matches the period of performance for the evaluation ● Data Collection Instruments: All relevant data collection instruments shall be included in an annex to the Inception Report. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 46 Task 2: Submit draft Inception or Evaluation Design Report On or about April 18 – 22, 2022, the evaluation team will conduct team planning meetings with the USAID/Indonesia Monitoring, Evaluation and Learning Platform (MEL-P), USAID and the IP. ● On or about May 9, 2022, the evaluation team must submit the draft Inception Report to MEL-P for review. ● On or about May 13 – 20, 2022, the evaluation team will present the evaluation design to USAID. ● On or about May 17, 2022, MEL-P will share the draft Inception Report with USAID for review. Task 3: Finalize Inception or Evaluation Design Report ● On or about June 1 – 7, 2022, the evaluation team will integrate all relevant feedback into the Inception Report, including providing responses on how feedback was or was not integrated into the Inception Report and why. ● On or about June 8, 2022, MEL-P will submit the final Inception Report to USAID for approval. COMPONENT 2 – DATA COLLECTION Task 1: Enumerator Training (as applicable) ● If applicable, the evaluation team will provide all relevant training to data collectors as outlined in the Inception Report ● If requested, the evaluator will provide relevant enumerator training materials Task 2: Instrument Piloting ● The evaluation team will pilot all data collection instruments before they are used and provide necessary piloting feedback as requested Task 3: Perform Data Collection ● The evaluation team will perform all data collection tasks as outlined in the Inception Report and provide timely updates to MEL-P. ● All quantified data will be digitally recorded and stored. All qualitative data will have summary sheets and transcripts provided upon request. COMPONENT 3 – DATA ANALYSIS AND PRELIMINARY FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Task 1: Data Cleaning ● The evaluation team will scrub all data sets of personal identification information in accordance with USAID policy. ● The evaluation team will follow best practice in preparing data sets for analysis as outlined in the Inception Report 47 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Task 2: Data Management ● All data must be stored in a secure drive that is only accessed by the evaluation team. If MEL-P or USAID requires access to raw data sets it will be done in accordance with USAID policy after removing all personal identification information. Task 3: Data Analysis ● All data will be analyzed using the methods outlined in the Inception Report. ● Final code books and statistical analysis framework will be submitted to MEL-P and USAID upon request ● Relevant quality control protocols (spot checks, inter-rater reliability checks, etc.) as outlined in the Inception Report will be adhered to. Task 4: Preliminary findings, conclusions, and recommendations ● The evaluation team will compile and present the preliminary findings, conclusions, and recommendations to MEL-P, USAID, IP and other stakeholders for feedback that will be incorporated into the evaluation report. COMPONENT 4 – EVALUATION REPORT AND EXIST BRIEFINGS Task 1: Draft and Final Evaluation Report ● On or about September 12, 2022, the evaluation team will be submitted a draft report to MEL-P. ● On or about September 23, 2022, MEL-P will submit the draft report to USAID and other stakeholders as appropriate for feedback. ● On or about October 6 – 13, 2022, the evaluation team will address and integrate all relevant feedback into the evaluation report ● On or about October 24, 2022, MEL-P will submit the final evaluation report to USAID for approval. Task 3: Final presentations or exit briefings ● On or about October 25 – November 1, 2022, the evaluation team will provide exit briefs to MEL-P, USAID, and other relevant stakeholders to present the final results of the evaluation. Task 4: Pause and Reflect ● If requested, during the evaluation or at its completion, the evaluation team will facilitate a pause and reflect session with MEL-P, USAID, and other relevant stakeholders. The pause and reflect will be timed when early results are available and will focus on possible changes to the evaluation to improve the usability of findings, to provide an update on the evaluation process, risks, etc. 9. ESTIMATED COST The estimated cost for this mid-term evaluation is USAD 197, 903 with document review, interview, site visits, and reporting. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 48 10. MANAGEMENT This evaluation will be conducted through MELP support. The MELP will establish an evaluation team with close coordination with USAID. This evaluation will require an evaluation team consisting of at least three experts with civil society, and local governance background. 11. DELIVERABLES All deliverables must be submitted to MELP who will in turn submit them to USAID. The evaluation team must promptly notify MELP of any problems, delays, or adverse conditions which materially impair the evaluation team’s ability to meet the requirements. The evaluation team is expected to deliver the following products or complete the following key tasks at the estimated due dates indicated below: Deliverables or key tasks Estimated due dates Initial team planning meeting with USAID and IP April 18 – 22, 2022 Submission of draft evaluation design or inception report May 17, 2022 In-brief to USAID on the evaluation design May 18- 20, 2022 Final evaluation design or inception report June 8, 2022 Presentation(s) to USAID and/or other stakeholders of key findings and preliminary conclusions and recommendations August 24 – 30, 2022 Draft evaluation report September 23, 2022 Final evaluation report October 24, 2022 Final presentations or exit briefings to USAID and other stakeholders October 25 – November 1, 2022 The evaluation team will ensure that the written products use plain language, concise, audience appropriate, representative of USAID achievements, and the USAID style guide is adhered to. REPORTING AND DISSEMINATION The format of the evaluation report shall follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing￾evaluation-reports). Evaluation team members shall be provided with the USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex 1). 12. SCHEDULE AND LOGISTICS The evaluation is expected to start in March 2022, including the engagement of the evaluation experts. MEL-P will be responsible for providing the evaluation team information on the implementation schedule and plans for performance monitoring and evaluation plans and indicators. In coordination with the MADANI AOR and MEL-P, the evaluation team must consult the MADANI IP to develop the evaluation design prior to the evaluation implementation. MEL-P shall be responsible for handling all logistics. The table below provides the estimated start and end dates for key tasks or deliverables. 49 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ESTIMATED EVALUATION TIMELINE Tasks or Deliverables Estimated start date (dd/mmm/yy) Estimated end date (dd/mmm/yy) Finalize SOW and develop evaluation budget and implementation schedule 04-Feb-22 25-Feb-22 Evaluation team: Development of SOWs, recruitment and contracting 04-Feb-22 16-Apr-22 Initial team planning meetings and desk review 08-Apr-22 14-Apr-22 Team planning meetings with USAID and IP 18-Apr-22 22-Apr-22 Develop evaluation design or inception report 25- Apr-22 09-May-22 Submit draft evaluation design or inception report, including presentation to USAID and USAID’s review and feedback 17-May-22 31-May-22 Finalize and submit evaluation design or inception report, including USAID’s approval 01-Jun-22 16-Jun-22 Data collection and analysis 17-Jun-22 08-Aug-22 Compilation of preliminary findings, conclusions, and recommendations. 09-Aug-22 23-Aug-22 Presentation(s) to USAID and/or other stakeholders of preliminary findings, conclusions, and recommendations 24-Aug-22 30-Aug-22 Report Writing 31-Aug-22 22-Sep-22 Submission of draft evaluation report 23-Sep-22 23-Sep-22 USAID’s and IP’s review of draft evaluation report 26-Sep-22 05-Oct-22 Drafting and submission of final evaluation report 06-Oct-22 24-Oct-22 Final presentations or exit briefings to USAID and other stakeholders 25-Oct-22 01-Nov-22 13. EVALUATION TEAM COMPOSITION As previously indicated, the evaluation team will consist of at least three experts, including an international team leader, a civil society expert, a local governance expert and research assistant. Collectively the team must have: ● Expertise on measurement with civil society and local governance support activities. ● Expertise with PE methods and data analysis, including developing and testing data collection instruments. ● A proven track record of successful implementation of PEs in developing countries. ● Expertise with field work in developing countries, including testing data collection instruments, implementing data quality protocols in the field, collecting data, and training and supervising enumerators (if any). ● Expertise in digital data collection, data quality processes and in qualitative and quantitative data analysis. ● Demonstrated success in working closely with the implementing partner to find a workable design that meets both the needs of the evaluation and matches the implementation realities. ● Expertise in meeting and workshop facilitation. USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 50 ● Experience in providing the above expertise in Indonesia is preferred but not required. USAID will review and provide approval to the composition of the evaluation team suggested by MELP prior to the implementation of the evaluation. 14. REGULAR COMMUNICATION Regular communication between the evaluation team, MEL-P, MADANI AOR, and USAID M&E Team will be essential to the successful execution of this mid-term evaluation. Through MEL-P, the evaluation team shall keep USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved evaluation design. 51 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX III: EVALUATION DESIGN MATRIX Primary EQ Sub-questions Data Collection Methods Data Analysis Approach 1. Has MADANI improved the Organizationa l Capacity of its CSO partners? If so, how? If not, why not? 1.1. How has MADANI improved the organizational effectiveness of its CSO partners to achieve their organizational results? Primary Data Collection: KIIs, Small Group and FGDs with all relevant stakeholders (i.e., MADANI staff & implementing partners, national, provincial & district support partners, lead CSO partners, LG officials, etc. Secondary Data Collection: Project documents (i.e., OPI reports and CSO self￾assessments, AMELP tools including performance indicator tracking tables, rapid assessments, baseline study, quarterly & annual reports, grant documents and workplans per each of the selected CSOs, mid￾term self-assessment, etc. - Quantitative & qualitative analysis of data points - Method Triangulation (i.e., comparative analysis between OPI scores and data from interviews - Source triangulation (i.e., comparative analysis of data from different stakeholders). - Evaluator triangulation (i.e., internal evaluation team discussions). - Descriptive Analysis & Causal Attribution (i.e., describe what change occurred and how and why it can be attributed to MADANI). - TOC approach (i.e., examine the intervention logic to assess MADANI’s hypothesis and assumptions to understand the relevance of actions and results and identify potential gaps in the causal pathways). 1.2. How has MADANI improved the organizational efficiency of its CSO partners' capacity to manage core issues? 1.3. Which approaches or tools are more effective for building organizational capacity? Why? What approaches or tools are less effective? Why? 1.4. What enabling factors (conditions) are most conducive for improving organizational capacity? 1.5. Did MADANI mainstream GESI into CSO organizational capacity development? If so, how? If not, why not? Were there any positive or negative outcomes related to this? − How has MADANI advanced women’s leadership and empowerment in CSOs? − How has MADANI promoted a GESI￾sensitive organizational culture in CSOs? − How has MADANI set a GESI-relevant advocacy agenda through CSO networks? 1.6. How has MADANI and its CSO partners coped with the pandemic? How has the pandemic affected MADANI’s effectiveness in delivering technical assistance? 1.7. What are the key lessons learned from MADANI in terms of building the organizational capacity of its CSO partners? 1.8. What are the barriers to and opportunities for improving CSO organizational capacity? 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1. Has MADANI enabled its CSO partners to advocate for government accountability in their communities? If so, how? If not, why not? − What are some of the major achievements (outputs and outcomes)? (i.e., MADANI activities improved LG responsiveness to CSO advocacy) − Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? Primary Data Collection: KIIs, Small Group and FGDs with all relevant stakeholders (i.e., MADANI staff & implementing partners, national, provincial & district support partners, lead CSO partners, LG officials, Learning Forums, etc. Secondary Data Collection: Project documents (i.e., OPI reports and CSO self- assessments, AMELP tools including performance indicator tracking tables, rapid assessments, baseline - Quantitative & qualitative analysis of data points - Method Triangulation (i.e., comparative analysis between any baseline assessments, perception surveys, etc. and data from interviews). - Source triangulation (i.e., comparative analysis of data from different stakeholders). - Evaluator triangulation (i.e., internal evaluation team discussions) - Descriptive Analysis & Causal Attribution (i.e., describe what change occurred and how and why it can be attributed 2.2. Has MADANI enabled its CSO partners to advocate for tolerance in their communities? If so, how? If not, why not? − What are some of the major achievements (outputs and outcomes)? − Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 52 Primary EQ Sub-questions Data Collection Methods Data Analysis Approach 2.3. Which social accountability tools or mechanisms were most effective and why? study, quarterly & annual reports, grant documents and workplans per each of the selected CSOs, mid￾term self-assessment, etc. to MADANI). - TOC approach (i.e., examine the intervention logic to assess MADANI’s hypothesis and assumptions to understand the relevance of actions and results and identify potential gaps in the causal pathways). 2.4. What factors (i.e., size and type of CSO, level of pre-existing capacity, funding sources, etc.) were most conducive for success? 2.5. Did MADANI mainstream GESI into its work on accountability and tolerance? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 2.6. What are the key lessons learned from MADANI in terms of working with sub national CSOs to strengthen accountability and promote tolerance? 2.7. What are the barriers and opportunities to improve CSO capacity to strengthen accountability and promote tolerance? 3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1. How has capacity improvement led to financial sustainability for the different types of CSOs? Primary Data Collection: KIIs, Small Group and FGDs with all relevant stakeholders (i.e., MADANI staff & implementing partners, national, provincial & district support partners, lead CSO partners, LG officials, public and private sector donors and partners, etc. Secondary Data Collection: Project documents (i.e., signed MOUs, policies adopted, AMELP tools including performance indicator tracking tables, rapid assessments, baseline studies, mid-term self￾assessment, quarterly & annual reports, grant documents and workplans per each of the selected CSOs, etc. - Quantitative & qualitative analysis of data points - Method Triangulation (i.e., comparative analysis between any baseline assessments, perception surveys, etc. and data from interviews). - Source triangulation (i.e., comparative analysis of data from different stakeholders). - Evaluator triangulation (i.e., internal evaluation team discussions) - Descriptive Analysis & Causal Attribution (i.e., describe what change occurred and how and why it can be attributed to MADANI). - TOC approach (i.e., examine the intervention logic to assess MADANI’s hypothesis and assumptions to understand the relevance of actions and results and identify potential gaps in the causal pathways). 3.2. What is the likelihood that this improvement will be sustained after MADANI support ends? 3.3. What factors contribute to facilitating sustainability? (i.e., size of CSO; experience; reputation; popularity of staff; relationships; history with LG or other stakeholders; endorsement from reputable party; religious or political affiliation; etc.) 3.4. Which approaches or tools are most effective for improving the financial sustainability of CSOs? Why? 3.5. Did MADANI mainstream GESI into the design and implementation of activities related to resource mobilization and improving the enabling environment to improve the sustainability of the CSO partners? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 3.6. What are the key lessons learned from MADANI in terms of building the organizational capacity of CSOs to mobilize resources? 3.7. What are the barriers and opportunities to improve CSO capacity to mobilize resources? 4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1. In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? Primary Data Collection: Limited KIIs and group discussions with stakeholders due to late-stage change in the EQ. Additional discussions with - Qualitative analysis - Method Triangulation (i.e., comparative analysis between interview data, project documents, and secondary sources) - Source triangulation (i.e., 53 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV Primary EQ Sub-questions Data Collection Methods Data Analysis Approach MADANI staff to supplement KIIs. Secondary Data Collection: Project documents and secondary sources on CSO enabling environment in Indonesia comparative analysis of data from different stakeholders). - Evaluator triangulation (i.e., internal evaluation team discussions) - Descriptive Analysis & Causal Attribution (i.e., describe what change occurred and how and why it can be attributed to MADANI). - TOC approach (i.e., examine the intervention logic to assess MADANI’s hypothesis and assumptions to understand the relevance of actions and results and identify potential gaps in the causal pathways). USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 54 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS INTERVIEW TEMPLATE – MADANI STAFF Date: Location: Name of organization: Name / Job title of respondent(s): DC / Jakarta / or field based (Province / District): Gender of respondent(s): Name of interviewer(s): Intro Questions Response A. What is your primary role / responsibility? B. What are MADANI’s overarching achievements in relation to its overarching goal? ● Relationship level… ● Process level… ● Results / Outcome level… C. What are some of the major obstacles / challenges to achieving the intended results? D. What areas of MADANI can be improved? E. After two years of implementation, should MADANI’s “theory of change” (results framework) be modified? If so, how? Evaluation Questions Response EQ 1. Has MADANI improved the Organizational Capacity of its CSO partners? If so, how? If not, why not? 1.1 Which approaches or tools have been most effective for building organizational capacity? Why? Please provide examples. 1.2 Which approaches or tools have been least effective for building organizational capacity? Why? Please provide examples. 1.3 What enabling factors (conditions) are most conducive for improving organizational capacity? 1.4 How has MADANI mainstreamed Gender Equity and Social Inclusion (GESI) into organizational capacity development? Were there any positive or negative outcomes related to this? ● Has MADANI advanced women’s leadership and empowerment? ● Has MADANI promoted a GESI-sensitive organizational culture? ● Has MADANI set a GESI-relevant advocacy agenda through CSO networks? 1.5 How has MADANI coped with the pandemic? ● How has the pandemic affected MADANI’s effectiveness in delivering technical assistance? 1.6 What are the key lessons learned from MADANI in terms of building organizational capacity? 1.7 What are the barriers to and opportunities for improving CSO organizational capacity? EQ 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1 Has MADANI enabled CSOs to advocate for government accountability in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? 55 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV (i.e., MADANI activities improved LG responsiveness to CSO advocacy) ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.2 Has MADANI enabled its CSO partners to advocate for tolerance in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.3 Which social accountability tools or mechanisms were… ● Most effective? Why? ● Least effective? Why? 2.4 What factors (i.e., size and type of CSO, level of pre-existing capacity, funding sources, etc.) were most conducive for success in strengthening the LG’s accountability and / or promoting tolerance? 2.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into its work on accountability and / or tolerance? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 2.6 What are the key lessons learned from MADANI in terms of working with sub￾national CSOs to strengthen accountability and promote tolerance? 2.7 What are the barriers and opportunities to improve CSO capacity to strengthen accountability and promote tolerance? EQ3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1 Has MADANI improved the financial sustainability of its CSO partners? If so, how? 3.2 What is the likelihood that this improvement in sustainability will be sustained after MADANI support ends? 3.3 What factors contribute to facilitating financial sustainability? (i.e., size of CSO; experience; reputation; popularity of staff; relationships; history with LG or other stakeholders; endorsement from reputable party; religious or political affiliation; etc.) 3.4 Which approaches or tools have been most effective for improving the financial sustainability of your CSO? Why? What approaches or tools have been least effective? 3.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into the design and implementation of activities related to resource mobilization and improving the enabling environment to improve the sustainability of the CSO partners? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 3.6 What are the key lessons learned from MADANI in terms of building the organizational capacity of CSOs to mobilize resources? 3.7 What are the barriers and opportunities to improving CSO capacity to mobilize resources? EQ4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1 In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 56 INTERVIEW TEMPLATE – LEARNING FORUMS Date: Location (MADANI District / Village): Name of Learning Forum: Name(s) of respondent(s) / title / affiliation Gender of respondent(s): Name of interviewer(s): Evaluation Questions Response 1. Why and how did you and your organization get involved in the Learning Forum? 2. What is the purpose of the Learning Forum? 3. How does the Learning Forum operate? 4. What are some of the major achievements / outcomes of the Learning Forum? 5. What are some of the major obstacles / challenges? 6. How has the COVID-19 pandemic affected the Learning Forum? What has been done to overcome these challenges? 7. What changes would you make to improve the effectiveness of the Learning Forum? (General Recommendations) 8. Has Gender Equity and Social Inclusion (GESI) been mainstreamed into the Learning Forum? If so, how? If not, why not? ● Has MADANI advanced women’s leadership and empowerment? ● Has MADANI promoted a GESI-sensitive organizational culture? ● Has MADANI set a GESI-relevant advocacy agenda through CSO networks? 9. Has your participation in the Learning Forum changed your organization's policies and practices related to Gender Equity & Social Inclusion? If so, how? ● Has MADANI advanced women’s leadership and empowerment? ● Has MADANI promoted a GESI-sensitive organizational culture? ● Has MADANI set a GESI-relevant advocacy agenda through CSO networks? EQ1: Has your participation in the Learning Forum improved your organizational capacity? If so, how? If not, why not? EQ2. Has the Learning Forum contributed to enabling CSOs to advocate for improved local government accountability? If so, how? If not, why not? EQ2. Has the Learning Forum enabled CSO to advocate for tolerance in their communities? If so, how? If not, why not? EQ2. What are the barriers and opportunities to improve CSO capacity to strengthen accountability and promote tolerance? EQ3.Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? EQ3. Has the Learning Forum helped to improve the enabling environment for CSO engagement in local development processes? If so, how? If not, why not? EQ3. What are the barriers and opportunities to improve the enabling environment for CSO engagement in local development processes? EQ4. In what ways has MADANI improved the enabling environment for the CSOs engagement? 57 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV EQ4. In what ways has MADANI improved the enabling environment at local level to enable CSOs engagement in the local development process? EQ4. What kind of enabling policies and environment have been improved? EQ4. How has the experience and lesson learned of MADANI been working with sub national CSOs? What are the barriers and opportunities in improving CSOs' enabling environment? Additional Comments USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 58 INTERVIEW TEMPLATE - CSO PARTNERS Date: Name of CSO: Location of CSO: Name of District / Village where MADANI is being implemented: Theme / Focus of MADANI activities: Name / Title of respondent(s): Gender of respondent(s): Name of interviewer(s): Intro Questions Response A. How and why did your organization get involved with MADANI? B. What did your organization hope to achieve by participating in MADANI? C. What has your organization achieved by participating in MADANI? D. Based on your experience, what have been some of the challenges? E. Based on your experience, what areas of MADANI can be improved? Evaluation Questions Response EQ 1. Has MADANI improved the Organizational Capacity of its CSO partners? If so, how? If not, why not? 1.1 How has MADANI improved your organization’s effectiveness to achieve your organizational results? (Doing the right things) 1.2 How has MADANI improved your organization’s efficiency to manage core issues? (Doing the thing right) 1.3 What type of support has your CSO received from MADANI? Which of these approaches or tools have been effective for building your organization’s capacity? Why? What approaches or tools have been less effective? Why? 1.4 What do you believe has helped your CSO improve its capacity the most over time? In other words, what are the enabling factors (conditions) necessary for improving your CSO’s capacity? 1.5 Did collaboration with MADANI help mainstream Gender Equity and Social Inclusion (GESI) into your organizational capacity development? If so, how (i.e., tools, resources, training etc.)? If not, why not? Were there any positive or negative outcomes related to this? ● Has collaboration with MADANI helped your CSO to advance women’s leadership and empowerment in your organization? ● Has collaboration with MADANI helped your CSO to promote a GESI-sensitive organizational culture? ● Has collaboration with MADANI helped to set a GESI-relevant advocacy agenda through your CSO networks? 1.6 How has MADANI coped with the pandemic? How has the pandemic affected MADANI’s effectiveness in delivering technical assistance? 1.7 What are the key lessons learned from MADANI in terms of building organizational capacity? 1.8 Based on your experience, what are the barriers to and opportunities for improving CSO organizational capacity? EQ 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1 Has your CSO advocated for government accountability in your communities? Please explain. 59 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ● If so, what type of support has MADANI provided to strengthen your CSOs capacity to advocate for government accountability? (What social accountability tools or mechanisms were introduced by MADANI?) Were they helpful? If so, how? If not, why not? ● What are some of your CSO’s major achievements (outputs and outcomes) as a result of MADANI’s support? (i.e., MADANI activities improved LG responsiveness to CSO advocacy). ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.2 Has your CSO advocated for tolerance and social inclusion in your communities? Please explain. ● If so, what type of support has MADANI provided to strengthen your CSOs capacity to advocate for tolerance? (What tools or mechanisms were introduced by MADANI?) Were they helpful? If so, how? If not, why not? ● What are some of your CSO’s major achievements (outputs and outcomes) in relation to tolerance as a result of MADANI’s support? ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes in relation to tolerance? 2.3 What factors (i.e., size and type of CSO, level of pre-existing capacity, funding sources, etc.) were most conducive for success in strengthening the LG’s accountability and/or promoting tolerance in your communities? ● Did MADANI’s support focus on strengthening these factors? 2.4 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into its support of your CSO’s work on accountability and/or tolerance? If so, how? Were there any positive or negative outcomes related to this? 2.5 What are the key lessons learned from your experience working with MADANI in terms of CSO’s roles and capacity to strengthen LG accountability and promote tolerance? 2.6 What are the barriers and opportunities to improve CSO capacity to strengthen accountability and promote tolerance? EQ3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1 Has MADANI improved the financial sustainability of your CSO? If so, how? ● Has MADANI also helped to build your CSOs capacity in financial management? 3.2 What is the likelihood that this improvement will be sustained after MADANI support ends? 3.3 What factors contribute to facilitating financial sustainability? (i.e., size of CSO; experience; reputation; popularity of staff; relationships; history with LG or other stakeholders; endorsement from reputable party; religious or political affiliation; etc.) 3.4 Which approaches and tools provided by MADANI have been most effective for improving the financial sustainability of your CSO? Why? What approaches or tools have been least effective? 3.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into the design and implementation of activities related to resource mobilization and improving the enabling environment to improve the sustainability of the CSO partners? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 3.6 What are the key lessons learned from MADANI in terms of building the organizational capacity of CSOs to mobilize resources? 3.7 What are the barriers and opportunities to improve CSO capacity to mobilize resources? USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 60 EQ4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1 In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? 61 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV INTERVIEW TEMPLATE – LOCAL GOVERNMENT Date: Location (MADANI District / Village): Name of LG Office: Theme / Focus of MADANI activities in this district: Name / Title of respondent(s): Gender of respondent(s): Name of interviewer(s): Intro Questions Response A. How and why did your office get involved with MADANI? B. What did your LG hope to achieve by participating in MADANI? C. What has your LG achieved by participating in MADANI? D. Based on your experience, what have been some of the challenges? E. Based on your experience, what areas of MADANI can be improved? Evaluation Questions Response EQ 1. Has MADANI improved the Organizational Capacity of its CSO partners? If so, how? If not, why not? EQ 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1 Has MADANI enabled CSOs to advocate for government accountability in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? (i.e., MADANI activities improved LG responsiveness to CSO advocacy) ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.2 Has MADANI enabled its CSO partners to advocate for tolerance in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.3 Which social accountability tools or mechanisms were most effective and why? 2.4 Does the LG find CSOs helpful in supporting them to improve accountability and promote tolerance? What matters most for LGs trusting CSOs to support their work in improving accountability and promoting tolerance? What factors (i.e., size and type of CSO, level of pre-existing capacity, funding sources, etc.) are most conducive for success in strengthening the LG’s accountability and/or promoting tolerance? 2.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into its work on accountability and/or tolerance? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 2.6 What are the key lessons learned from MADANI in terms of working with sub￾national CSOs to strengthen LG accountability and promote tolerance? 2.7 What are the barriers and opportunities to improve CSO capacity to strengthen LG accountability and promote tolerance? EQ3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1 Are the MADANI CSOs actively involved in local development processes? If yes, how? If not, why not? Are there any regulations or planning documents issued based on MADANI’s CSO recommendations? Has MADANI contributed to policies, processes, or other factors that improve the enabling environment for CSO engagement in local development processes? If so, how? If not, why not? 3.2 What are the key lessons learned from MADANI in terms of improving the enabling environment for CSO engagement in local development processes? 3.3 What are the barriers and opportunities to improve the enabling environment for CSO engagement in local development processes? USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 62 EQ4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1 In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? 63 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV INTERVIEW TEMPLATE – USAID Date: Location: Name of organization: Name / Job title of respondent(s): DC / Jakarta / or field based (Province / District): Gender of respondent(s): Name of interviewer(s): Intro Questions Response A. What is your primary role / responsibility? B. What are MADANI’s overarching achievements in relation to its overarching goal? C. What are some of the major obstacles / challenges to achieving the intended results? D. What areas of MADANI can be improved? E. After two years of implementation, should MADANI’s “theory of change” (results framework) be modified? If so, how? Evaluation Questions Response EQ 1. Has MADANI improved the Organizational Capacity of its CSO partners? If so, how? If not, why not? 1.1 Which approaches or tools have been most effective for building organizational capacity? Why? Please provide examples. 1.2 Which approaches or tools have been least effective for building organizational capacity? Why? Please provide examples. 1.3 What enabling factors (conditions) are most conducive for improving organizational capacity? 1.4 How has MADANI mainstreamed Gender Equity and Social Inclusion (GESI) into organizational capacity development? Were there any positive or negative outcomes related to this? 1.5 How has MADANI coped with the pandemic? How has the pandemic affected MADANI’s effectiveness in delivering technical assistance? 1.6 What are the key lessons learned from MADANI in terms of building organizational capacity? 1.7 What are the barriers to and opportunities for improving CSO organizational capacity? EQ 2. Has MADANI strengthened the local governments’ accountability and promoted tolerance? If so, how? If not, why not? 2.1 Has MADANI enabled CSOs to advocate for government accountability in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? (i.e., MADANI activities improved LG responsiveness to CSO advocacy) ● Have any processes been established between CSOs and LGs that could result in future outputs and outcomes? 2.2 Has MADANI enabled its CSO partners to advocate for tolerance in their communities? If so, how? If not, why not? ● What are some of the major achievements (outputs and outcomes)? ● Have any processes been established between CSOs and LGs that could USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 64 result in future outputs and outcomes? 2.3 Which social accountability tools or mechanisms were… ● Most effective? Why? ● Least effective? Why? 2.4 What factors (i.e., size and type of CSO, level of pre-existing capacity, funding sources, etc.) are most conducive for success in strengthening the LG’s accountability and/or promoting tolerance? 2.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into its work on accountability and/or tolerance? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 2.6 What are the key lessons learned from MADANI in terms of working with sub￾national CSOs to strengthen LG accountability and promote tolerance? 2.7 What are the barriers and opportunities to improve CSO capacity to strengthen LG accountability and promote tolerance? EQ3. Has MADANI improved the sustainability of targeted CSOs in bringing about change in policy and practice? If so, how? If not, why not? 3.1 Has MADANI improved the financial sustainability of its CSO partners? If so, how? 3.2 What is the likelihood that this improvement in sustainability will be sustained after MADANI support ends? 3.3 What factors contribute to facilitating financial sustainability? (i.e., size of CSO; experience; reputation; popularity of staff; relationships; history with LG or other stakeholders; endorsement from reputable party; religious or political affiliation; etc.) 3.4 Which approaches or tools have been most effective for improving the financial sustainability of your CSO? Why? What approaches or tools have been least effective? 3.5 Did MADANI mainstream Gender Equity and Social Inclusion (GESI) into the design and implementation of activities related to resource mobilization and improving the enabling environment to improve the sustainability of the CSO partners? If so, how? If not, why not? Were there any positive or negative outcomes related to this? 3.6 What are the key lessons learned from MADANI in terms of building the organizational capacity of CSOs to mobilize resources? 3.7 What are the barriers and opportunities to improving CSO capacity to mobilize resources? EQ4. Has MADANI improved the enabling environment for the CSOs engagement? If so, how? If not, why not? 4.1 In what ways has MADANI improved the enabling environment at the local level to enable CSOs engagement in the local development process? 65 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX V: SOURCES OF INFORMATION DOCUMENTS REVIEWED MADANI Work Plans and Reports (in topic and time sequence) ▪ Activity Monitoring, Evaluation, and Learning Plan (AMELP), ver. July 12, 2019 ▪ Activity Monitoring, Evaluation, and Learning Plan (AMELP), ver. February 12, 2021 ▪ Activity Monitoring, Evaluation, and Learning Plan (AMELP), ver. December 1, 2021 ▪ Implementation Plan, FY 2019, ver. May 3, 2019 ▪ Implementation Plan, FY 2020, ver. September 27, 2019 ▪ Implementation Plan, FY 2021, ver. September 23, 2020 ▪ Implementation Plan, FY 2022, ver. September 22, 2021 ▪ Quarterly Progress Report Q2, FY 2019, ver. April 29, 2019 ▪ Quarterly Progress Report No.2, FY 2019, ver. July 30, 2019 ▪ Annual Progress Report, FY 2019, ver. January 2, 2020 ▪ Quarterly Performance Report Q1, FY 2020, ver. January 30, 2020 ▪ Quarterly Report Q2, FY 2020, ver. May 22, 2020 ▪ Quarterly Performance Report Q3, FY 2020, ver. August 18, 2020 ▪ Annual Progress Report, FY 2020, ver. December 10, 2020 ▪ Quarterly Performance Report Q1, FY 2021, ver. January 29, 2021 ▪ Quarterly Performance Report Q2, FY 2021, ver. April 29, 2021 ▪ Quarterly Performance Report Q3, FY 2021, ver. September7, 2021 ▪ Annual Progress Report, FY 2021, ver. December 28, 2021 ▪ Quarterly Performance Report Q1, FY 2022, ver. March 4, 2022 ▪ Quarterly Performance Report Q2, FY 2022, ver. April 29, 2022 ▪ Performance Indicator Tracking Table, FY 2020 ▪ Performance Indicator Tracking Table, FY 2021 ▪ Performance Indicator Tracking Table, FY 2022 MADANI Technical Documents (in time sequence) ▪ Associate Cooperative Agreement MADANI between USAID and FHI360, 4 February 2019 ▪ Rapid Assessment on Tema Toleransi dan Rekomendasi Program untuk Adopsi atau Adaptasi oleh MADANI, by Lilis Nurul Husna, Bahasa Indonesia version, July 2020 ▪ 1st Organizational Performance Index Report of 32 MADANI Lead Partners, Year 2020 ▪ Summary of OPI Report 2020, September 2020 ▪ MADANI Baseline Study Final Report, by REDI, November 2020 ▪ Panduan Fasilitasi Simpul Belajar MADANI: Saling Belajar dan Aksi Bersama, Bahasa Indonesia version, December 2020 ▪ Laporan Survei Warga Layanan Pemerintah Selama Pandemi COVID-19, by INFID, Bahasa Indonesia version, FY 2020 ▪ MADANI Social Network Analysis Inception Report (in Makassar and Wonosobo), January 2021 ▪ Analysis of Results of Initial Round of Lead Partners’ Organizational Performance Index (OPI) Self-Assessments, March 2021 ▪ Panduan Fasilitator Pengukuran Indeks Kinerja Organisasi, Bahasa Indonesia version, July 2021 ▪ Laporan Singkat Hasil Survei Persepsi Masyarakat Terhadap 32 Mitra Utama MADANI, Bahasa Indonesia version, July 2021 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 66 ▪ Penggalangan Sumber Daya OMS Mitra Utama MADANI: Hasil Pemetaan Cepat, July 2021 ▪ Scoping Study: Rising Intolerance and Universal Access to Public Services in Indonesia, by PUSAD Paramadina, August 2021 ▪ MADANI Programmatic Study on Resource Mobilization and Financial Sustainability of Local CSOs in Indonesia, by Indonesia untuk Kemanusiaan (IKa), September 2021 ▪ 2nd Organizational Performance Index Report of 32 Madani Lead Partners, Year 2021 ▪ Analysis of First Round of MADANI’s Forum Performance Index in Banten, West Java, Central Java, East Java, South Sulawesi and West Kalimantan, FY 2021 ▪ Studi Inventarisasi Pengalaman OMS dalam Program Self-Regulation, Self-Registration, dan Self Certification di Indonesia, by Rustam Ibrahim, Bahasa Indonesia version, FY 2021 ▪ iHub/Simpulmadani as A Learning and Resources Platform for Civil Society in Indonesia: Year 2021-2022 Road Map ▪ Analysis of MADANI’s Local CSO Partners’ Resources Mobilization Assessment, by Prio Sambodho, March 2022 MADANI Publications (in topic and time sequence) ▪ MADANI District Walkabouts January-April, October-December 2020 ▪ Fact Sheet, ver. 2022 ▪ Newsletter Vol.1, January - April 2020 ▪ Newsletter Vol.2, May - July 2020 ▪ Newsletter Vol.3, August - October 2020 ▪ Newsletter COVID-19 Edition, September 2020 ▪ Newsletter Vol.4, November 2020 - January 2021 ▪ Newsletter Vol.5, February - April 2021 ▪ Newsletter Vol.6, May - July 2021 ▪ Newsletter Vol.7, August - October 2021 ▪ Newsletter Vol.8, November 2021 - January 2022 ▪ Three Impact Stories Year 2021 ▪ Working Paper 1. Collaborative Governance in Strengthening Accountability and Tolerance in Decentralized Indonesia – A MADANI Program Assessment, by Prio Sambodho, July 2020 ▪ Working Paper 2. MADANI Assessment on Youth in Civil Society, by Lia Toriana, March 2022 ▪ Working Paper 3. Studi Keberlanjutan Multi-Stakeholder Forum: Identifikasi Faktor Pendukung dan Penghambat, by Godril D. Yuwono, Bahasa Indonesia version, March 2022 ▪ USAID Reference Materials ▪ Province/District in Figures of Regional Statistical Bureaus 67 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV LIST OF RESPONDENTS No. Format Respondent Type Institution Gender Female Male BANTEN Tangerang District 1 Group Interview Lead Partner FOPKIA 0 2 2 Group Interview District Support Partner Perekat Demokrasi 0 2 3 Group Interview Learning Forum Fatayat NU 2 2 IBI Tangerang District FKSPN Visi Nusantara 4 Group Interview Community Kampung Melayu Barat Village - Midwife and cadres 9 0 Tegal Angus Village - Cadres 5 Group Interview Local Government Kampung Melayu Barat Village - Head of village 0 2 Tegal Angus Village - Head of village 6 KII Local Government District Health Office 1 0 7 KII Implementing Partner FHI360 - Field Coordinator 1 0 Serang District 8 Group Interview Lead Partner PD Aisyiyah 6 1 9 KII District Support Partner Pattiro 1 0 10 Group Interview Learning Forum PD Nasyiatul Aisyiyah 1 1 Learning Forum ICMI 11 KII Implementing Partner FHI360 - Field Coordinator 0 1 12 Group Interview Local Government Bakesbangpol 0 2 13 KII Local Government Bappeda 0 1 WEST JAVA Bandung District 14 Group Interview Lead Partner SAPA Institute 2 2 15 KII Learning Forum KPI of Bandung District 1 0 16 Group Interview Provincial Support Partner BTRUST 0 5 Bogor City 17 Group Interview Community Babakan Pasar Village - Cadres 2 6 Babakan Pasar Village - Karang Taruna PIK-R Local Government Babinkamtibnas / TNI Bhabinmas / POLRI 18 KII Local Government Kesbangpol 0 1 19 Group Interview Learning Forum LCl YPD 1 3 Kokema 20 Group Interview District Support Partner BASOLIA 1 4 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 68 No. Format Respondent Type Institution Gender Female Male 21 KII Implementing Partner FHI360 - Field Coordinator 0 1 22 Group Interview Lead Partner Metamorfosis 4 1 23 KII Local Government Babakan Pasar Village - Head 0 1 Garut District 24 KII Local Government Bappeda 0 1 25 KII Private Sector Lazismu 0 1 26 Group Interview Lead Partner PD Nasiyiatul Aisyiyah 4 1 27 Group Interview Learning Forum FAASIH Garut 1 2 28 KII Local Government DPMD 0 1 29 Group Interview Local Government Puskesmas Haurpanggung 2 0 30 Group Interview Community Haurpanggung Village - Cadres Local Government 5 1 Haurpanggung Village - Secretary of village Community Sakina Rapih Village - Cadres 31 KII Implementing Partner FHI360 - Field Coordinator 0 1 CENTRAL JAVA Surakarta City 32 Group Interview Lead Partner KOMPIP 1 1 33 Group Interview Learning Forum KOMBES 0 2 LPTP Solo 34 KII Implementing Partner FHI360 - Field Coordinator 0 1 35 KII Local Government Bappeda 0 1 36 Group Interview Local Government Bakesbangpol 1 2 37 KII Community PEKKA Mojosongo Urban Village 0 1 Sukoharjo District 38 Group Interview Provincial Support Partner YSKK Jateng 1 1 Boyolali District 39 Group Interview Learning Forum FORMMAD 1 2 40 Group Interview Lead Partner LKTS 1 1 41 Group Interview Local Government BP3D 0 2 42 KII Implementing Partner FHI360 - Field Coordinator 0 1 Wonosobo District 43 KII Learning Forum Head / Mafindo 1 0 44 Group Interview Lead Partner KITA Institute 2 2 45 KII Implementing Partner FHI360 - Field Coordinator 0 1 46 KII Local Government Bappeda 0 1 47 Group Interview Local Government Puskesmas Wonosobo 1 4 1 48 KII Community Forum Warga 0 1 EAST JAVA Jember District 69 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV No. Format Respondent Type Institution Gender Female Male 49 Group Interview Learning Forum STAPA Center Fatayat NU 2 1 YPSM 50 Group Interview Lead Partner GPP 5 0 51 KII Implementing Partner FHI360 - Field Coordinator 1 0 52 Group Interview Local Government Puskesmas Sumbersari 2 0 53 KII Local Government DP3AKB 0 1 Jombang District 54 KII Local Government Bappeda 0 1 55 Group Interview Learning Forum Bank Sampah Induk 2 0 KPI 56 Group Interview Local Government District Environment Office 0 2 57 Group Interview Lead Partner Sanggar Hijau Indonesia 3 1 58 KII Implementing Partner FHI360 - Field Coordinator 1 0 59 Group Interview Community Kaliwungu Urban Village - Environment cadres 10 3 Malang District 60 Group Interview Learning Forum Pattiro Malang 0 2 Implementing Partner FHI360 - Field Coordinator 61 KII Local Government Bakesbangpol 1 0 62 KII Local Government District Health Office 0 1 63 KII Provincial Support Partner LPKP Jatim 0 1 64 Group Interview Lead Partner Yayasan Paramitra 1 2 WEST KALIMANTAN Singkawang City 65 Group Interview Learning Forum Education Board STKIP Singkawang 1 2 Forum Peduli Kesehatan 66 KII Local Government District Education Office 0 1 67 KII Local Government Bakesbangpol 1 0 68 KII Local Government Bappeda 0 1 69 Group Interview Lead Partner PKBI Singkawang 2 2 Pontianak 70 Group Interview Implementing Partner FHI360 - SFC & MERCO 1 1 71 KII Provincial Support Partner Yayasan Dian Tama 1 0 Mempawah District 72 Group Interview Local Government Diskominfo 2 0 73 Group Interview Local Government Bappeda 1 2 74 Group Interview Private Sector Bank KalBar 1 2 75 Group Interview Learning Forum Fatayat NU 0 4 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 70 No. Format Respondent Type Institution Gender Female Male PPDI PDDRU MMC 76 Group Interview Lead Partner PD Aisyiyah 6 0 SOUTH SULAWESI Pangkajene Kepulauan District 77 Group Interview Lead Partner LEKRAC 1 1 78 Group Interview Learning Forum MABACA 1 1 79 Group Interview Local Government Midwife / Puskesmas 3 0 Community Cadre & Village Facilitator Bulukumba District 80 Group Interview Lead Partner PD Nasyiatul Aisyiyah 4 0 81 Group Interview Learning Forum FORMAP KIA 0 3 82 Group Interview Community Salasae Village - Cadre 4 0 83 KII Local Government Bappeda 0 1 84 KII Implementing Partner FHI360 - SFC 0 1 85 Group Interview Private Sector BAZNAS 1 1 Makassar City 86 KII Local Government Bakesbangpol 0 1 87 Group Interview Local Government Bappeda 2 2 88 Group Interview Learning Forum KPI Sulsel 1 1 Peace Generation Makassar 89 Group Interview District Support Partner YASMIB Sulawesi 2 1 90 Group Interview Lead Partner ICJ Makassar 2 1 91 Group Interview Learning Forum Forum Pemuda 0 2 Forum Tiba 92 KII Provincial Support Partner PSP / LSKP Sulsel 0 1 DKI JAKARTA 93 KII USAID USAID - Director of DRG 0 1 94 KII Implementing Partner FHI360 - COP 0 1 95 Group Interview Implementing Partner FHI360 - DCOP & Technical Director 1 1 96 KII USAID USAID - AOR of MADANI 0 1 97 Group Interview Implementing Partner FHI360 - Staff 2 2 98 Group Interview National Service Provider Gusdurian 2 1 99 Group Interview National Service Provider Remdec Swaprakarsa 1 2 100 Group Interview National Government Ministry of Law and Human Rights 2 4 ONLINE 101 KII Implementing Partner FHI360 - Staff 0 1 102 Group Interview National Service Provider ICW 2 2 103 KII Implementing Partner FHI360 - Staff 0 1 71 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV No. Format Respondent Type Institution Gender Female Male 104 KII National Service Provider SATU NAMA 0 1 105 Group Interview National Service Provider GESI Consultant 2 1 106 Group Interview National Service Provider ATMA Connect 1 1 107 KII National Service Provider Yappika 0 1 108 KII National Service Provider IDEA 1 0 109 Group Interview National Service Provider Women Research Institute (WRI) 2 0 110 KII National Service Provider GESI Consultant 1 0 Total by Gender 138 135 Grand Total 273 USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 72 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 73 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 74 75 | MADANI MIDTERM PERFORMANCE EVALUATION USAID.GOV USAID.GOV MADANI MIDTERM PERFORMANCE EVALUATION | 76 ANNEX VII: EVALUATION TEAM MEMBERS Summary information about evaluation team members, including qualifications, experience, and role on the team. A four-person team will carry out the evaluation activities. The MEL-P team will provide additional support related to technical guidance, logistics, and quality oversight of data collection, analysis, and reporting. MEL-P Program Director, Irene Velez, served as an evaluation advisor and provided technical input and support in drafting the summary brief and evaluation report. The specific qualifications and roles anticipated for each evaluation team member are listed below. Mr. Peter Bauman served as the Team Leader. He is responsible for overall direction, writing, analysis, and interface with USAID and FHI360. Equipped with a MA in Conflict Resolution and Coexistence and over 15 years of experience, Peter has worked with a broad range of public and private institutions on complex and politically sensitive programs and processes worldwide. He has extensive experience designing, managing, and evaluating programs in the areas of conflict resolution, peacebuilding, civil society and local governance, land & natural resource management, environmental conservation, stabilization, countering violent extremism, human rights, economic development, and humanitarian relief & recovery. He also has significant experience conducting multidisciplinary analyses and applied field-based research in developing and conflict-prone environments across Africa, Asia, Europe, and the Middle East. Mrs. Dianty Ayu served as the Local Governance Expert for the evaluation team. Equipped with an MBA and over 10 years of experience, Dianty has worked with a broad range of national and international organizations on topics related to local governance, social development, and poverty reduction. She has participated in several evaluations and studies for the World Bank in Indonesia mainly in the fields of decentralization, sub-national and village government capacity, and LG accountability. Dianty is currently working as a consultant assisting Bappenas in the development of web and android applications to assist local and village governments in better planning using valid data and analysis. To date she has contributed to the improvement of the village information system (SID), community-based monitoring, social economy registry reform and better targeting for the national social assistance program. Dr. Saut Sagala served as the Civil Society Expert for the evaluation team. Equipped with a PhD in Urban Planning and Management and over 20 years of experience, Saut is a researcher & lecturer at the School of Planning and Policy Development, Institute of Technology Bandung (ITB). His interdisciplinary academic and applied work focuses on a broad range of topics including national and local governance, policy analysis, monitoring and evaluation, natural resources and the environment, sustainable development, and vulnerable groups (i.e., disability, LGBTQ, elderly, marginality, displaced people). Mr. Basyrah Alwi served as the M&E Specialist / Research Assistant for the evaluation team. Equipped with a BA in Anthropology and over 14 years of experience, Alwi will support the data collection design, implementation, analysis, and report writing. He has worked with a broad range of national and international organizations on topics related to health, education, and livelihood issues at the community level. Alwi has supported many program evaluations throughout Indonesia, and he is a member of the Indonesian Development Evaluation Community. U.S. Agency for International Development / Indonesia Jl. Medan Merdeka Selatan 3-5 DKI Jakarta, Indonesia