1 USAID Regional Governance Activity (RGA) in Colombia, 2015-2020: Endline Report1 Authors Michael G. Findley Professor, University of Texas at Austin Member, AidData Research Consortium p: 512.749.3815 e: mgfindley@gmail.com Oliver Kaplan Associate Professor, University of Denver p: 303.871.2377 e: oliver.kaplan@du.edu Ana Carolina Marrugo Ph.D. Student, University of Pittsburgh p: 202.770.7941; e: acm189@pitt.edu Alejandro Ponce de León Ph.D. Candidate, University of California, Davis p: 530.761.3221 e: poncedeleon@ucdavis.edu Daniel Walker Independent Data Scientist p: 661.607.8606; e: dannywalker101@gmail.com Joseph K. Young Professor, American University Member, AidData Research Consortium p: 850.284.7192 e: jyoung@american.edu 30 April 2021 1 We thank María Camila Alarcón, Katie Dunn, Mike Denly, Erin Kearns, Nathalia Rojas, Natalie Southwick, staff at the USAID mission, implementing partner Management Systems International, and the Innovation for Peace and Development research team at the University of Texas at Austin for their helpful collaboration, assistance, and comments. 2 Table of Contents Acronyms 3 Abstract 4 Executive Summary 5 RGA Program Background 5 Summary of Evaluation Approach 5 Evaluation Findings 6 Evaluation Purpose 9 Activity Description and Background: Implementation of the RGA 10 Evaluation Questions and Methodology 14 Evaluation Findings and Conclusions 15 Context: Leading up to the Endline, and Trends 17 A Deteriorating Security Situation Made Implementation Difficult 18 Steady Skepticism about the Peace Process 22 Illegal Economies: A Source of Livelihoods 23 Component 2: Initial, Varied Improvement in Financial Management and Public Attitudes 24 Total Revenues Increased Over Time 25 RGA Municipalities Obtained Greater Royalty Revenues 26 Elites are More Optimistic about Tax Collection than Citizens Perceptions 28 Performance: Limited Improvement in Service Delivery 30 Tumaco: A Singular Case? 32 Component 4: Increased Citizen Participation 33 Declining Performance of Mayors 33 General Declines in Perceived Corruption 37 Limited RGA Effects on Participation 39 Citizen Oversight Groups (Veedurías) 40 Unclear Impacts or Null Results 44 Region-Specific Dynamics 45 Vulnerable Populations 49 Evaluation Recommendations and Conclusion 52 Conclusions 53 Recommendations for Policy and Programming 55 3 Acronyms AGC – Gaitanista Self-Defense Forces ASOCOMUNAL - Municipal Association of Junta Councils BACRIM - Criminal Bands CAQDAS - Computer-assisted qualitative data analysis software CBPS - Covariate Balancing Propensity Score CDA - Centro Democrático Alternativo CDCS - Country Development Cooperation Strategy CEDE - Centro de Estudios sobre Desarrollo Económico CONPES - The National Council for Economic and Social Policy DCOF - Displaced Children and Orphans Fund DNP – Department of National Planning ELN - The National Liberation Army FARC - The Revolutionary Armed Forces of Colombia GOC - Government of Colombia IED - Improvised Explosive Device IR - Intermediate Results JAC - Juntas de Acción Comunal JEP – Special Jurisdiction for Peace LGBTQ - Lesbian, Gay, Bisexual, Transgender, and/or Queer LWVF - Leahy War Victims Fund MSI - Management Systems International PDET - Territorially Focused Development Programs RGA - Regional Governance Activity SENA - Servicio Nacional de Aprendizaje SGP - Sistema General de Participaciones SGR - Sistema General de Regalías USAID - United States Agency for International Development VOT - Victims of Torture Program 4 Abstract The Colombia Regional Governance Activity (RGA) funded by USAID Colombia Mission, in partnership with the Government of Colombia (GOC), aimed to improve sub-national governance in 40 conflict￾affected municipalities of Colombia. RGA occurred from 2015-2021 with the evaluation period spanning 2015-2020. The RGA program was implemented by Management Systems International (MSI), a TetraTech Company. The program consisted of five components, and we were tasked by USAID only to evaluate, the extent to which RGA had an impact in two component areas: (Component 2) Improved financial management and performance of targeted municipalities and (Component 4) Increased citizen participation for enhanced transparency and accountability. The results of the evaluation are highly mixed with some results suggesting initial positive impacts, but most of the results suggesting little (or unclear) impact. The security conditions in Colombia deteriorated over the life of the program, across RGA and non-RGA municipalities alike, which made the impact of RGA much more difficult to achieve and detect. 5 Executive Summary RGA Program Background The Colombia Regional Governance Activity (RGA) funded by USAID Colombia Mission, in partnership with the Government of Colombia (GOC), aimed to improve sub-national governance in 40 conflict￾affected municipalities of Colombia over the period 2015-2021 with the evaluation period spanning 2015- 2020.2 The RGA program began at a time of great optimism and continues as the implementation of the peace accord signed with the FARC insurgent group in 2016 advances amid difficulties. Negotiations were initiated between the Government and the ELN insurgents but were broken off and remain uncertain. The RGA program is implemented by Management Systems International (MSI), a TetraTech Company. The program consisted of five components, two of which we were tasked explicitly by USAID to evaluate, and that provided the basis for the endline evaluation: 2. Improved financial management and performance of targeted municipalities; 4. Increased citizen participation for enhanced transparency and accountability; The RGA also aims to increase the inclusion of women and vulnerable groups through a set of activities designed to address gender, ethnic and racial barriers to participation, including strengthening the capacity of the GOC to respond to the needs of these populations. Summary of Evaluation Approach Based on the theory of change advanced by USAID, the RGA program should increase municipal capacity and performance as well as citizen participation, which were lacking in Colombia’s conflict-affected regions, at least at the outset of the RGA. As capacity and legitimacy increase, consequently, the population should become more supportive of the government and less supportive of illegal armed groups. These interventions and expected outcomes fit within the USAID Colombia mission’s goal of making Colombia more capable of successfully implementing a sustainable and inclusive peace through the intermediate goals of more effectively delivering services prioritized by citizens (IR 1.1) and increasing citizen participation in democratic processes and governance (IR 1.2). All intermediate and end results advance the objectives outlined in the Country Development Cooperation Strategy 2014- 2020 (CDCS). Although the program is nearing completion, some elements of the RGA’s theory of change require time to achieve, so it may still be premature to expect substantial changes in outcomes. To evaluate impact, we identified a set of comparison units by conducting a statistical “matching” exercise prior to the baseline data collection.3 This exercise identified a set of control municipalities that were 2 This research was made possible through support from USAID Colombia provided via cooperative agreement AID-OAA-A-12-00096 between AidData at the College of William and Mary and the USAID Global Development Lab. The views expressed here do not necessarily reflect the views of AidData, USAID, or the United States Government. The endline evaluation occurred prior to the conclusion of the project only because the project was extended. The endline evaluation was scheduled and started before the extension of the program itself. 3 We briefly summarize the key design features of the evaluation here. We also discuss them more extensively in Annex 2. And full design details appear in the reports for earlier waves of the evaluation. The evaluation approach and methods were circulated to the USAID mission and the implementing partner, discussed extensively, and ultimately approved well ahead of the baseline data collection. Our approach at endline is consistent with what was approved at the outset. 6 similar to the treatment municipalities in all possible ways except that treatment municipalities would receive the RGA intervention and the control municipalities would not. In the baseline we demonstrated that the municipalities were indeed balanced. Since the matching approach was successful at baseline, at the endline stage, any differences that emerge between treatment and control municipalities should be attributable to the RGA intervention and not to other potentially confounding factors (to the extent that we can account for competing explanations and other potential interventions in the region). In this endline, we conducted a very similar set of evaluation activities across three separate waves. We conducted 6,692 citizen surveys at endline, meaning 20,002 surveys across three waves; 290 in-depth elite interviews at endline, meaning 1,193 interviews across three waves; 20 focus groups at endline, for a total of 60 focus groups across three waves; and substantial administrative data collection and analysis.4 The statistical analysis estimates the determinants of endline outcomes using RGA activities weighted by the intensity of implementation across municipalities (the “treatment effect on the treated”), controlling for baseline values of outcome data.5 We also computed the results in other ways, including using difference￾in-differences statistical analysis, which considers the difference in outcomes at endline (between treatment and control) relative to the difference in baseline measures for outcomes (between treatment and control). We report the key results in graphs, and discuss statistical meaningfulness in the text. Full statistical results are reported in Annex 2d, Table A2.9. Evaluation Findings Our impact evaluation of RGA tells a story of an ambitious program set against a difficult implementation environment, resulting in some initial modest improvements, with most areas either unaffected or, in a few rare cases, negatively affected. The quantitative results yield only a modest number of statistically meaningful relationships. Qualitative information from the municipalities also provides helpful lessons and clarifies some of the relationships. Our analysis provides several key findings on the anticipated outcomes of the two RGA program components being evaluated: Municipal Financial Management and Performance ● At the endline, RGA treatment municipalities experienced slightly greater overall and property revenues relative to control municipalities., and higher royalty revenues at endline relative to midline (though not to baseline). Expenditures on those royalty revenues showed some small increases in treatment municipalities relative to controls from midline to endline, which are indicators of possible positive RGA effects. ● Treatment municipalities demonstrated improvements in fiscal performance relative to control municipalities at the midline (2015-2017), but those effects attenuated at endline (2017-2019). 4 The data were collected in the same areas as the baseline, and wherever possible we attempted to survey the same individuals. We were able to conduct repeat surveys with a significant proportion of the initial survey respondents who participated in earlier waves. As with most panels, however, it was difficult to re-contact some subjects. When we could not re-contact the exact individuals, we re-sampled based on a sampling strategy discussed in Annex 4. Of those that participated in the endline, 1,914 participated in both the baseline and midline as well; 1,320 also participated in either the baseline or the midline (but not both); 754 were different persons but in the same household; 582 were a new person or family but in the same house structure; 2,117 were new persons altogether but on the same street/block as an individual from the baseline; and 5 were the same person who responded in the baseline or midline, but who had moved to an entirely different municipality. 5 We weighted the results by the numbers of people treated, by a population-based treatment measure, numbers of treated, and other ways. The results we reported are largely consistent across these weighting approaches. 7 ● Citizen perceptions indicate steady access to services and only few perceived changes in access. The sole exception is that people report greater access to gas services in treatment moving from baseline to endline, which was not an explicit RGA target and only partially determined at local levels, but possibly could have been catalyzed through the mobilization of royalty resources or another mechanism such as indirect engagement of private service providers. ● Effects for outcomes such as municipal service provision may only obtain after the program and endline data collection. Such outcomes are slower to be influenced and detected, since they depend on lengthier implementation and governing processes and greater citizen awareness. ● Municipal administrators in treatment municipalities improved their understanding of the royalties system (SGR) thanks to technical training and standardization of processes. This has positively affected how municipalities apply for project approvals, implement projects, and receive funding. ● Community leaders in treatment municipalities feel that investments are more visible but do not necessarily feel their development needs are addressed. ● Perception of transparency remained low, especially among non-elite community members. Citizens indicate mayors do not adequately inform communities about their activities and expenditures of tax payers’ contributions and royalties. ● Community leaders and municipal administrators referenced improvements in the tax culture in treatment municipalities, with community leaders suggesting that people are more willing to pay taxes and local governments encouraging payments through educational campaigns, better communication, and payment arrangements. ● RGA programming is associated with almost no change in citizen-level attitudes about payment of taxes, which can be critical for generating support for public revenues and works. Citizen Participation ● Levels of social participation (not trends) remain similar across all RGA and comparison municipalities. ● Although there were limited positive examples of citizen oversight committees earlier in the RGA program cycle, they generally improved over time and became far more active, an increasing number of community elite in treatment municipalities attributed this to RGA programming. However, experiences with oversight committees were accompanied by substantial security fears. Interviews show that knowledge and positive perception of oversight committees improved for community leaders but not for focus group participants. ● Perceived levels of corruption remain high but appear to be decreasing on average, both in RGA and comparison municipalities. Interviews and focus groups reveal that community leaders see corruption as a structural problem and therefore not one that local leaders can easily change. Community leaders improved their relationship with local authorities. They went from rarely meeting with local authorities to having regular meetings (from twice to four times per year). Security and Context ● The implementation of the peace agreement with the FARC has been uneven across the country. Individuals show little change, and perhaps declining support for the peace process. Coupled with growing insecurity, this suggests that RGA has faced a difficult implementation environment. 8 ● Although security is not anticipated to be directly affected by RGA activities, the security situation provides important context. Citizens reported that specific armed groups can still move freely in their municipalities and that security remains a concern. Community leaders and focus group participants shared this perception. ● The peace process with the FARC has alleviated fears in some municipalities and, as a result, created greater openings for civic organizing and political participation. Programming and Evaluation Recommendations We discuss seven recommendations for policy and programming here, and include a recommendations actions matrix in Annex Table A1.1, which also includes key audience for each recommendation. ● RGA activities are associated with increased revenue streams, especially in terms of total revenues, of which royalties are a component. These program results should be emphasized early in future programs to help achieve later downstream outcomes, such as boosting trust and combating corruption. Concretely, future programming should take sequencing of activities seriously, and prioritize activities that promise to make a short-run difference and catalyze other outcomes. ● Because service provision typically lags behind increased revenue flows, services should be relatively slower to materialize. Communications strategies should focus on encouraging realistic expectations among the public about the timing of improvements in service provision. Moreover, USAID should consider conducting a follow-up evaluation in a couple of years, even if on a smaller scale, to assess whether revenues and royalty expenditures ultimately result in greater local service provision. ● There is an association between revenue increases and corruption decreases, with resources generally going up and corruption generally going down. The evidence is correlational in both the survey and interviews and cannot provide specific insights about the potential causal processes at play. It is possible that resources are allocated to the best-managed localities, which are also able to maximize the proper use of them. If so, USAID should consider devoting resources to the best￾managed localities so as to maximize proper use of the resources. Alternatively, when resources are provided to some municipalities it could cause them to improve management and reduce corruption. Given the general relationship, USAID should conduct additional analysis to identify which causal processes may be at play. ● RGA programming depends on local social leaders, who face continuing risks of violence in the post-peace agreement period. RGA-style programs should take into account the burdens and exposure they may create for these leaders, and their security concerns should be addressed so they can more safely and fully participate in programs. The longer-term success of Component 4 activities likely hinges on creating a safe context for citizen engagement. If security threats arise in the context of future programming, implementers could more closely engage with the Colombian National Protection Unit (UNP) to request protection for at-risk individuals. ● To boost trust and legitimacy, local elites should be engaged to play greater roles in raising awareness and sharing information about RGA-style programs with the general public. There are 9 divergences in perceptions of RGA programming between local elites and the general citizenry. As influential and trusted individuals in their communities, local elites can help explain how the benefits of programming may contribute to improved livelihoods. Additional publicity activities could engage local elites to focus messaging about programming activities and successes in the more isolated rural areas of municipalities. ● Regional analysis of RGA indicated variations in programming and outcomes across RGA focus areas. Future governance and development programming should consider the region-specific gaps in outcomes as well as the unique contextual challenges of implementing programming. These efforts should continue to prioritize citizen oversight and participation in regions with higher levels of royalties to ensure accountable public investments. We identified some mismatches between the locations of royalty-based projects and where oversight activities were most prevalent. This geographical targeting of oversight will help ensure that there is sufficient citizen oversight in areas with the most resources and public works projects. ● Future programming to promote citizen oversight should be tailored to the specific exigencies of pandemic response. There have been reports of corruption in the procurement of COVID-19 related supplies and services. These contracting challenges could provide a politically popular opening to more sustainably promote and entrench citizen oversight capacities across Colombian municipalities. Specifically, USAID could leverage RGA’s efforts to support citizen oversight committees by providing these committees with additional training to monitor pandemic-related expenditures. Additional committees or oversight processes could also be promoted more widely in other Colombian municipalities. Such initiatives could be framed and presented to the Colombian public as being “essential for protecting public health” (e.g., “Hagamos Control para la Salud”). Evaluation Purpose This endline evaluation6 report begins with a brief description of the Colombia Regional Governance Activity (RGA), the evaluation approach, and the findings, and then it reports the results of the endline follow-up data collection that occurred in the fall of 2019 and early 2020, 7 with the complete set of data fully delivered in March 2020. The Colombia Regional Governance Activity (RGA) funded by the USAID Colombia Mission, aimed to improve sub-national governance in 40 conflict-affected municipalities of Colombia over the period 2015- 2021. The RGA program was implemented by Management Systems International (MSI), a TetraTech Company, and consisted of five primary components: 1. Improved decentralization to enhance operational capacity of departmental and local governments; 2. Improved financial management and performance of targeted municipalities; 6 The baseline and midline evaluation activities are all reported in the baseline and midline reports and annexes available through USAID or from Findley at the contact information above. 7 The majority of the endline evaluation data collection occurred in Fall 2019. Due to some security concerns, the team conducted some additional data collection in early 2020 as well. As such, most dates in the report refer to endline data collected in late 2019, but others note the full duration including early 2020. 10 3. Improved normative, institutional, and procedural frameworks for development and maintenance of secondary and tertiary road infrastructure; 4. Increased citizen participation for enhanced transparency and accountability; 5. The improved electoral process to limit state capture at the sub-national level. The evaluation team was tasked with evaluating only components 2 and 4. Based on the theory of change advanced by USAID for components 2 and 4, the RGA program was designed to increase municipal capacity and legitimacy,8 which were lacking in some of Colombia’s conflict-affected regions, at least at the outset of the RGA. From RGA’s task order, the purpose of the RGA was to improve sub-national governance in 40 conflict-affected municipalities of Colombia. Municipal capacity to effectively deliver services to citizens was intended to play a key role in building the legitimacy of the state at the local level. Given that the lack of state legitimacy at this level contributes to conditions that foster or perpetuate conflict, increased legitimacy was designed to contribute to minimizing conflict. These interventions and expected outcomes fit squarely within the USAID Colombia mission’s goal of making Colombia more capable of successfully implementing a sustainable and inclusive peace through the intermediate goals of more effectively delivering services prioritized by citizens (IR 1.1) and increasing citizen participation in democratic processes and governance (IR 1.2). All intermediate and end results advance the objectives outlined in the Country Development Cooperation Strategy 2014 -2020 (CDCS). It is worthwhile mentioning that the RGA’s theory of change involves several steps that may take a reasonable amount of time to achieve. Therefore, it may be the case some changes in outcomes attributable to RGA programming may only obtain or be observable after the end of the program and endline data collection. Although RGA activities were implemented at the national and subnational levels, we only evaluated the impact of the subnational components, as per USAID direction. The impact evaluation specifically addresses questions relating to the extent to which the RGA has an impact on two of the five total component objectives: (2) Improved financial management and performance of targeted municipalities and (4) Increased citizen participation for enhanced transparency and accountability. Because we articulated the theory of change at length in the baseline report, we only provide an abbreviated discussion in this report. Please refer to the original statement of work and baseline report for more background. Activity Description and Background: Implementation of the RGA At every stage of the evaluation – pre-baseline, baseline, midline, and endline – the evaluation team interfaced with MSI staff to discuss MSI’s program activities and data resources and ensure a mutual understanding of RGA. MSI staff provided us with their comprehensive database on RGA training activities, which contain entries for attendance and participation in workshops and trainings by different individuals for different aspects of the program for components 2 and 4. In this section, we present basic descriptive data about RGA activities and then note that the analysis of impact occurs in the sections that follow. Note that all of the quantitative regression results that we report are weighted by RGA activity, thereby accounting for variation in implementation. 8 By capacity, we mean the ability of the municipality to implement preferred policies (Young 2013). By legitimacy, we mean the compatibility of the results of governmental (municipal) output with the value patterns of those individuals or group affected (Stillman 1974). Legitimacy is different than public support. Legitimacy relates to whether citizens view their government as having the right to govern. Support is more about a citizen’s assessment of the current government and how well a municipality is governing. 11 Up to the time of this analysis (mid-2020), the database contains 88,670 total individuals trained across all years, with 11,945 individuals trained outside of treatment municipalities (Year 1: 4,764; Year 2: 31,066; Year 3: 41,469; Year 4: 27,438).9 According to the data, there was a large increase in training the later stages of the program in 2018 (post-midline) but training activities began winding down by middle-to-late 2019 (see Figure 1).10 Figure 1: MSI Self-Reported Activities 2015-2019 (Aggregated) As displayed in Figure 2, most content was delivered via either technical assistance, an ongoing form of training for municipal staff, or workshops (talleres), which are mostly directed at citizen participation (especially, the “Hagamos control ciudadano”/ “Let’s do citizen control” program) as well as some public works projects. Policy dialogues were largely held for Component 1 of the program. Figure 3 below shows the distribution of attendance at RGA activities per capita by municipality. In addition to the individual trainings, MSI reported mobilizing hundreds of millions of dollars over the life of the program through the formulation and management of public works projects. A sizable increase in royalties funding – $24.7 million – for municipal projects mobilized by RGA was linked to the OCAD Paz (Órgano Colegiado de Administración y Decisión; Decision-Making and Administrative Collegial Body for Peace).11 The RGA program also enabled numerous community-driven projects or grants. MSI held a call for proposals (convocatoria) to select local organizations such as associations of women, youth groups, and local NGOs for projects in areas such as trainings in participation, social oversight, and development plans, as well as artistic and cultural activities to educate the public about the aforementioned subjects. The grantees are local and from the “territories” and tend to reach more people. 9 The program trainings started in June 2015. Some individuals received training in multiple years. 10 The database undercounts the participation of vulnerable populations since these characteristics are self￾reported and attendees do not always list their ethnicity, victim status, or sexual orientation when registering their attendance at events. Additionally, youths have participated in different activities, including in oversight committees (veedurías). But, as minors, they cannot sign their names or officially vote. 11 Management Systems International. 2019. “Support to Strengthen Municipal Financial Management Regional Governance Activity.” March 22, 2019. 12 Figure 2: Form of Training by Region, 2015-2019 Figure 3: Per capita Attendances at RGA Activities 2015-2019, by Municipality MSI staff also noted success with citizen oversight committees (veedurías). According to figures from MSI, 234 committees for oversight activities, were conducted by citizens, organized and supported as a result of RGA assistance during the period 2016-2019 involving 1,261 participants. Of the veeduría participants, 57 percent were women, and the committees also involved members of the LGBTQ populations. Roughly 13 25 percent of the veedurías were in each of the areas of education, health, and road projects, respectively (Figure 4). In San Calixto (Norte Santander), the Hagamos Control Ciudadano training moved three citizens to form an oversight group in 2019 to oversee the completion of road works planned by the municipal administration. In 2019 in Rioblanco (Tolima), the “Healthy Women” oversight group monitored the implementation of the “Cervical Cancer Promotion and Prevention program,” raising awareness about the importance of screenings and persuading the hospital to increase its screening offerings. Figure 4: New Citizen Oversight Committees (Veedurías) in Treatment Municipalities 2015-2019, by Sector Because some municipalities have greater needs than others, MSI did diagnostics for each municipality to adapt development plans and RGA programming to local conditions. These needs contributed to a heterogeneity of RGA treatments across municipalities and departments. From a programming perspective, such heterogeneity is good practice; it does pose some evaluation challenges, however, especially if the different specific practices are not conceptually similar.12 Variations in programming also resulted from variation in the receptivity of different municipalities to the program. According to MSI staff, implementation went well in the municipality of Puerto Libertador (Córdoba), which improved its administrative efficiency through the RGA program. The same may have occurred with Tumaco (Nariño), despite a series of mayors being removed on corruption charges during the period of the program.13 These successes meant receiving bonuses from the central government for good fiscal management. However, Tumaco had to increase its spending on police forces due to public security issues. 12 A per capita analysis (of intensity of trainings per person) is more appropriate for Component 4 (public participation) than Component 2 (Royalties and administration) because Component 2 usually involves continued training with the same municipal staff members. 13 https://tubarco.news/sin-categoria/se-repite-pesadilla-cae-otro-alcalde-de-tumaco-salpicado-por-corrupcion/ 14 According to the implementer, other municipalities such as Nechí and Tarazá were reportedly less receptive to RGA recommendations and had large deficits for the last fiscal years. Over the life of the program, security affected some program activities, but in most cases, work was able to continue unimpeded. Examples of security challenges include bombings of police stations and killings of police in some coastal areas due to the criminal bands' (BACRIM) “plan pistola” operations. MSI had to stop work at some moments, and in some cases had to base their staff in neighboring municipalities due to insecurity. USAID approved a branding waiver for MSI for Norte Santander and Arauca not to have to signal that the program is associated with the U.S. due to security concerns (in these departments, historical armed conflict also has depressed participation of women). In Teorama, Norte Santander, for example, MSI staff were stopped by armed actors and told not to come back. For the endline, we visited the municipality of Chaparral in Tolima to observe the data collection procedures used by Cifras y Conceptos, including participating in a focus group with residents of the town center (cabecera). We also had several meetings with MSI staff in Bogotá and via videoconference. For the midline, the evaluation team met with MSI’s implementation team (September 29, 2017) in Montería, Córdoba, and visited the municipalities of Tierralta and Planeta Rica. For the baseline, the evaluation team visited Florida and Miranda in Valle de Cauca and Cauca Departments and carried out similar activities. Overall, the staff’s perceptions and insights about their effectiveness were vital for understanding the nature and strength of their program treatments and program effects. The data on program implementation, especially the variation in programming across RGA geographies suggests there was heterogeneous application of the treatment. This is understandable given the changing situation on the ground, and adaptation during the programming cycle. If the theory of change is correct, this may predict differential impacts being observed from location to location. To account for this, we weigh the statistical results based on the strength of the treatment across municipalities. Evaluation Questions and Methodology The full evaluation design is described more fully in the original statement of work and the baseline report, with additional information in the midline report. We mention the basic evaluation framework here and point the reader to Annexes 4-7 for more information, especially on the re-sampling strategy for adapting to the challenge of respondents whom we could not survey a second and third time. There were also a few additional modifications to the baseline and midline, including how data was collected in the Department of Arauca, where certain security challenges were encountered during the baseline fieldwork. The evaluation consisted of a quasi-experimental design that used spatial matching techniques to (1) determine site selection for interviews, surveys, focus groups, and administrative data collection, (2) structure the data analysis of municipalities receiving USAID assistance relative to control municipalities, and (3) provide new and underutilized types of data for analysis of impact. The matching design enabled a comparison of a balanced set of control and treatment municipalities. With a baseline, midline, and now endline evaluation, all of which rest on a carefully selected comparison of municipalities, we draw conclusions about changes within given municipalities across contexts, changes in municipalities over time, and joint comparisons across contexts and over time. These different types of changes are tracked at the level of intermediate outcomes and overall mission objectives. For the treatment municipalities selected by USAID and GOC, we used spatial matching techniques to identify the most appropriate comparison municipalities. That is, we used matching techniques to create a sample of control and treated units that are – in expectation – equally balanced on all observable 15 covariates that could affect the success of the RGA. By balancing, we accounted for possible confounders and non-random selection into treatment, which should enable more accurate identification of any unique causal impacts of the RGA. In addition to these statistical matching techniques, we consulted Colombian experts to consider logical comparisons, examined maps, and identified relative features that make municipalities appropriate comparison units. As with all impact evaluation approaches, there are limitations to this design, which we have worked to address. All types of data provide unique challenges – perception-based surveys, administrative data, interview responses, focus group discussions, implementing partner reported data, and background mission data – which motivated the multi-method approach we employ. Synthesizing across different types of data is not always straightforward, but the conclusions that follow reflect the summary of a comprehensive set of statistical analyses coupled with reporting both of common trends and also notable anomalies derived from the qualitative coding. To avoid reliance on the bounds of any one statistical estimation approach, we also carried out the analysis using different estimation techniques. See Annex 4 for further information. Evaluation Findings and Conclusions The overall results of the evaluation are summarized in Table 1, which covers both security context and the key RGA components of interest. This table aggregates multiple sub-categories in order to give an overall perspective. For a further disaggregated reporting of impact across a greater number of outcome possibilities, see the Findings, Conclusions, and Recommendations (FCR) table in Annex 3 (Table A3.1). The rows capture three broad outcome categories, labeled Context, Component 2, and Component 4. Within those broad categories, there are subcategories (e.g., security/violence or trust/legitimacy) and then specific measures derived from the coding and analysis. The caption below the table describes the precise indicator used in this summary. The columns indicate changes in outcomes in treatment municipalities relative to control municipalities for both the contextual and impact outcome categories listed in the rows. The green/turquoise (+) column indicates that RGA increased the row outcome measure in question. The yellow (=) column indicates that RGA had no discernible effect on the row outcome measure in question. And the red (-) column indicates that RGA decreased the row outcome measure in question. 16 Table 1: Summary of Impact in Key Categories of Interest OUTCOME OR CONTEXT CHANGE Treatment 2015-2020 IMPACT OUTCOMES Treatment / Control + = - + = - CONTEXT SECURITY AND VIOLENCE CITIZEN SECURITY ILLEGAL ARMED GROUPS TRUST AND LEGITIMACY TRUST LEGITIMACY COMP. 2 FINANCIAL MANAGEMENT MUNICIPAL PERFORMANCE REVENUES EXPENDITURES SURPLUS / DEFICIT SERVICE PROVISION COMP. 4 CIVIC OVERSIGHT POLITICAL PARTICIPATION ACCOUNTABILITY OVERSIGHT NOTE: For “Impact Outcomes,” results are based on regressions with proportional treatment weights and controls For “Citizen Security,” results are based on the homicide rate For “Illegal armed groups,” there was a statistically significant increase in 3/6 outcome cases, so it is recorded as increasing For “Level of trust,” there was a statistically significant increase in only 1/19 outcome cases, so it is recorded as no change For “Legitimacy,” results are based on Corruption” For “Service Provision,” results are based on “Access to public services” For “Accountability,” results are based on whether “Municipal Authorities Invite Opinions and Feedback.” For “Oversight,” results are based on “Fiscal performance” For “Participation,” results are based on “Have participated in political party events.” 17 Context: Leading up to the Endline, and Trends Between the midline and endline data collection activities, several events occurred in Colombia that directly shaped the conditions in which the RGA program was operating as well as its desired outcomes. Iván Duque of the Alternative Democratic Center (Centro Democrático Alternativo - CDA) won the 2018 presidential election and took office on August 7, 2018, succeeding Juan Manuel Santos. Much of this period encompassed the initial implementation of the peace agreement between the Colombian government and the FARC guerrillas.14 Duque’s CDA party vowed to “tear up” the peace agreement, and although critics have pointed to his administration’s lackluster implementation of the agreement’s provisions, the government has proceeded with various peace activities and programs. Peace implementation greatly shaped the context within which the RGA program was implemented. In the three years since the peace agreement, guerrilla ex-combatants have laid down most of their weapons and gathered in 26 rural transition zones (now referred to as ETCRs) where they began the process of demobilization and reincorporation into civilian life (some of which are in or nearby RGA municipalities).15 Ex-combatant reintegration has been led by the Agency for Reintegration and Normalization (ARN) and supervised by the National Reintegration Council. Of the 14,178 names of FARC-EP members presented by the organization, 13,51016 have been accredited to participate in the peace process, allowing them to receive legal, economic and productive benefits.17 Some political parties aligned with the ruling party and the CDA claim the FARC have not complied with the terms of the agreements, an indictment that has gained strength after one of the chief negotiators for FARC announced they would take up arms again in September 2019. Although insecurity persists, this move does not appear to have greatly exacerbated violence so far. For its part, the FARC political party argues the government has not delivered on its part of the agreement, especially concerning security guarantees for ex combatants. Security-wise, since the agreement was signed, national security statistics have improved compared with the pre-agreement period. Nevertheless, insecurity has steadily increased in rural areas in the post￾agreement period. Many FARC members abandoned the peace process and organized dissident fronts, many in coca-growing regions. The BACRIM known as the Gaitanista Self-Defense Forces (AGC) has expanded its presence and is competing with ELN fronts in some regions to control former FARC territories. Right-wing armed groups have begun to target demobilized FARC members, and NGOs have also reported a surge in killings of social leaders advocating for land and human rights.18 The Attorney General created a Special Investigations Unit to investigate and prosecute criminal organizations and their support networks. Regarding the rural reform proposed in the agreement, government has pushed ahead with the Territorially Focused Development Programs (PDET). Data show that around 1,095 projects were either finished or in process in 76 municipalities. To increase financing for PDETs, royalty funds have been aligned to support projects in the local development plans of 140 municipalities. 14 https://www.nytimes.com/2016/06/23/world/americas/colombia-farc-peace-deal-rebels-cease-fire-santos.html 15 http://politicalviolenceataglance.org/2017/05/24/hacking-for-successful-peace-implementation-in-colombia-part-1/ http://politicalviolenceataglance.org/2017/07/25/hacking-the-reintegration-of-farc-rebels-in-colombia/. The purpose of this endline is not to report on the progress of peace implementation, but for further information, The UN Verification Mission in Colombia report documents the progress in implementation.: “Report of the Secretary￾General on the United Nations Verification Mission in Colombia”, 26 December 2019. 16 http://www.reincorporacion.gov.co/es/reincorporacion/Paginas/La-Reincorporaci%C3%B3n-en-cifras.aspx 17 https://www.nytimes.com/2017/08/03/opinion/farc-colombia.html 18 http://politicalviolenceataglance.org/2017/05/02/protect-the-people-to-protect-the-peace/ 18 Political reintegration has been successful insofar as FARC was able to register before the National Electoral Council as a legal political party, known as the People’s Alternative Revolutionary Force (FARC). The FARC had 308 candidates running for office in the October 2019 local elections.19 The day of the election saw fewer violent incidents than the past local elections and turnout was above 61%, in part due to the installation of more polling centers in rural zones where security conditions improved after 2016.20 Finally, the period saw several robust social movements. In April and May of 2019 there was a widespread Indigenous Minga protest advocating for peace, justice, development concentrated in the southern part of the country. There was also a two-month nation-wide strike began on November 21, 2019, uniting indigenous and Afro-descendant movements, unions, and students in opposition to fiscal reform, demanding compliance with the Peace Accords, and calling for the protection of social leaders and FARC ex-combatants. On top of all of this, the political and economic crisis continued to envelop neighboring Venezuela, moving hundreds of thousands of Venezuelans to flee their country, with many arriving in the Colombian border areas of Arauca and Norte de Santander (which are also RGA activity areas). The data collection at endline offers useful information about the security, governance, and social conditions on the ground in both treatment and control municipalities. Comparing the data from baseline to endline and between treatment and control areas confirms that the RGA program is targeted at municipalities with ongoing challenges in the areas of security, illegal economies, the peace process, and trust in government. A Deteriorating Security Situation Made Implementation Difficult Security and violence are contextual factors that affect the ability of RGA to function and could also be considered long-term outcomes of the program. Indeed, as the original tasking indicated “the purpose of the RGA is to improve sub-national governance in 40 conflict-affected municipalities of Colombia. Municipal capacity to effectively deliver services to citizens plays a key role in building the legitimacy of the state at the local level. Given that the lack of state legitimacy at this level contributes to conditions that foster or perpetuate conflict, increased legitimacy will contribute to minimizing conflict [emphasis added].” RGA was not specifically tasked with shorter-term conflict reduction, though they conducted many activities in support of this, and report extensively in annual reports about their efforts in this regard. For example, RGA reports supporting peace implementation throughout year 3, including working with peace managers in 6 municipalities and helping develop policy about peace and reconciliation. Regardless of whether RGA was directly or indirectly engaged, if the RGA program plays out according to the theory of change, then violence will hopefully be reduced over the long-term, but perhaps over a much longer time horizon than the roughly five-year RGA program. Therefore, for purposes of the evaluation, we primarily consider security and violence as short-term contextual factors that affect participation in the program and perceptions of program success. As discussed below, across all of the information gathered from the endline survey, a central conclusion is that people perceive mixed changes, with some increase in security as relates to armed political groups, while also perceiving a decrease in security as relates to private criminal groups. As shown in Figure 5, residents view the BACRIMs as more able to move freely at the endline than at the baseline (5 percentage point increase in control, 3 percentage point increase in treatment). Survey 19 https://pares.com.co/2019/10/28/partido-farc-en-elecciones-locales-2019-un-bautizo-politico/ 20 https://colombia.unmissions.org/sites/default/files/n1942150.pdf 19 respondents report, however, a slight decline in the FARC’s ability to move freely (no change in control, 3% decline in treatment) as well as paramilitaries (4 percentage point decline in control, 3 percentage points in treatment). Results are mixed for the ELN with a decline in control (3%) and slight increase in treatment municipalities (2 percentage points; Annex 2d shows that most of these differences are meaningful and not simply occurring by chance).21 Figure 5: Groups Are Free to Move About the Municipality As Figure 6 illustrates, endline survey respondents’ feelings of safety fell precipitously. Citizens were more afraid to travel to the municipal center in almost all municipalities than they were in 2015 when the baseline was conducted. Nearly all municipalities show increases in fear of traveling (blue to red dot, baseline to endline). Only the municipalities that had high levels of fear at the baseline saw significant improvements (for example, Planeta Rica or Florida). 21 In the baseline, this question refers to the FARC armed group. We slightly changed the phrasing of the question for the midline and endline to account for the FARC’s demobilization and disarmament in mid-2017 as part of the peace process while also recognizing that survey respondents could still perceive activity by some dissident fronts. 68.0% 32.0% 64.0% 36.0% 63.0% 37.0% 61.0% 39.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Can group move freely? Percentage Bacrim 76.0% 24.0% 71.0% 29.0% 79.0% 21.0% 69.0% 31.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Can group move freely? Percentage ELN 72.0% 28.0% 59.0% 41.0% 75.0% 25.0% 59.0% 41.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Can group move freely? Percentage FARC 69.0% 31.0% 63.0% 37.0% 73.0% 27.0% 66.0% 34.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Can group move freely? Percentage Paramilitaries 20 Figure 6: Municipality Level Changes in Feelings of Safety Figure 7 disaggregates survey respondents’ reports of being a victim of crime in the previous year by municipality as well as by control and treatment status. Municipalities with the highest initial levels of victimization (blue dot is baseline levels), such as La Montañita, saw large improvements by endline (red dot). This is true for both control and treatment but the effect appears to be largest in a number of the treatment municipalities. However, many other municipalities, like the control municipality of Puerto Tejada, saw worsening levels of victimization. Ataco La Macarena Planadas Ituango Puerto Rico San Juan de Arama San Vicente del Caguán Buenavista Chitagá Puerto Lleras Angostura Solano La Esperanza Rioblanco La montañita Amalfi Peque Mapiripán Cáceres Tarazá Dabeiba Mesetas Vista Hermosa Zaragoza San José Del Fragua San Antonio Tierralta Anorí Valdivia Milán Caloto Puerto Tejada Tibú Ortega Valencia Yarumal Miranda Sardinata Remedios Villa Rica El Bagre Corinto Nechi Ábrego Caucasia San Martín Segovia San Andrés de Tumaco Convención Puerto Libertador Briceño San Calixto Santander De Quilichao Montelíbano Chaparral Pradera El Carmen Florida Planeta Rica Teorama 0.0 0.2 0.4 0.6 Afraid to travel to Municipal center? Municipality Change in Feelings of Safety, 2015−2019 21 Figure 7: Respondent Reported Victimization by Municipality and Treatment/Control Status Survey respondents, community leaders, and municipal administrators all recognize the importance of improving security to achieve sustainable and lasting peace. They continue to be concerned about the risks posed by illegal armed groups. The results from our interviews with community leaders at the midline in 2017 indicated some signs of improvement in security conditions (52.1% to 59.5% in control; and 56.7% to 66.2% in treatment). However, by the endline, there was a significant worsening of security conditions when compared to the baseline situation five years prior (the percent reporting worsened conditions rose 13.9% to 18.6% in control; and 12.9% to 23.3% in treatment). Although some elites still perceived improvements in security at endline, the percentage is lower than at the baseline (52.1% to 43.4% in control; and 56.7.% to 42.5% in treatment). The community leaders we interviewed indicated that security has significantly worsened over the past two years across all regions (See Table 2). As we reported in the midline report, it is important to highlight that community leaders have begun to grow worried about their personal safety. Several leaders we talked to said that they or someone in their organizations have been either threatened or forcibly displaced at some point in the past years. Similarly, in both treatment and control municipalities, armed groups were said to be affecting the work of the local administration, especially in rural areas. Focus group participants and community leaders point to weak police and armed force presence in municipalities as one the major security problems. Community and political leaders alike similarly report that these challenges are a widespread, national problem. Control Treatment 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 Buenavista Ataco Vista Hermosa Ituango La Esperanza La Macarena Peque Zaragoza Valencia Valdivia Mesetas Chitagá Yarumal El Bagre Sardinata Cáceres Nechi Planeta Rica Angostura San Vicente del Caguán Planadas Convención Montelíbano Solano Puerto Tejada Amalfi Tierralta Teorama Puerto Rico San Juan de Arama Corinto Caloto San Andrés de Tumaco Dabeiba Rioblanco Anorí San Antonio Tibú Segovia Miranda Puerto Libertador Tarazá Villa Rica Ábrego San José Del Fragua San Calixto Chaparral El Carmen Remedios Pradera Milán San Martín Santander De Quilichao Caucasia Ortega Puerto Lleras Briceño Florida Mapiripán La montañita Victim of a Crime? Municipality Victimization by Municipality, 2015−2019 22 Table 2: Change in security conditions in past two years as reported (community elites) Control Treatment 2015 2017 2019 2015 2017 2019 Improved 52.1% 59.5% 43.4% 56.7% 66.2% 42.5% Same 29.2% 16.5% 31.0% 26.3% 30.97% 17.8% Worsened 13.9% 9.5% 18.6% 12.9% 8.2% 23.3% NA 4.9% 14.6% 7.1% 4.1% 11.3% 16.4% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Steady Skepticism about the Peace Process Peoples’ views of the peace process experienced almost no change across the life of the project. Views were overwhelmingly negative at baseline and continue as such at endline (See Figure 8). Nearly 90% of survey respondents in control and treatment, at both baseline and endline, reported that the peace process did not take into account their views, which is consistent with these perceptions about inclusion, Figure 9 shows that only citizens in Antioquia and Putumayo experienced sizable increases in average support for the peace agreement. 22 The increases in the sampled municipios of these departments may be surprising given that many accounts of the peace process suggest that support has attenuated over time (the optimistic results could be a result of the municipios sampled). Indeed, all other departments experienced either moderate, or in the case of Meta, steep, declines in average support for peace. These are also stark results given the various efforts of the Colombian government and international cooperation actors to engage communities in the implementation of peace programming in PDET areas. 22 Note that due to security concerns in Putumayo, the enumerators only carried out surveying in urban areas. The positive results could, therefore, be due to selection problems. To ensure results were not being driven by this, we re-estimated all regression results excluding Putumayo and its closest comparison from the matching (Tumaco), and the results are all qualitatively similar. We further re-estimated all regression models limiting entirely to urban rather than rural respondents and the results are again qualitatively similar. We are thus reassured that sample selection challenges should not be responsible for any treatment-control impact comparisons. Individual municipality inferences, such as those in Figure 9 for Putumayo, should be interpreted accordingly. 23 Figure 8: Peace Process Takes Into Account My Views Figure 9: Changes in Average Support for Peace by Department Citizens in focus groups and community elites see the decline in security conditions as a result of a failed peace process between the FARC and the Colombian Government. As a participant of a Focus group in Florida, Valle del Cauca, told us, “The process had a negative outcome because we went back to war. Look at Corinto (Cauca), they are experiencing armed confrontation again, they are going back to how things were [before the Peace Accords].” Other issues such as the return to the armed fight of chief FARC negotiator Jesús Santrich, the killings of an alarming number of FARC ex-combatants, and breaches of the agreement by both sides are also often cited as evidence of the failure of the peace process. As this same focus group participant continued, “FARC rearmed. They are [now] more aggressive and we don’t know how they were treated in the camps.” Illegal Economies: A Source of Livelihoods Survey respondents’ views on illegal economies and coca production have changed over the course of the project. On net, according to data from UNODC, coca production sky-rocketed in Colombia from baseline to endline.23 As Figure 10 shows, while fewer people consider Coca production to be positive (13 percentage point decline in control, 5 percentage point decline in treatment), most people think it guarantees survival (Figure 11) and there is little change in this belief (63% agree in control and 66% in treatment at endline). Thus, while stemming activity in illegal economies is an important contextual support factor, it appears that basic perceptions about appropriateness have moved in a positive direction, but are still secondary to perceptions about its necessity for survival. 23 https://www.unodc.org/unodc/en/frontpage/2018/September/coca-crops-in-colombia-at-all-time-high--unodc￾report-finds.html https://www.usnews.com/news/world/articles/2020-06-17/colombias-potential-cocaine-output-up-15-in-2019-says￾un-drug-agency 89.0% 11.0% 85.0% 15.0% 88.0% 12.0% 87.0% 13.0% Baseline Endline Control Treatment Disagree Agree Disagree Agree 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 Peace Process Takes Into Account My Views? Percentage Arauca 1.73 Córdoba 2.06 Caquetá 2.2 Antioquia 2.46 Cauca 1.84 Meta 1.53 Nariño 1.57 Norte de Santander 1.79 Putumayo 2.7 Arauca 2 Córdoba 2.18 Caquetá 2.2 Antioquia 1.89 Cauca 2.02 Meta 2.19 Nariño 1.69 Norte de Santander 1.9 Putumayo 1.51 Baseline Endline 1.0 1.5 2.0 2.5 3.0 Avg. Support for Peace Green = Increase, Red = Decrease 24 Figure 10: Views that Coca Production is Positive Figure 11: Views that Coca Production Important for Survival Component 2: Initial, Varied Improvement in Financial Management and Public Attitudes We now turn to the impact of RGA for components 2 and 4. RGA appears to have had small, positive effects on select fiscal performance indicators. Other results may be realized only later over the longer￾term, if at all. The USAID mission needs to continue tracking the program’s outputs/outcomes in RGA municipalities and, more complicated, in comparison municipalities, to assess any larger effects.24 In Colombia, municipal financial management and performance are complicated, with revenues originating from several sources and subject to many constraints. We focus on the elements of the public finance system that have bearing on our evaluation of RGA components 2 and 4. We report some elements here in the main report, including tax revenues and royalties, and report others such as fiscal performance in the Annexes (A2.5-A2.6). Municipalities derive their funding from four main sources: national government transfers, royalties funds (SGR), municipal taxes, and other funds. First, they receive some funds from national government transfers, which are intended as general support for municipalities, such as from the Sistema General de Participaciones (SGP). As a second source of funding, municipalities receive some resources from the national royalties account, the Sistema General de Regalías (SGR). This funding first includes direct royalties which are disbursed to each municipality so long as the municipality meets basic management criteria to ably administer the funds. The receipt of basic royalty revenues to a degree reflects municipalities’ performance as they qualify for the funds (municipalities get bonuses from the National government for good usage of funds over the three previous years). The municipalities are also eligible for indirect revenues from the OCAD funding mechanism, and it is here that RGA would be expected to have the greatest fundraising impacts (MSI reported a $24.7 million increase in such revenues). But project-related expenditures of royalties 24 The mandate of this evaluation concludes with this endline report, but to assess impact fully the municipalities in the RGA program would need to be tracked over the longer term, and more complicated, the comparison municipalities would need to be tracked as well. On standard program and evaluation timelines, this takeaway/recommendation may not be sufficiently conclusive, but the evaluation team sees substantial merit in the RGA program, and early indications of positive impact, although few that are conclusive at this point. 40.0% 60.0% 44.0% 56.0% 49.0% 51.0% 47.0% 53.0% Baseline Endline Control Treatment Disagree Agree Disagree Agree 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Coca Production is Positive Percentage 29.0% 71.0% 30.0% 70.0% 32.0% 68.0% 30.0% 70.0% Baseline Endline Control Treatment Disagree Agree Disagree Agree 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Coca Production Guarantees Survival Percentage 25 themselves may be even more important, because they comprise a key element of the theory of change about financial management contributing to improving conditions in the municipality. (See Annexes for a discussion of expenditures.) Third, municipalities receive money from local taxes, including property taxes (tributary revenues). National taxes include income taxes, value-added taxes (VAT), or corporate taxes, and some of those taxes may return back to the municipalities through transfers. Finally, some additional revenues are available through other agreements and mechanisms, such as international cooperation, direct contributions to projects from private companies (obras de impuesto),25 and special programs, such as the OCAD-Paz peace royalties financing program as mentioned above, for which municipalities propose specific public works projects to access funds. Many municipalities look to achieve better financial management, better financial positions, and better municipal performance. Based on the theory of change, improved financial management should lead to a better financial situation, which should translate to better performance. We consider many possible indicators, as outlined in the original Statement of Work (see Annex 7) and reported in the baseline and midline. We report on several administrative and survey data indicators that suggest RGA had a positive effect at least in some areas. In many others, on the other hand, there were only unclear, null, or negative results as reported in the section below on unclear/null results. Total Revenues Increased Over Time Looking at all municipalities, overall revenues increased over time with a big jump between baseline and midline, but little change between midline and endline (See Figures 12 and 13). Of course, RGA only affects some components of total revenues, but the aggregate numbers should nonetheless be responsive to those changes affected by the RGA program. There are small, statistically significant differences between treatment and control municipalities in overall revenues. Figure 13 shows the data by region. Most regions experience very small increases in total revenues, but Nariño is a clear outlier, with sizeable increases in total revenues over time. We review the shifts in different types of revenues below. Figure 12: Log of Total Revenue by Group 25 https://www.semana.com/nacion/articulo/obras-por-impuestos-un-mecanismo-para-combatir-la￾desigualdad/202100/ Figure 13: Log of Total Revenues by Region 26 As depicted in Figures 14 and 15, property revenues also increased steadily across waves, and increased more in treatment than in control. Regionally, there was an increase in Cauca from baseline to endline, perhaps suggesting some program effects. RGA Municipalities Obtained Greater Royalty Revenues Although there were only small overall changes in revenues, there were differences in royalty revenues and expenditures. Figures 16 and 17 show box plots of royalty revenues and then expenditures on those revenues for control and treatment municipalities (left and right panes respectively). Each box shows the average (mean) and distribution of revenue/expenditures at different points during the lifecycle of RGA, as evidence is available on these indicators. Royalty revenues decreased at the midline relative to the baseline but rebounded and increased again by the endline. The RGA cannot affect the base royalty allocation, though that allocation increased substantially in treatment municipalities relative to control. (See Figure 16 and Table A2.9 in Annex 2D.) RGA could have an impact on expenditures through improvement of local government capacity that enables access to the royalties, and the evidence indicates that treatment municipalities had slightly higher expenditures on the royalty revenues than the control municipalities (see Figure 17 as well as Table A2.9 in Annex 2D). Figure 14: Property Revenue by Wave and RGA Activity Figure 15: Property Revenue by Wave and Region 27 Figure 16: Royalties Revenue by Group Figure 17: Expenditures of Royalty Revenue Comparing baseline to endline, there is improvement in the way municipal administrators understand and navigate the SGR. At this managerial level, administrators in treatment municipalities tend to report that their offices have experienced a general “technification” throughout these five years. This transformation is said to have significantly improved the way municipalities navigate the royalties system, positively impacted the way the municipality applies for and implements projects and receives funding, and also improved communication with the community leaders. This latter group, for their part, said that they meet frequently with the local authorities: every three to five months in control municipalities and two to four in treatment municipalities. This represents a stark shift from baseline, where community leaders said that they had rarely met with municipal representatives. As shown in Table 3, in the interview transcripts we conducted with municipal administrators in treatment municipalities in 2015, 3.9% indicated difficulty in accessing SGR funds. In the endline, no interviewees mentioned such challenges. By contrast, in control municipalities, references about difficulties in accessing SGR funds surged from 2.2% to 30.8%. But there is still much work to be done on this front, as municipal administrators in control and treatment municipalities declare that they keep having issues with the royalties system. While they did not declare that it was difficult to access funds in endline they did acknowledge, however, that the total money that they are receiving was not yet what they were expecting. To solve this issue, they say they need more staff as well as technical and normative assistance. Table 3: Municipal Administrators Perceptions of the Royalties System coded references. Perception about SGR. Mayor's office Control Treatment 2015 2017 2019 2015 2017 2019 Corrupt system 0.0% 2.3% 0.0% 2.6% 0.0% 0.0% Difficult to access funds 2.2% 11.4% 30.8% 3.9% 16% 0.0% Important source of revenue 35.6% 38.6% 30.8% 28.9% 25.3% 33.3% Not enough money 31.1% 11.4% 30.8% 14.5% 9.3% 8.3% Unfair system (municipality produces too much for little revenue) 20.0% 20.5% 0.0% 27.6% 9.3% 41.7% NA 11.1% 15.9% 7.7% 22.4% 40.0% 16.7% 28 Elites are More Optimistic about Tax Collection than Citizens Perceptions Tax collection is an important source of revenue for municipalities. A strong culture around taxation—a willingness to pay taxes among citizens—is likely shaped by perceptions of legitimacy of municipal administrations and can help them accumulate the necessary resources to develop and implement public works. We assess shifts in perceptions about taxation, because changes in citizens’ or community elites’ perceptions could be the basis for positive longer-term changes in tax payments and therefore revenue capacity. Of course, perceptions about tax collection are only indirect indicators, but are part of a multimethod approach to assessing impact. The measurement and evaluation of impact in the governance space is extremely difficult and requires consideration of a broad range of evidence. As shown in Figure 18, over 40% of survey respondents considered that paying taxes is losing money. The left pane shows the baseline values and the right pane shows the endline. The upper row shows the control municipalities and the lower row shows the treatment municipalities. The red bars show disagreement with the statement that paying taxes is losing money and the blue/green bars show agreement with the statement. Both control and treatment were similar at the endline at 43% indicating small increases of 2 percentage points and 1 percentage point respectively from prior levels, though not statistically differently. As Figure 19 further shows, across departments, regionally this feeling is increasing. The green lines show regions in which there is an increase from baseline to endline, whereas the red lines show regions with decreases. Figure 18: Percent Agreement that Paying Taxes is Losing Money Figure 19: Change in Agreement that Paying Taxes is Losing Money The largest shift among respondents is the decline in the proportion of people who strongly agree that paying taxes is a civic duty from baseline to endline (13 percentage point decline in control and 12 in treatment; See Figure 20). This shift is largely consistent among the treatment and control groups and therefore suggests a more universal erosion of positive cultures of taxation. During focus groups, two issues were voiced as affecting attitudes towards tax payment. First, the informality of land ownership impedes tax payments since, without legal property rights, citizens did not feel compelled to pay taxes. To address this issue, the Colombian government is expected to implement a new multi-purpose cadastre system with the help of a US $50 million loan from the Inter-American Development Bank (IDB). Second, ambiguity about the destined uses of public funds and the absence of state regulation left citizens uncertain about the benefits of making their tax contributions. 58.0% 42.0% 58.0% 42.0% 57.0% 43.0% 57.0% 43.0% Baseline Endline Control Treatment Disagree Agree Disagree Agree 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Paying Taxes is Losing Money Percentage Arauca 3.03 Córdoba 3.35 Caquetá 3.27 Antioquia 3.37 Cauca 3.61 Meta 3.05 Nariño 3.05 Norte de Santander 3.57 Putumayo 3.41 Arauca 3.47 Córdoba 3.38 Caquetá 3.18 Antioquia 3.15 Cauca 3.92 Meta 3.8 Nariño 3.25 Norte de Santander 3.93 Putumayo 4 Baseline Endline 2.5 3.0 3.5 4.0 Paying Taxes is Losing Money Green = Increase, Red = Decrease 29 Figure 20: Percent Agreement that Paying Taxes is a Civic Duty Community leaders in treatment municipalities, on the other hand, do note a positive change in the way the funding is being invested when comparing baseline to endline. They are, however, critical about the decision-making processes for investments. Many junta members we spoke to declared that they were meeting with their local authorities but that these meetings did not necessarily translate into their concerns being addressed. As a leader from El Bagre (Antioquia) told us, “The projects are decided in the mayor’s office but they do not consult the community about our needs.” Coded references of this form were higher in control than in treatment municipalities. While municipal administrators said that their administrations were able to implement (on average) around 70% to 90% of their municipal plan, community leaders still argue that the municipality is not resolving their most urgent needs, that it lacks consistency, or that it provides poor service. Community leaders are also critical of the way private operators run utility companies with their tax money. Paradoxically, there is an increase in the number of leaders who claim that the municipality is receptive to their questions, especially in treatment municipalities. This may suggest that while there may have been an improvement in the way communities engage with their local authorities, these interactions are not without disagreements. This may, at the same time, illuminate on why we see a decline in attitudes towards tax payment amongst survey respondents. We also asked community leaders and municipal administrators about whether tax collection was effective. Tables 4 and 5 report the results of these inquiries. Attitudes toward tax collection in the mayor’s offices seem to have improved. And among community leaders, fewer are saying that tax collection is not effective. This provides modest evidence in support of a better tax culture. Interviews with both community leaders and municipal administrators suggest that municipalities have been improving in the areas of tax collection throughout the past five years. 10.0% 11.0% 23.0% 17.0% 11.0% 29.0% 11.0% 12.0% 23.0% 17.0% 11.0% 27.0% 10.0% 13.0% 25.0% 25.0% 13.0% 16.0% 7.0% 12.0% 29.0% 23.0% 13.0% 16.0% Baseline Endline Control Treatment Strongly Disagree Disagree Slight Disagree Slight Agree Agree Strongly Agree Strongly Disagree Disagree Slight Disagree Slight Agree Agree Strongly Agree 0.0 0.1 0.2 0.3 0.0 0.1 0.2 0.3 Paying Taxes is a Civic Duty Percentage 30 As it was the case with royalties funding, representatives from the Mayors offices tend to attribute these improvements to recent transformations in terms of technification and planning of their offices and functions. When it comes to community leaders, they suggest that it is due to recent changes in the way tax collection is being implemented. Municipal administrators also give credit to the effectiveness of recent pedagogical strategies aimed at teaching the general population about their rights and responsibilities regarding tax collection. Overall, there has been a technical transformation of local state capacities in Colombia, which has impacted both treatment and control municipalities alike, suggesting that RGA has had little impact on these perceptions. This is acknowledged by many of the survey respondents and is evident in comparisons in opinions regarding tax collection from baseline to endline. Table 4: Is tax collection in the municipality effective? Community Leaders Control Treatment 2015 2017 2019 2015 2017 2019 Yes 49.7% 48.3% 47.8% 33.7% 45.2% 33.9% No 34.5% 35.1% 37.2% 52.0% 39.0% 46.8% NA 15.9% 16.6% 15.0% 14.4% 15.8% 19.4% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Table 5: Is tax collection in the municipality effective? Municipal Administrators Control Treatment 2015 2017 2019 2015 2017 2019 No 40.0% 33.3% 38.5% 44.7% 28.6% 50.0% Yes 55.6% 57.8% 61.5% 38.2% 65.3% 41.7% NA 4.4% 8.9% 0.0% 17.1% 6.1% 8.3% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Performance: Limited Improvement in Service Delivery Based on the theory of change, better public financial management should translate into improved service provision. But public management systems can change slowly and it can take time to increase the flow of resources, so shifts in municipal service provision can take longer than expected. We did not see any meaningful changes in service provision between treatment and control municipalities at endline. However, while both groups of municipalities saw increases in municipal expenditures, treatment municipalities increased theirs to a greater degree. This suggests that RGA may have not only impacted revenues but also spending, which could generate service provision improvements over time. Figure 21 shows survey respondents’ general on access to several different public services. Service provision in areas including electricity, sewage, garbage collection, and cell service remained mostly stable across time, both in treatment and control, with most respondents enjoying access to these services. The largest shifts occurred in access to gas service, especially in treatment areas with reports of access rising 31 from 30% of respondents to 68%. Importantly, services such as cellular cannot be affected by RGA given that they are provided at the national level, and so any improvements observed here cannot be attributed to RGA. Other services such as gas provision are only partly provided at the local level, and so a connection to RGA may be difficult to isolate. For those services that might possibly be affected by RGA, there are null results, reinforcing the lack of effectiveness of RGA programming in this area.26 Figure 22 displays respondents’ perceptions of changes in service provision over time. In sum, most people see modest changes, at best, or no changes at all in service provision over the life of the RGA program. 26 We note that RGA may not have prioritized all particular services, but had broad aspirations of improving service delivery. In consultation with the mission and implementing partner, all services were reported on in earlier phases of the evaluation and we maintain consistency by continuing to report them here. 32 Figure 21: Perceptions of Access to Services Figure 22: Perceptions of Change in Access to Services Tumaco: A Singular Case? Tumaco presents a difficult case, despite having received substantial RGA programming. The mayor in office during our endline measurements had been elected through an atypical election, which led to a relatively short and ineffective administration.27 In interviews social leaders characterized the mayor as being highly ineffective by the endline. Survey responses corroborate this perception, as numerous indicators of mayoral performance plummeted in Tumaco from baseline to endline. The percent of 27 https://360radio.com.co/vinculan-al-nuevo-alcalde-tumaco-las-farc/; https://pagina10.com/web/gobernador￾nombra-alcalde-e-en-el-municipio-de-tumaco/; https://www.elcolombiano.com/colombia/regresa-julio-cesar-rivera￾a-la-alcaldia-de-tumaco-narino-CG8822567; https://www.eltiempo.com/justicia/delitos/capturan-a-alcalde-de￾tumaco-por-presuntos-hechos-de-corrupcion-307586 33 respondents with confidence in the mayor's office dropped from 35% to 4%, those believing the municipio takes the community into account for decision-making dropped from 21% to 1%, and those believing the municipality is effective in collecting taxes dropped from 67% to 51%. Problems with water and drainage became critical in Tumaco. The community leaders who we spoke with did not trust the new mayor, and some started to interact directly with the national government instead. Insecurity has made accountability impossible. As a community leader recounted, “Here in Tumaco, 70% of our community leaders, including myself, have received death threats.” Due to the port’s importance for ocean access in relation to the coca trade, the violence during the period of the program negatively impacted community participation and grassroots accountability processes. While these declining perceptions were not a result of RGA, given the deep governance challenges in the municipality the program was also unable to mitigate these issues. Component 4: Increased Citizen Participation To promote accountable governance, an additional goal of RGA is to increase citizen participation in municipal decision-making. The mayors and municipal staffs can incentivize and support citizen participation through their responsiveness and performance. The evidence does not show, however, that RGA is associated with improved mayoral responsiveness and performance. That said, mayors tend to be viewed more positively in treatment municipalities. From the bottom-up perspective, there have been some positive impacts for Component 2’s core outcome of citizen involvement in participatory budgeting. RGA also helped elevate some underrepresented individuals as possible leaders in communities. So, while mayoral performance did not improve during the period of the program, greater citizen participation in oversight may boost future accountability of municipal administrations. Declining Performance of Mayors Effective and broad-based citizen participation depends in part on engagement by mayors and their willingness to provide information on their activities. This willingness of citizen’s to participate in budgeting and other oversight processes can therefore in turn be influenced by how they perceive the engagement of municipal officials. In many municipalities, survey respondents reported that their mayors were doing a poor job (See Figure 23). Any improvements from baseline to endline (above the red line) are almost all in treatment municipalities. This is positive for RGA, suggesting that mayors in fact performed better at least in a subset of RGA areas. The municipalities on the left side of the blue line are below average. Those municipalities that are left of the blue line and also below the red line are below average and getting worse. La Montañita, Cáceres, and San Vicente Del Caguán—RGA areas—are among the few municipalities that are above average and improved over time (above red line and to the right of the blue line). 34 Figure 23: Mayoral Performance by Municipality, Differences Relative to the Average Community leaders confirm the poor perceptions of mayors. Across most measures, leaders do not report very positively. In particular, at endline a lower percentage of leaders reported that local government performance had increased relative to the percentage at baseline. This effect is stronger in control than in treatment. A higher percentage of leaders reported a worsening at endline when compared to the baseline, but the increase was stronger in control than treatment. And in RGA municipalities, a higher percentage of respondents reported that local government performance remained the same, which is in contrast to the control condition. Taken together, RGA municipalities are less negative than control municipalities, but generally the perceptions of overall performance were not strong (Table 6). Table 6: Has performance of the local government improved in the past two years? Performance of Local Government According to Community Leaders Control Treatment 2015 2017 2019 2015 2017 2019 Improved 50.3% 55.4% 38.1% 52.0% 55.6% 46.6% Worsened 13.8% 18.9% 31.0% 13.4% 22.5% 25.3% Same 11.7% 20.6% 9.7% 7.9% 12.4% 13.0% NA 24.1% 5.1% 21.2% 26.7% 9.6% 15.1% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% However, these negative perceptions were more common among community leaders in control municipalities. In treatment municipalities, by contrast, community leaders recognize improvement along several RGA lines of intervention, such as citizen participation, transparency, and oversight committees. As a community leader in Montelibano, Córdoba told us: Ábrego Amalfi Angostura Anorí Ataco Briceño Buenavista Cáceres Caloto Caucasia ChaparralChitagá Convención Corinto Dabeiba El Bagre Florida Ituango La Macarena La montañita Mapiripán Mesetas Milán Miranda Montelíbano Nechi Ortega Peque Planadas Planeta Rica Pradera Puerto Libertador Puerto Lleras Puerto Rico Remedios Rioblanco San Andrés de Tumaco San Antonio San Calixto San José Del Fragua San Juan de Arama San Martín San Vicente del Caguán Sardinata Segovia Solano Tarazá Teorama Tibú Tierralta Uribe Valdivia Valencia Villa Rica Vista Hermosa Yarumal Zaragoza −1.0 −0.5 0.0 0.5 −0.3 0.0 0.3 Average of Baseline and Endline Difference between Endline and Baseline Treatment a a Control Treatment Mayor Performance by Municipality 35 “We started seeing change with this administration. They built parks, improved roads, built a sewage system and brought drinking water. Before we did not know how money was spent in the municipality, now we know because we have a planning manager, a citizen oversight committee, JACs are involved. The municipality is much better.” Even more, in some cases community leaders explicitly commented on the role of USAID programs. In words of a leader in San Vicente del Caguán, Caquetá: “In the past two years the mayor’s office had a great management capacity, they were doing a lot. USAID was here training them in planning, public spending, social inclusion… This incoming mayor is just starting but we can tell they are well organized.” Moving to a more specific indicator of mayoral performance, a key activity that shapes the perceptions of mayors is public accounting events —rendiciones de cuentas. In these events, the mayors can interface with constituents, explain their management and spending decisions, and take questions. According to the theory of change, rendiciones should provide spaces for constituents to connect with mayoral offices, build trust and legitimacy. They should generally improve public relations and, by extension, perceptions of the community towards the mayor’s offices. Figure 24 shows that the number of public accountings decreased from baseline to endline, and fell more in treatment areas relative to controls (however, missing data for some municipalities may belie this general finding as 2019 data is only available for 25 municipalities. The decline could perhaps also be a function of taking measurements at different phases of the municipal public financial management (PFM) and budgeting cycle and the mayoral terms at baseline and endline). Even with the limited results on public accountings, community members did meet with officials through other channels, such as the juntas comunales and citizen oversight committees. Figure 24: The number of public accountings (Rendiciones de Cuenta) 36 Given a shortage of observations on public accountings, qualitative interview material is needed to supplement this analysis. As previously mentioned, for many citizens as well as community leaders, lack of transparency remains one of the most salient issues for participation and trust in local government. Although “rendiciones de cuentas” are more frequently mentioned in the endline than the midline for treatment and control municipalities, these events are still seen as ineffective in communicating municipal plans and budgets. As confirmed by a focus group participant in Planeta Rica, Córdoba: “The mayor does the “rendiciones de cuentas” but they are not useful for us. If people want to participate or make a comment they need to register in advance. If you go to the meeting to ask a question they won’t let you talk, you need to tell them what your question is in advance.” Citizens were also asked whether they agreed that mayors “inform the community” about mayoral activities. Consistent with the administrative data on rendiciones, citizens report that their mayors do not adequately inform communities about their activities. There are large decreases in control municipalities (over 13 percentage points) but also small decreases in treatment (2 percentage points). See Figure 25. Figure 25: Mayor Informs the Community about their Mayoral Activities Interviews with community leaders and focus groups also indicate mayors do not adequately inform communities about their activities and expenditures of tax payers’ contributions and royalties. Lack of information from the mayor was most evident when citizens reported on investment of public funds, tax collection, and differences between public spending with royalties or with tax money. A focus group participant in Tumaco offered a representative response when asked to recount which public works were completed with tax revenues and which ones with royalties: “We have no idea, we don’t see that kind of information. We have no idea about financial activity of the municipality. Here no one knows what resources come in or how they are spent” The modest results on informing communities are further reinforced by general perceptions of whether the municipal budget is well invested. Table 7 shows that the percentage of references by community 37.0% 39.0% 25.0% 42.0% 44.0% 15.0% 55.0% 34.0% 12.0% 50.0% 37.0% 13.0% Baseline Endline Control Treatment Never Sometimes Yes, Regularly Never Sometimes Yes, Regularly 0.0 0.2 0.4 0.0 0.2 0.4 Mayor Informs the Community Percentage 37 leaders about budgets being well-invested declined from baseline to endline (16 percentage points in control and 15 percentage points in treatment), and fell more steeply from midline to endline for treatment municipalities (6 percentage points in control and 12 percentage points in treatment). Thus, treatment municipalities showed some resilience from baseline to midline, but worsened dramatically from midline to endline. Curiously, several community leaders in treatment municipalities abstained from giving an answer when asked about their perceptions of municipal investment. The percentage of NA responses in control increased by only 5 percentage points whereas in treatment increased by 17 percentage points. This may have been a source of some sensitivity, meaning that the non-response could be masking some underlying and likely negative concerns. Table 7: Is municipal budget well invested? Is municipal budget well invested? Community Leaders Control Treatment 2015 2017 2019 2015 2017 2019 Yes 48.3% 37.9% 31.9% 42.1% 38.7% 27.4% No 27.6% 39.9% 38.9% 29.2% 36.9% 26.0% NA 24.1% 22.2% 29.2% 28.7% 24.4% 46.6% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% General Declines in Perceived Corruption Consistent with the baseline and midline, we analyzed citizen perceptions of corruption among the municipal leadership. RGA was not specifically tasked with working on corruption, but it represents one indicator of whether citizens feel that exercising their rights and responsibilities will be efficacious. It also speaks to the question of whether social oversight among citizen groups is working, or is perceived to be working. Indeed, in numerous reports, RGA reports anti-corruption activities, including for example, “Citizen participation is a mechanism for transparency, as it allows citizens to oversee municipal decisions like public spending and public procurement, thus mitigating corruption” (Year 3, Annual Report). In contrast to deteriorating results for some mayoral activities, there are some positive shifts in perceptions of corruption. As Figure 26 shows, perceptions of corruption of public officials persists but have decreased over the life of RGA. Survey data shows that perceptions of corruption decreased by 17 percentage points in treatment municipalities and 11 percentage points in control municipalities. Per Figure 26, most of the change is driven by the survey respondents who shifted away from the category of “strongly agreeing” that corruption is present. This is a highly positive development, though it is not statistically significant and not clearly associated with RGA. Still, the large magnitude of the widespread decline suggests positive shifts in citizen perceptions of municipal leadership. 38 Figure 26: Percent Agreement that Corruption is an Official Problem, Over Time and Across Treatment/Control We also considered community leaders' perceptions on corruption. Central to the “tax-evasion” culture, is the deeply-rooted belief that corruption is endemic to Colombian politics writ large. While attitudes and dispositions regarding corruption have not profoundly changed in the last five years, we do note some positive improvements in the way community leaders express their concerns regarding corruption. In baseline interviews, social and civic leaders were severely critical of their leaders across control and treatment municipalities. At endline, leaders still think that there is a lot of corruption and lack of clarity about municipal spending. However, they seem to trust their leaders and consider corruption a structural problem. That is, respondents appear to worry about corruption but do not feel as if the local politicians are opportunistically more corrupt, and rather that there are much deeper issues that can only change over the long haul. Interviews with community leaders corroborate the results from the citizen survey by suggesting a less severe problem or one that is less linked to individual politicians. Echoing the idea that the corruption is structural, a leader from Tierralta, Córdoba, noted that: “Political institutions in Colombia are very fragile and they favor corruption. Our leaders justify anything, their offices give viability to any investment that they [the leaders] feel like. In that way they are safeguarded and can take advantage of the resources of the municipality.” 11.0% 89.0% 9.0% 91.0% 22.0% 78.0% 26.0% 74.0% Baseline Endline Control Treatment Disagree Agree Disagree Agree 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 Corruption a Problem? Percentage 39 Limited RGA Effects on Participation We considered the extent to which citizens participate in the social and political lives of their communities by examining their engagements with several different activities and organizations. Perhaps the activity most germane to RGA is participatory budgeting. Figure 27 displays the levels of social participation in a variety of municipal activities, and only one outcome is statistically significant, indigenous marches (Mingas), but this increase is not related to RGA activity, indicating no effect of RGA work on participation. Participation in the various events and organizations we surveyed about did not markedly increase over time. However, involvement in participatory budgeting improved more in treatment than in control, likely due to RGA activities, though the result is not statistically significant. Participatory budgeting occurred at higher rates in both treatment and control from baseline and midline but dropped sharply by endline. . This decline could reflect differences across particular moments of the municipal public financial management (PFM) cycle, where more public accountings are provided during the early budget development phase of the cycle at the beginning of mayoral terms (which occurred around the time of our baseline measurements). However, the endline percentage in treatment remained much higher than in control, suggesting a smaller drop off where RGA operated. Broader structural factors could have universally reduced participatory budgeting, but the positive effect suggests RGA was nonetheless able to encourage participatory budgeting activity despite the general decline. This could be interpreted as a success. Figure 27: Participation in participatory budgeting and other activities Broader political participation, which would be an indirect function of RGA, did not change much from midline to endline. In the endline, a larger proportion of municipal administrators said they felt political participation remained the same in their municipalities. Similarly, there were no significant changes in the perception of community leaders or focus group participants about how easy or difficult it was to participate in politics. As in the midline, both community leaders and focus group participants reported that electoral participation is easy in the endline. Similarly, they indicated that more voting tables had been 40 installed and security conditions improved, making people feel free to vote for whichever candidate or political party they like, though it is unclear what these measures and context are due to. In words of two community leaders, the former from El Bagre, Antioquia and the latter from San Juan de Arama, Meta: “[participating in politics] is easy, we don’t get threats anymore, here people are interested in politics and we can participate freely.” “Now we have the security conditions [to participate in politics]. Thank God we have diversity of thought...it was hard before, groups like FARC, they would call us and threaten us so we wouldn’t run or vote. Now we can participate everywhere.” Focus group participants also pointed out the high number of candidates that have participated in the past local elections, or the number of council members in the municipality as examples of how easily it has become to run for popular vote offices. Nevertheless, Community leaders and focus group participants do recognize that to have these positions you need a political patron and a certain economic status. In words of a community leader from Tierralta, Córdoba: “It is hard because politics work like the market. Before the elections you know who is going to get elected and who isn’t for two reasons: First, because you know who can invest a lot of money, second because you know who is backed by armed groups, or the electoral authority, or the police and military… the government and the police are witnesses of electoral fraud and they say nothing because they are directly involved.” Citizen Oversight Groups (Veedurías) To understand citizen oversight groups, we focused our interview sampling to include community leaders that had specifically participated in citizen oversight groups. We interviewed community leaders that MSI identified as participants of the RGA program or that had been involved in some kind of oversight in their community. Across treatment and control areas, 55% of the community leaders we interviewed were or had been part of citizen oversight committees (see Table A2.4 in Annex 2). We found a meaningful difference in knowledge about citizen oversight groups between community leaders and lay citizens who participated in focus groups. Across focus groups, a strikingly low number of participants in both endline and midline reported awareness of oversight groups in their municipalities, despite the evaluation team’s direct efforts to include oversight committee participants in endline focus group selection. For some, oversight is limited to government watchdog agencies, as expressed to us during a focus Group in Valdivia, Antioquia: “We are from the rural area of the municipality. We know all municipalities need to have oversight groups (veedurías) like the Contraloría. The problem with this oversight is that they never go outside the urban area of the municipality. [Government] Veedores never come here to ask about our demands.” 41 For community leaders, the two most important motivations for participating in oversight groups were ensuring projects met the community’s needs and contributed to their well-being and to verify that municipal budget was well spent (Table 8). We did not observe significant changes between treatment and control municipalities from midline to endline (nor from baseline to endline) in what motivates community leaders to participate in oversight groups. However, we did observe some variation in what discourages community leaders from participating in citizen oversight groups, including in particular security issues in treatment municipalities. In treatment municipalities, lack of interest from the community was not as important in the endline as it was in the midline. In 2017, community leaders in treatment and control municipalities held similar views about general apathy and organizational difficulties in communities as presenting obstacles to organizing citizen oversight groups. In the endline interviews with leaders, we did not hear about this obstacle as much in treatment areas, although in control municipalities the narrative continued. Lack of training or knowledge about veedurías was also not as salient in 2019 as in 2017 for both control and treatment municipalities. This similarity could stem from our sampling approach for interviews that intentionally prioritized leaders with oversight experience, community elites more generally did not report receiving any training from RGA specifically. In municipalities with RGA programming, community leaders manifested fewer problems with organizing into oversight groups and less disinterest from the community from midline to endline. But security issues came to pose a greater obstacle than in the midline. At the endline, security issues were an especially salient obstacle for community leaders in treatment municipalities for performing oversight activities. Table 8 lists the various motivations or deterrents for participating in oversight committees and shows an increase in participants reporting security issues from 3.4% to 13.7%, whereas no difference is observed in the control (4.8% to 4.65%). Table 8: Factors Discouraging Participation in Oversight (Community leaders) Control Treatment 2015 2017 2019 2015 2017 2019 Bad perceptions of overseers (veedores) 0.0% 4.9% 2.33% 6.0% 4.4% 0.0% No impact of veedurías 0.0% 2.4% 4.65% 4.3% 3.3% 7.8% No interest from community 15.5% 17.1% 13.95% 9.5% 17.8% 7.8% No resources (time and money) 4.8% 3.7% 6.98% 4.3% 8.9% 9.8% No training or knowledge about oversight (veedurías) 16.7% 30.5% 16.28% 15.5% 21.1% 9.8% Obstacles to organizing 9.5% 6.1% 9.30% 6.9% 8.9% 3.9% Relations with local government 11.9% 9.8% 6.98% 7.8% 7.8% 5.9% Security issues 4.8% 3.7% 4.65% 3.4% 5.6% 13.7% Corruption 0.0% 0.0% 4.65% 0.0% 0.0% 2.0% NA 36.9% 22.0% 30.23% 42.2% 22.2% 39.2% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 42 Comments from a number of community leaders illustrate the security challenges with forming and maintaining veedurías. As a participant from the focus group in Tumaco said: “I wouldn’t like to join a veeduría due to security conditions. If you want to do things right you risk your life.” A community leader from Valdivia, Antioquia, puts it in a more direct way: “Wherever there is a veeduría there is an armed group. The same happens with social workers, they are killed. Armed groups target them” Another obstacle frequently mentioned by the participants of our focus groups was the stigmatization of leadership and involvement with oversight activities. As a participant from Planeta Rica noted, “[If you join a veeduría] people judge you, they say you think you are better than them or they say you have no authority to do that.” Similar to community leaders, citizens in focus groups also mentioned education as an obstacle to form veedurías as well as the limited impact they have on municipal management. For these citizens the risks and difficulties of forming an oversight group surpass the perceived benefits. As a leader from Peque, Antioquia, puts it: “We’ve tried [forming a veeduría] but we haven’t been able because most of us in the junta only went to primary school. To have a veeduría we need educated people that know how to ask questions and understand administrative processes” We also spoke with municipal administrators about their work with citizen oversight groups. In general, perceptions about these organizations and their contributions to local governance and accountability were positive (See Table A2.5 in Annex 2). However, it was also common to hear disclaimers about how veedurías were “politicized” and suspicions that they look for ways to thwart the work of the mayor. As a municipal administrator in Montelibano, Córdoba, said: “Well-managed oversight groups can help the mayor’s office. If oversight groups are objective, they can be very good. Sometimes they obstruct our work when they have political biases” According to the community leaders we interviewed, overseeing public investments is still a risky endeavor. As previously mentioned, while security conditions have improved for the general population, social leaders remain at risk. In our interviews with leaders, this new context hampered oversight initiatives in both treatment and control municipalities. Unlike most community leaders--who either had already participated in veedurías or manifested interest in doing so--focus groups participants were slow to participate in oversight mechanisms, such as veedurías. Despite focus group participants indicating that residents generally had little knowledge about the veedurías, they were still too afraid to participate or viewed them as ineffective. 43 Still, clearly there is a lot of participation in some isolated contexts, though not many. A community leader from Tibú, Norte de Santander said, “In general, citizens participate in oversight activities (hagamos control ciudadano). Our organization has the capacity to oversee all resource-related public policies” Although the veedurías may sometimes be perceived as dangerous, the community-centered nature of Juntas de Acción Comunal (JAC) makes them an easier–and less risky—form of political participation and oversight. According to activity reports, MSI worked with JACs in various places during the RGA project. Focus group participants observed that it is easier to participate in a junta, provided that one has political support. The juntas are perceived as intermediaries between the mayor’s office communities. Representatives of the mayor’s offices in both treatment and control municipalities have reported increased interactions with Juntas over the life of RGA. The relation with Juntas is seen as fundamental to their decision-making processes through “mesas de trabajo” (working groups). However, participation, transparency, and accountability are still constrained by the effects of electoral politics. As a participant in a focus group in Montelíbano stated: “What happens is that the [current] Juntas are not 100% supported by the municipality, just a fraction of them. If the Junta is not from the same political party of the [current] administration, [the municipality] does not support it. The Board does poorly, and no one is going to open a door for them.” Indeed, there is some indication in the statistical results that community action boards (Juntas de Acción Comunal; JACs) in RGA areas increasingly came to function as important organizational platforms for forming oversight committees (See Figure 28). (Note that other measures of oversight were not significant, as reported in Table A2.9 in Annex 2d, as well as Figures A2.22-A2.24 in Annex 2.) Figure 28: Does the community action board help with the oversight of public works? 44 This pattern is consistent with RGA’s emphasis on supporting the JACs and encouraging their engagement in oversight (RGA Annual Report, Year 4). As one community leader from Caquetá stated in the RGA Annual Report (Year 3): “I was a member of my community action board, but now I do advocacy work in the entire municipality…. For me, it is very important to build peace through oversight groups” During our interviews, almost all explicit references to RGA came from community leaders when they spoke about the citizen oversight groups in their municipalities. Community leaders recognize that education and training programs directed towards oversight committees have been helpful to both create and strengthen the committees, as well as to educate community members about how local government works and their rights as citizens. As a leader from Nechí, Antioquia observed, “RGA has given us knowledge about the administrative structure of the local government, the Local Development Plan, and how to do oversight. It has been important for all organizations in the municipality. We can now make demands on the local government” Oversight trainings were also deemed as good opportunities to build new leadership in municipalities and to autonomously replicate citizen oversight in new areas. As a Community leader in Nechí Antioquia explains, “I’m part of the victim’s organization that participated in RGA and we now go to schools and elsewhere to teach what we learned. We are training a group in SENA so they can do oversight of the municipal public health system” Effective veedurías in the health sector may be especially important now given the onset of COVID-19 and the need for well-managed healthcare. There have been several reports of corruption in contracts and over-spending in efforts to address the COVID-19 crisis.28 In April 2020, President Ivan Duque called on Colombian citizens to exercise oversight of contracts and spending to prevent corruption.29 Future programming to promote citizen oversight could be tailored to the specific exigencies of pandemic response. Unclear Impacts or Null Results RGA appears to have had an impact on some isolated outcomes, mostly positive, albeit sometimes negative, as discussed above. Yet, it is important to note that most of the statistical results generated by comparing changes over time and between treatment and control groups are not statistically meaningful. Indeed, many of the analyses indicate unclear or null effects despite hopes or expectations that the RGA 28 https://www.semana.com/noticias/corrupcion-y-coronavirus/ https://www.eltiempo.com/economia/sectores/coronavirus-en-colombia-aumento-de-la-corrupcion-en-el-pais￾541345 https://transparenciacolombia.org.co/2020/08/18/persisten-riesgos-contratos-covid19/ https://fr.reuters.com/article/salud-coronavirus-colombia-corrupcion-idESKBN22Y0LH https://www.elespectador.com/opinion/corrupcion-coronavirus-y-veedurias-ciudadanas-columna-914681/ 29 https://id.presidencia.gov.co/Paginas/prensa/2020/Presidente-Duque-hace-llamado-ciudadanos-actuen-como￾veedores-denuncien-oportunamente-manipulacion-funcionarios-publ-200416.aspx 45 would improve these outcomes. We emphasize that statistical correlations rely on a host of assumptions and depend on accurate measurement. We went to great lengths to validate measurement, but it is important to take a holistic approach to the evaluation data, weighting qualitative interview and focus group material heavily. We highlight a number of key Component 2 and Component 4 analyses that suggest null or unclear results. We cannot discuss all of the unclear results, but note that the impact on key indicators is all summarized in Annexes 2b and 2c (by Component) and Annex 2d (full regression results). As the tables illustrate, most results are null, suggesting no RGA impact in many areas. A couple of results are negative. Among the non-results, a few are particularly notable. On Component 2, there are no changes in municipal fiscal performance nor are there improvements in the municipal performance index (MDM), with control municipalities much higher than RGA municipalities and with slightly more improvement over midline compared to RGA municipalities (See Annex 2). Indeed, with the MDM the results are negative at endline, though importantly the data are only available through 2018 and need to be contextualized. Additionally, there are further null results for various types of revenues, though of course not all types can be influenced directly by RGA, and many of the attitudes on taxation have not changed much. See Annex 2. To summarize findings on Component 4, social participation levels are mixed, with some improvements in oversight, but a lack of participation in activities, such as public accountings. More broadly, the social environment has not been favorable to social participation. The 2015-2020 period saw little engagement of community leaders with mayoral offices and rising security threats that deterred citizen activism. Given the challenging environment, it is not surprising that the impacts of RGA might be limited to isolated areas of focus. Still, the improvements in citizens’ oversight knowledge and skills could be considered a modest success and could pave the way to greater oversight of assistance funds for the COVID-19 pandemic. Region-Specific Dynamics In addition to assessing the net effects of RGA, we also conducted sub-national, regional analyses on security context and RGA impacts. We first considered contextual security dynamics, including those that affect various forms of participation, by region. Table 10 reports difference-in-differences analysis for a number of security and outcome indicators by region and shows very few regional differences (but note that the results within regions reported in Tables 10 and 11 have low statistical power). 30 Reported perceptions of security rose in Catatumbo, but the homicide rate rose in Cauca/Valle de Cauca, each significant at only the 10% level. Otherwise, regional differences in security were not pronounced. 30 See earlier note in this report, the original SOW, and earlier evaluation rounds for more background on difference-in-differences analysis. 46 Table 10: Regional Differences in Key Security and Participation Measures Outcome Arauca Catatumbo Cauca/ Valle Central Norte Sur Easy to participate in political life -0.03 0.02 -0.01 -0.02 0.02 -0.13 Discouragement to do oversight 0.00 -0.21 0.43 -0.27 0.44 0.00 Participatory budgeting 0.01 0.00 -0.01 -0.02 -0.02 -0.06 Municipal responsiveness -0.56 0.37 0.05 0.36 0.18 -0.58 Feelings of safety 0.01 0.18* -0.08 -0.07 -0.01 -0.02 Security conditions -0.14 0.22 -0.11 0.05 0.09 -0.17 Impact of armed groups on Mayor's office -0.15 0.09 0.03 0.30 0.02 -0.33 Peace process support -0.05 -0.02 -0.17 0.19 -0.31 -0.68 Armed groups allowed to move freely -0.20 0.03 0.07 0.07 0.08 0.09 Homicide rate -8.92 27.93 60.81* 9.11 -0.09 48.51 Victim of a crime in last year -0.04 0.00 -0.02 0.04 0.01 -0.25 Based on differences-in-differences analysis; cell values indicate the relative improvement of treatment to control municipalities within each region, based on the scales for the different survey questions from Endline to Baseline. Arauca values are based on differences between Endline and Midline due to missing baseline data. Data is missing for the South region (Tumaco and Putumayo). *=Significant at 0.1 level. We then calculated differences-in-differences for the municipal mean values within each region for key outcome indicators from the survey from baseline to midline to endline (comparing relative treatment group versus control group changes by region). Although these results are not statistically significant given the small intra-regional samples of municipalities (and consequent low statistical power), they provide some look into where RGA programming seems to have had stronger or weaker effects. Table 11 shows the shifts in outcome indicators by regions that might be attributable to RGA activities (because of the small number of municipalities in the Sur region and the lack of baseline data for Arauca, the results for these regions are displayed but not analyzed more deeply). The Central and Norte regions saw the greatest improvement in citizens’ ratings of trust in the mayor’s office, whereas the Arauca and Cauca/Valle regions saw relative declines in performance (apparently) related to RGA (See Table 11 and Figure 29). The Catatumbo and Central regions are far above the other regions in terms of RGA-associated shifts in municipal engagement with the population as gauged by increases in perceptions of municipal authorities inviting the community to express their opinions and taking citizens into account in decision-making. Community leaders in the Central region confirm this perception, especially leaders from Meta, but offer an important caveat: expressing concerns and needs does not necessarily mean those claims will be included in governmental plans and actions. A community leader from Vista Hermosa, Meta said, “They [the mayor’s office] meet with us and they do listen but then everything has to go through bureaucratic processes, and I have been continuously let down.” The Central and Norte regions saw the greatest RGA-associated improvement in a culture of paying taxes as indicated by the municipality being higher rated in effectiveness in collecting taxes and perceptions that paying taxes is a civic duty. In terms of improvements in public services provision, Catatumbo and Norte saw the largest RGA-associated improvements in the provision of electricity and water services. 47 Table 11: Regional Analysis of Relative RGA-Associated Changes in Key Outcome Indicators Outcome Indicator Arauca Catatumbo Cauca/ Valle Central Norte Sur Confidence in Mayor's office -0.15 0.09 0.03 0.30 0.02 -0.33 Municipio invites community to express opinions -0.56 0.37 0.05 0.36 0.18 -0.58 Municipio takes community into account for decision￾making -0.35 0.15 0.06 0.22 0.12 -0.69 Municipality is effective in collecting taxes 0.30 -0.08 -0.01 0.41 0.52 -0.02 Paying taxes is a civic duty 0.44 -0.20 -0.20 0.26 0.31 0.66 Access to electricity 0.00 0.00 0.00 0.03 0.00 -0.03 Electricity service rating 0.03 0.33 -0.06 -0.25 0.07 0.84 Access to water -0.01 0.03 0.00 0.09 0.00 -0.05 Water service rating -0.13 0.30 -0.12 -0.11 0.08 1.00 Know RGA -0.02 0.01 0.07* -0.01 0.02 0.00 RGA in village/ municipio 0.03 -0.42 -0.05 0.13 -0.38 0.11* RGA beneficiary -0.01 0.00 0.06* 0.00 0.03 0.02 Based on differences-in-differences analysis; cell values indicate the relative improvement of treatment to control municipalities within each region, based on the scales for the different survey questions from Endline to Baseline. Arauca values are based on differences between Endline and Midline due to missing baseline data. Data is missing for the South region (Tumaco and Putumayo). *=Significant at 0.1 level. Qualitative accounts indicate that, across all regions, community leaders’ trust in mayoral offices declined. Positive references about trust in local governments declined by around 20 percentage points from baseline to midline. In the Norte region, the cases of El Bagre and Nechí, Antioquia stand out as mayors in both cities had corruption problems that forced them out of office.31 A municipal administrator in Nechí described the governing instability as “administrative trauma.” The same administrator noted that they worked with the veedurías to encourage better relations between citizens and the mayor’s office to sustain trust during those difficult times. This comment suggests that, even if interactions between community members and mayors are fraught, local administrators can still draw on political participation mechanisms to productively engage with communities. The takeaway with respect to RGA is that regional contextual variation may have affected RGA outcomes in key ways. 31 https://web.archive.org/web/20190408162931/https://www.elcolombiano.com/antioquia/nechi-no-sabe-quien-es￾el-alcalde-tras-condena-contra-miguel-franco-DM10492357 48 Figure 29: Regional Differences-in-Differences in Mayoral Performance (Baseline-Endline) As far as awareness of the RGA program, we assessed surveys responses on whether citizens know of RGA, whether they know it is operating in their village or municipality, and whether they reported the RGA program being active in their village or municipality. Citizens generally reported greater awareness of the program in the Catatumbo and Norte regions (in treatment relative to control municipios) relative to other regions (See Figure 30). Figure 30: Regional Differences-in-Differences of RGA in Village/ Municipio (Baseline-Endline) Interviews similarly show that more community leaders in the Norte region have knowledge about RGA activities than leaders from other regions. As in other regions, these leaders emphasize the work with RGA in organizing veedurías. In the Centro region, community leaders also mentioned RGA’s “support in budgeting and financial execution to improve efficiency of the administration” (Community leader, 49 Planadas Tolima). Others mentioned learning about participatory planning and budgeting, especially around PDETS as an outcome of RGA’s activities. Vulnerable Populations While much could be considered with respect to vulnerable populations, we report a few interesting results here.32 Contextually, it appears that, at baseline and endline, the rates of victimization are relatively similar for women and men, but at endline, a higher proportion of men were targeted more often than women. See Figure 31. Figure 31: Victimization by gender and across waves Some interview respondents and focus groups participants in the baseline, midline, and endline mentioned domestic violence and gender-based violence as key security issues they felt were not being adequately addressed by the mayor’s office or the police. As one community leader from Tibú observed, "Security improved somewhat over the past two years. We cannot say it is totally safe but we are seeing less violent crimes. Now insecurity is caused by theft, robbery, and domestic violence because people are still machista and women don’t speak up.” We heard commentaries of this type in all three waves of data collection, but there were slightly more references at the endline. 32 The replication data for all three waves of the evaluation are publicly available and much could be considered related to vulnerable populations. 50 There was not much difference between feelings of security across genders despite different rates of victimization. As Figure 32 shows, the percent of people feeling afraid of traveling to the city center during the day and night rose across ethnic groups. The largest increases occurred among indigenous survey respondents, especially in the treatment group. Figure 32: Fear of traveling at day/night by type of vulnerable population : Related to Component 4, in all three waves of the evaluation we asked individuals from ethnic minority groups whether their leaders invited their opinions. As shown in Figure 33, for nearly all groups, the average levels dropped over time. These results suggest that, at least for one key index, RGA activities may have been less effective at benefitting these vulnerable populations, or at least that there was a slow deterioration of intertest in individuals from those ethnic minority groups. 51 Figure 33: Do municipal leaders invite the opinions of ethnic minorities? We also asked individuals in the survey who can be a leader, an indicator of greater social participation and inclusion, and report the results for four groups traditionally underrepresented: women, Afro￾Colombians, former combatants, and LGBTQ individuals. Figure 34 shows the results, in particular, there is increasing support for all kinds of people being leaders in Colombia. Most survey respondents agree that a woman (93% in control, 95% in treatment) or Afro-Colombian (84% in control, 86% in treatment) can be a leader in their community. There were improvements in treatment (8 percentage points) for whether a LGBTQ person can be a leader—consistent with MSI/RGAs emphasis on inclusion of that population in veedurías. But, overall, people provided more mixed support (56% in control, 58% in treatment) as compared to views on women and Afro-Colombians. Few people support former combatants (20% in control, 22% in treatment), and there was little change over time. The result for women as leaders is statistically meaningful, though the others were not. 52 Figure 34: Who Among These Groups Can Be a Leader in the Community? Evaluation Recommendations and Conclusion The RGA treatment and control municipalities comprise a sector of the Colombian population that, in recent years, has largely experienced deteriorating conditions and a mixed outlook on the future. The 40 municipalities chosen for the RGA program were considered insecure, lacking in trust and legitimacy and with poor public financial management and citizen oversight. The results at the endline of the program indicate some piecemeal impacts but not the comprehensive impacts envisioned by the RGA’s program design. The program faced an extremely difficult implementation environment, and based on the theory of change, originally expected changes on a longer time-horizon. The findings tell a story that is consistent with some of the expectations of the theory of change. The results for a few outcomes shifted in the expected direction. Unfortunately, the statistical results are mostly not meaningful, suggesting little impact of RGA on broader outcomes of interest. Revenues have increased, likely because increasing funding is a relatively rapid task to accomplish. Also, per expectations, other downstream elements of the theory of change—such as starting programs, providing services, addressing corruption, and improving governance—show mixed results and have moved relatively slowly. If the theory of change is correct, increases in revenues and subsequent increases in municipal capacity and oversight should generate larger gains in future years, though the evidence for such a trajectory at this time is limited. There are some limitations to the scope of these impact assessments. The theory of change anticipates that some elements of RGA are likely to shift more rapidly than others, and this is what is observed. This includes outcomes that result from relatively simple institutional changes or that are directly affected by 11.0% 89.0% 11.0% 89.0% 7.0% 93.0% 5.0% 95.0% Baseline Endline Control Treatment No Yes No Yes 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 Percentage A Woman? 20.0% 80.0% 20.0% 80.0% 16.0% 84.0% 14.0% 86.0% Baseline Endline Control Treatment No Yes No Yes 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 Percentage An Afro−Colombian? 82.0% 18.0% 77.0% 23.0% 80.0% 20.0% 78.0% 22.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Percentage A Former Combatant? 46.0% 54.0% 50.0% 50.0% 44.0% 56.0% 42.0% 58.0% Baseline Endline Control Treatment No Yes No Yes 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Percentage LGBT Person? Who can be a Leader? 53 RGA training and assistance, such as improved public administration or increases in available municipal revenues. Moreover, there are several national-level contextual factors that could shape the effectiveness of the RGA program. These include the conclusion of the peace agreement with the FARC and the transition in presidential administrations. We also were only able to survey within one single electoral period for municipal administrations and mayors, and therefore cannot assess impacts independent of these local political conditions. Because these factors do not vary sub-nationally, we were unable to observe how they may affect RGA activities and outcomes. Conclusions Component 2: Improved municipal financial management and performance ● At the endline, treatment municipalities experienced greater overall revenues relative to control municipalities. Specifically, royalty revenues improved since the baseline, with RGA municipalities receiving more net royalty revenues than control municipalities. Interviews with municipal administrators indicated that they felt that technical training and standardization processes helped them access royalty funds. While this suggests a possible impact of RGA through the OCAD funding support for public works projects, the algorithm for allocating direct royalty revenues is not something that RGA can easily influence. Expenditures on royalty revenues can provide a better measure of RGA’s effects. The results indicate that expenditures tracked with RGA activity and trended higher in treatment relative to control over time, though the result for expenditures is not as definitive. ● For most Component 2 measures, ranging from tax revenues to fiscal and administrative performance, there is little evidence that RGA municipalities experienced substantial improvements. Although at the midline treatment municipalities demonstrated some improvements in fiscal performance relative to control municipalities, those effects attenuated at endline. This could indicate that the gains in financial management among RGA municipalities may be relatively limited to the area of royalties. ● RGA programming is associated with minimal change in attitudes about payment of taxes, which can be critical for generating support for municipal revenues and public works. The RGA program included a public relations campaign to increase education about the importance of tax contributions for the public good. These publicity efforts were either less effective than hoped or, for whatever reason, the survey sample was not broadly exposed to these messages. ● Citizen perceptions indicate there is limited provision of public services across the municipalities sampled. There was also little change in access across different public services and institutions over time. The sole exception is a significant increase in reported access to gas in the treatment group, though gas service is not in the purview of RGA, so some other factor is likely driving that service delivery improvement. It appears that prices are perceived to remain high or are increasing, putting services further from reach. Public services and perceptions about access could continue to improve in the coming years as RGA-mobilized projects deliver on their promised benefits. ● Across the regions where RGA operated, descriptive evidence suggests that RGA programming and improvements in conditions are most visible in the Norte (Antioquia and Córdoba), Central (Tolima/Meta), and Catatumbo (N. Santander) regions for the indicators we examined. The ratings Arauca, Cauca/Valle, and South regions saw fewer overall perceptions of improvements associated with RGA activities, although these conclusions are subject to data limitations. 54 Component 4: Citizen participation for enhanced transparency and accountability ● The peace process with the FARC has alleviated fears in some municipalities and created greater openings for civic organizing and political participation—which may contribute to increased participation related to RGA initiatives, especially in participatory budgeting and citizen oversight of projects. However, in some locations, deteriorating security may have depressed participation in these activities. ● Corruption related to royalty-funded projects has been identified as a risk by the Colombian Contraloría. Fortunately, corruption appears to have decreased across the board over time, both in RGA and comparison municipalities. This broad trend decrease suggests a possible nationwide shift rather than being a result of RGA-specific effects. However, corruption remains a stubborn problem, as interviews and focus groups reveal that community leaders see corruption as a structural problem. Local leaders may therefore be challenged in making large gains against corruption.. ● Citizens still feel municipal authorities are not sufficiently transparent about their activities and expenditures due to lack of accessible information. Although RGA focused on increasing the use of public accountings by mayors to communicate progress on municipal finances and development initiatives, there were not notable changes between treatment and control areas. Citizens similarly did not report improved perceptions of mayors informing about their activities. These patterns could possibly be a function of the public financial management cycle. Relatively more budgeting activity and public engagement may have occurred around the time of the baseline data collection, which coincided with the beginning of the mayors’ terms and their preparation of their municipal development plans. ● Participation levels (not trends) remain high across all RGA and comparison municipalities. But with the exception of a large increase in indigenous marches (Mingas), which RGA did not aim to affect, participation did not increase in treatment areas relative to controls. Despite the absence of an evident broader shift in citizen participation, the qualitative accounts from interviews and focus groups do highlight some anecdotal increases in pro-social engagement in RGA areas. ● Citizen oversight committees generally improved over time and became far more active. Community elites in treatment municipalities attributed this to RGA programming. However, some citizens reported that participation in the oversight committees also meant incurring security risks. A striking difference in knowledge and perception about oversight committees persisted between community leaders and lay citizens. The latter are largely still poorly informed about how oversight committees work and their purpose, suggesting continued openings for education and training on these processes. Summary of Contextual Data ● The peace process with the FARC has been uneven across the country. Most individuals in RGA areas were no more favorable in their opinions of the peace process at the end of the program than they were in 2016. Coupled with growing insecurity, this suggests that RGA faced a difficult implementation environment. ● Although security is not anticipated to be directly affected by RGA activities, the security situation provides important context. We asked citizens whether specific armed groups can move freely in their municipalities, and find that the presence and activity of armed groups remain substantial and statistically meaningful concerns, especially in RGA municipalities. 55 ● Key RGA outcomes such as municipal service provision are slower to be influenced by program activities and perceived by citizens. Public services and works are slower-moving and depend on lengthier implementation and governing processes to generate greater citizen awareness. Some of these effects may only obtain and be observable one or more years after the program has ended and after the endline data was collected. Recommendations for Policy and Programming We discuss seven recommendations for policy and programming here, and include a recommendations actions matrix in Annex Table A1.1, which also includes key audience for each recommendation. ● RGA activities are associated with increased revenue streams, especially in terms of total revenues, of which royalties are a component. These program results should be emphasized early in future programs to help achieve later downstream outcomes, such as boosting trust and combating corruption. Concretely, future programming should take sequencing of activities seriously, and prioritize activities that promise to make a short-run difference and catalyze other outcomes. ● Because service provision typically lags behind increased revenue flows, services should be relatively slower to materialize. Communications strategies should focus on encouraging realistic expectations among the public about the timing of improvements in service provision. Moreover, USAID should consider conducting a follow-up evaluation in a couple of years, even if on a smaller scale, to assess whether revenues and royalty expenditures ultimately result in greater local service provision. ● There is an association between revenue increases and corruption decreases, with resources generally going up and corruption generally going down. The evidence is correlational in both the survey and interviews and cannot provide specific insights about the potential causal processes at play. It is possible that resources are allocated to the best-managed localities, which are also able to maximize the proper use of them. If so, USAID should consider devoting resources to the best￾managed localities so as to maximize proper use of the resources. Alternatively, when resources are provided to some municipalities it could cause them to improve management and reduce corruption. Given the general relationship, USAID should conduct additional analysis to identify which causal processes may be at play. ● RGA programming depends on local social leaders, who face continuing risks of violence in the post-peace agreement period. RGA-style programs should take into account the burdens and exposure they may create for these leaders, and their security concerns should be addressed so they can more safely and fully participate in programs. The longer-term success of Component 4 activities likely hinges on creating a safe context for citizen engagement. If security threats arise in the context of future programming, implementers could more closely engage with the Colombian National Protection Unit (UNP) to request protection for at-risk individuals. ● To boost trust and legitimacy, local elites should be engaged to play greater roles in raising awareness and sharing information about RGA-style programs with the general public. There are divergences in perceptions of RGA programming between local elites and the general citizenry. As influential and trusted individuals in their communities, local elites can help explain how the benefits of programming may contribute to improved livelihoods. Additional publicity activities 56 could engage local elites to focus messaging about programming activities and successes in the more isolated rural areas of municipalities. ● Regional analysis of RGA indicated variations in programming and outcomes across RGA focus areas. Future governance and development programming should consider the region-specific gaps in outcomes as well as the unique contextual challenges of implementing programming. These efforts should continue to prioritize citizen oversight and participation in regions with higher levels of royalties to ensure accountable public investments. We identified some mismatches between the locations of royalty-based projects and where oversight activities were most prevalent. This geographical targeting of oversight will help ensure that there is sufficient citizen oversight in areas with the most resources and public works projects. ● Future programming to promote citizen oversight should be tailored to the specific exigencies of pandemic response. There have been reports of corruption in the procurement of COVID-19 related supplies and services. These contracting challenges could provide a politically popular opening to more sustainably promote and entrench citizen oversight capacities across Colombian municipalities. Specifically, USAID could leverage RGA’s efforts to support citizen oversight committees by providing these committees with additional training to monitor pandemic-related expenditures. Additional committees or oversight processes could also be promoted more widely in other Colombian municipalities. Such initiatives could be framed and presented to the Colombian public as being “essential for protecting public health” (e.g., “Hagamos Control para la Salud”).