This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by EnCompass LLC for the USAID/OTI M&E Support Services Task Order (IDIQ No. AID-OAA-1-15-0021, Task Order No. 7200AA19F00017). The views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. COLOMBIA TRANSFORMA TRIPARTITE MODEL EVALUATION REPORT March 2022 PHOTO | USAID/OTI/COLOMBIA ii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV CONTENTS ABSTRACT AND ACKNOWLEDGMENTS............................................................................................III ACRONYMS AND ABBREVIATIONS...................................................................................................... IV GLOSSARY ...................................................................................................................................................... IV EXECUTIVE SUMMARY............................................................................................................................... VI INTRODUCTION ...........................................................................................................................................1 Evaluation Purpose and Questions................................................................................................................................ 1 Program Context............................................................................................................................................................... 3 Colombia Transforma and the Tripartite Model ....................................................................................................... 5 EVALUATION METHODOLOGY ..............................................................................................................7 Design................................................................................................................................................................................... 7 Sampling ............................................................................................................................................................................... 8 Data Collection Methods................................................................................................................................................. 9 Analysis...............................................................................................................................................................................10 Evaluation Strengths and Limitations..........................................................................................................................11 FINDINGS........................................................................................................................................................13 State-Society and Intra-State Relations......................................................................................................................13 Community Capacity, Empowerment, and Ownership..........................................................................................22 Community Resilience and Adaptability.....................................................................................................................30 Cross-Cutting Findings...................................................................................................................................................34 Contextual Findings.........................................................................................................................................................37 CONCLUSIONS AND RECOMMENDATIONS ...................................................................................42 Conclusions.......................................................................................................................................................................42 Recommendations...........................................................................................................................................................44 REFERENCES...................................................................................................................................................48 ANNEXES ........................................................................................................................................................49 Annex A: Summary of Results by Study Arm...........................................................................................................50 Annex B: Detailed Methodology..................................................................................................................................51 Annex C: Background on Evaluation Team...............................................................................................................61 Annex D: Statements of Difference ............................................................................................................................62 Annex E: Conflict of Interest Disclosures.................................................................................................................63 Annex F: Evaluation Scope of Work...........................................................................................................................68 Annex G: Evaluation Phase 2 Data Collection Tools..............................................................................................75 iii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ABSTRACT AND ACKNOWLEDGMENTS ABSTRACT The United States Agency for International Development (USAID) Office of Transition Initiatives (OTI) engaged EnCompass LLC to conduct an evaluation of the Colombia Transforma program’s tripartite model, as used in both the program’s first (2015–2019) and second (2019–2022) phases. Colombia Transforma used the model to implement small infrastructure projects in rural communities affected by Colombia’s 50-year conflict. In the tripartite model, OTI, the community, and the municipality each contribute resources. Based on prior monitoring and evaluation, the shared responsibility for project implementation was hypothesized to strengthen state-society and intra-society relations, as well as community capacity and empowerment for further local development projects; the model also has the potential to indirectly affect community resilience. This evaluation included three study arms: a baseline/endline, plus two rounds of retrospective; each study arm featured a balanced set of intervention and comparison communities. In two phases of data collection conducted in late 2020 and again in late 2021, this evaluation collected data from 28 communities (equally divided between treatment and comparison) through over 600 interviews and focus groups. The evaluation concludes that the tripartite model has strong effects on community capacity, empowerment, and project ownership. The model also results in clear effects on state-society and intra-society relationships, most notably in trust in local government and community-municipal relations, and to a lesser extent in understanding of the peace accords and perceptions of transparency. Evidence of indirect results on community resilience were not observed. The evaluation offers recommendations to OTI Colombia and USAID Colombia, and other actors. ACKNOWLEDGMENTS This report was authored by Zachariah Falconer-Stout, Cecilia Papariello, and Gina Cabarcas. The team consisted of Zachariah Falconer-Stout as Team Lead, Gina Cabarcas as subject matter expert, Cecilia Papariello as Senior Evaluation Specialist, and Gloria Wazni as Project Coordinator. Maria Jose Lievano, Anais Henriquez, and Valentina Socias Maluenga assisted the team with data analysis. This evaluation uses data collected during the baseline and first retrospective phase, which included team members Oliver Kaplan, Anna Kinzer, Carolina Koldys, and Monica Herrera. Cifras y Conceptos collected the data during both phases, led by Viviana Arias Vargas with team members Juan Luis Camacho Aljure, Liliana Florez, Amanda Cabezas, and Edward Gelvez. Crystal Cason designed the report and Maeve Tobin edited it. Ghazia Aslam was the MEL Advisor for the overarching M&E Support Services Task Order with OTI and supported this team throughout. The evaluation team gratefully acknowledges the support from the Colombia Transforma Team, as well as the OTI MEL Team in Washington, D.C. Both teams provided indispensable guidance and programmatic background, as well as essential support for data collection. Above all, the team thanks the respondents who took the time to provide data, without which this report would not be possible, and the Colombian citizens who are working every day to make the peace process a reality through their actions. iv | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ACRONYMS AND ABBREVIATIONS ART Agency for Territorial Renewal (Agencia de Renovación del Territorio) CT Colombia Transforma FARC-EP Revolutionary Armed Forces of Colombia—Army of the People (Fuerzas Armadas Revolucionarias de Colombia—Ejército del Pueblo) FGD Focus group discussion ICRC International Committee of the Red Cross JAC Community action board (Juntas de acción comunal) KII Key informant interview OTI Office of Transition Initiatives PDET Development programs with a regional focus (Programas de Desarrollo con Enfoque Territorial) SME Subject matter expert USAID United States Agency for International Development GLOSSARY Agencia de Renovación del Territorio (ART): Agency for Territorial Renewal, a public institution designed to oversee the implementation of multiple parts of the peace agreements, particularly the PDETs. Categories for Districts and Municipalities: Districts and municipalities are the lowest decentralized territorial subdivision of local government in Colombia. Law 1551 of 2012 establishes a special category and categories 1 through 6 for allocating public budget to districts and municipalities. All the municipalities where Colombia Transforma works are category 5 and 6, which means they are ranked as the municipalities with less population and less income per year. Consejo Comunitario: Legally constituted political institution for a recognized Afro-Colombian territory. A consejo comunitario can include multiple veredas. The leadership body is a Junta Directiva Consejo Comunitario. The juntas of consejos comunitarios are the principal authority in their territory and some of their roles are similar to JACs. Convenio solidario: Type of contractual agreement between local government and a legally formed communal organization to execute projects of the least value (mínima cuantía). Some Colombia Transforma infrastructure projects used this agreement type; in these contexts, a convenio solidario did not replace the tripartite model, but changes the dynamics of the relationship between the three parties. Through convenio solidario, the community administrated the municipality's funds for skilled labor. Corregimiento: Territorial subdivision within a municipality that includes more than one vereda. JAC: Community action organization comprised by all the persons that live in a particular territory and are not part of a different communal organization. The board is appointed by popular vote and are in charge of making administrative and social decisions affecting a particular territory. Each vereda (for rural areas) or barrio (for urban areas) can have a JAC. Also referred to simply as juntas. v | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Junta veredal: Afro-Colombian community leaders at the vereda level. Critical mass: As operationalized by OTI, this concept refers to both density of projects within a geographic area (density of multiple projects within a municipality or corregimiento) and also about the density of participation within each project (multiple community actors participating in a specific project, resulting in ownership in that project). Put together, these two elements of critical mass are hypothesized to lead to a snowball effect facilitated in part by word of mouth, as new communities and new community members gain interest in how they can realize their own projects, and previous project groups explain the way they came together to achieve their own project. That is, previous project groups explain how the tripartite model works. Peace accord: Peace negotiations conducted between 2012 and 2016 between the Colombian government of Juan Manuel Santos and the Revolutionary Armed Forces of Colombia (FARC–EP) to bring an end to the 60-year armed conflict. The final agreement was signed September 2016, but it lost by a slim majority (less than 0.5 percent) in a popular referendum in October 2016; portions of the text were renegotiated, which led to a revised text that was ratified by Congress in November 2016. Programas de Desarrollo con Enfoque Territorial (PDET): Development programs with a regional focus, an instrument designed to implement the peace agreements. Sixteen regions were selected because they comprised the zones most affected by the armed conflict. Programa Nacional Integral de Sustitución de Cultivos Ilícitos (PNIS): Integrated National Program for Illicit Crop Substitution, the instrument designed to implement chapter four of the Peace Accord, which relates to countering drug trafficking and its effects. Vereda: Smallest administrative subdivision of a municipality in Colombia for rural areas. In this report, they are also referred to as communities and study sites. vi | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY This report presents the findings, conclusions, and recommendations from a two-phase external evaluation conducted by EnCompass LLC of the United States Agency for International Development (USAID)/Office of Transition Initiatives (OTI) Colombia Transforma program. The evaluation focuses specifically on the program’s tripartite model for implementing small infrastructure projects in rural conflict-affected areas, a key component of the broader Colombia Transforma program. Evaluation questions are organized into domains corresponding to the program’s direct results (state-society and intra-society relationships and community capacity, empowerment, and ownership), potential indirect results (community resilience and adaptability), and contextual influences on the program. Evaluation results will inform adjustments to the third phase of Colombia Transforma; results could also support strategy in other OTI country programs or USAID/Colombia Mission programs engaged in community development projects. The primary evaluation audience is OTI/Colombia and USAID/Colombia. Secondary audiences include other OTI country programs, the Government of Colombia and its agencies, and other development actors in Colombia. PROJECT BACKGROUND Colombia Transforma began implementation in 2015 and has since carried out more than 700 small infrastructure activities across Colombia with the goal of supporting government entities in quickly delivering on promises from the 2016 peace accord between the Colombian Government and, the Revolutionary Armed Forces of Colombia (FARC-EP). After more than 50 years of intractable armed conflict, the peace accord opened the door to peace with the largest guerilla insurgency and inclusive development in poor rural areas where most of the conflict took place. Colombia Transforma takes advantage of this opportunity by creating connections between government at all levels and communities, and by working with community-level organizations to support collaborative community development processes that can create an enabling environment for durable peace in the long term. The first phase of Colombia Transforma (CT 1) ran from July 2015 to July 2019 and the second phase (CT 2) ran from July 2019 to February 2022. This evaluation focused on tripartite model small-scale infrastructure projects implemented in both CT 1 and CT 2; OTI launched a third phase of the program (CT 3) as this evaluation was concluding. Evaluation Questions 1. To what extent and through what mechanisms does the tripartite model create or strengthen relationships between local governments and community members? 2. To what extent does participation in the tripartite model empower community members and leaders? 3. To what extent does participation in the tripartite model increase community members’ and leaders’ capacity to manage local development processes? 4. To what extent does participation in the tripartite model increase communities’ resilience and adaptability as an indirect program outcome? 5. How has the context affected the implementation of the tripartite model? vii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Through trial and error, CT 1 developed a methodology for implementing small infrastructure projects that came to be known as the tripartite model. In this model, OTI, the community, and municipality each contribute resources and share responsibility for the project: the municipal government provides skilled labor, the community volunteers manual labor, and Colombia Transforma provides in-kind materials and equipment. All three parties participate in planning and implementation. Beyond fulfilling the immediate community need for infrastructure, the tripartite model has the potential to strengthen relationships between municipalities and communities, and enhance community capacity to manage local development processes. All Colombia Transforma activities are intended to directly support peace accord implementation; this evaluation also investigates whether the tripartite model has the potential to indirectly support community resilience and adaptability. EVALUATION METHODOLOGY The evaluation consists of two phases (Phase 1 and Phase 2) that collected data under three separate study arms: Retrospective 1 (conducted during Phase 1), Retrospective 2 (conducted during Phase 2), and a baseline/endline study arm (conducted at both phases). Each study arm features comparisons between treatment and comparison sites. The evaluation questions are the same across the study arms, but each study arm focused on a distinct sampling frame of tripartite model projects in order to collectively represent the program’s breadth. Retrospective 1 focused on tripartite projects implemented during CT 1, representing the model’s earlier stages and the pre-COVID-19 pandemic era. The baseline/endline study arm sampled from CT 2 tripartite model communities, representing the standard application of the model, albeit under compressed implementation timeline due to the evaluation and COVID-19. Retrospective 2 sampled from tripartite model communities that had been granted a convenio solidario by the municipality, a special legal mechanism that allows Colombian Government entities to grant funds directly to the community; some of the Retrospective 2 sample communities also represented critical mass zones where Colombia Transforma had achieved a heightened density of implementation. Phase 1 data collection occurred December 2020–March 2021. Phase 2 data collection occurred November–December 2021. Data collection methods for all study arms are primarily qualitative, though supplemented with some closed-ended quantitative questions. While data are primarily qualitative, the sampling reflects a two-stage design that blends probability sampling at the first stage (intervention/comparison community site sampling) and purposive sampling at the second stage (selection of individuals). Comparison sites are then selected based on matching techniques to construct a counterfactual. While first-stage sampling is random, assignment (determination of intervention) is not, representing a quasi-experimental design. Sampled communities represent about five percent of the overall number of treatment sites where the tripartite model was implemented. Across the two evaluation phases, data collection reached just over 600 respondents. The evaluation’s primary limitation is that purposive stage two sampling means the selected respondents may not be representative of the larger universe; the large overall sample and treatment/comparison design mitigate this limitation. Because participation in the tripartite model is premised upon willingness to engage in a participatory community development process, there may also be a self-selection bias that influences who opts in to the program. Additional limitations include the potential for recall bias (particularly in retrospective study arms) and social desirability bias. Finally, the compressed evaluation timeline due to COVID-19 meant led to a lower density of neighboring projects; the risk is an underestimation of results in the baseline/endline study arm. viii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV FINDINGS The following exhibit summarizes differences between treatment and comparison communities for the tripartite model’s key outcomes; the exhibit includes both the intended and indirect outcomes. Domain Dimension Baseline Endline Retrospective 1 Retrospective 2 State-society and Intra￾society Relations 1. Trust in municipal government No difference Treatment higher Treatment higher Treatment higher 2. Municipal transparency No difference No difference Treatment higher Treatment higher 3. JAC-municipal relations Treatment higher Treatment higher Treatment higher 4. Public service provision No difference No difference No difference No difference 5. Peace accord understanding No difference Treatment higher No difference Treatment higher Community Capacity, Empowerment, and Ownership 6. Community project management No difference Treatment higher Treatment higher Treatment higher 7. Role of the JAC Treatment higher Treatment higher Treatment higher 8. Project funding No difference Treatment higher Treatment higher 9. Project ownership No difference Treatment higher Treatment higher 10. JAC Project Initiative No difference Treatment higher Treatment higher Community Resilience and Adaptability 11. Ability to manage insecurity No difference No difference No difference Treatment higher 12. Resilience No difference No difference No difference No difference 13. Civic engagement No difference No difference No difference No difference STATE-SOCIETY AND INTRA-SOCIETY RELATIONSHIPS Evidence suggests that the tripartite model improves community trust in local government, and in the baseline/endline study arm, prevented backsliding (based on an observed decline in trust in control communities). Overall, the level of trust ranged from low to medium; program and municipal staff perceived the level of trust to be higher than community members did, but community respondents in treatment sites consistently exhibited higher trust than respondents from comparison communities, an effect seen across all three study arms. Perceptions of municipal transparency ranged from low to moderate. Some evidence suggests that stronger doses of the tripartite model—as represented by the two retrospective study arms—improved community perceptions of municipal transparency; the effect was not seen in the baseline/endline study arm. Community members and leaders understood transparency in different ways; some considered transparency to be about communication, whereas for others it was related to fulfilling promises. There is mixed evidence that overall suggests the tripartite model leads to modest improvements in community-municipal relations. In all three study arms, there were responses in each treatment site that indicated improvement, even though each site also had many respondents who did not express improvement. This mixed evidence occurs within a broader context of an overall sense of abandonment in rural areas. Across all study arms, communities perceive public service provision as being poor; ix | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV this does not vary between treatment and control communities and consequently this evaluation finds that the tripartite model does not seem to alter this perception. Finally, evidence suggests that the tripartite model generally improves community members’ understanding of the peace accord, specifically the connection between local development projects and accord. With one exception, local governments generally feel under-resourced to implement the accords. COMMUNITY CAPACITY, EMPOWERMENT, AND OWNERSHIP Evaluation evidence is clearest in detecting the tripartite model’s effect on community capacity, empowerment, and ownership of local development projects. There is strong evidence that the tripartite model increases the capacity of JACs to manage more complex and ambitious projects. All respondent types conveyed the central role of the JAC in communities, particularly with regards to managing local development projects. This view is true in both treatment and control communities, however, respondents in control areas more frequently perceived there were not enough projects to implement. Across all regions, but particularly in Norte de Santander, municipal officials at endline reported that communities involved in the tripartite model are now well-versed in municipal processes. Municipal staff saw capacity building for JACs as a major strength of the model, as community groups are now able to solicit and manage their own projects, manage funds, and take care of contracting. Treatment and control communities across study arms consistently feel they have insufficient access to funds for all the projects required to meet community needs. However, communities also have multiple creative strategies for small local fundraisers they have used to collect community match resources; fundraising strategies appear somewhat more ambitious in treatment communities. There is strong evidence that the tripartite model results in high degrees of project ownership by communities. Evaluation evidence suggests the tripartite model leads to greater community initiative for local development projects, particularly vis-à-vis interactions with the local government. COMMUNITY RESILIENCE AND ADAPTABILITY Communities’ resilience and adaptability are not intended outcomes of the tripartite model. At the same time, due to the connection between these facets and the broader goal of supporting a durable peace, it was important to explore these facets to understand if the tripartite model could indirectly lead to results in this area. Evaluation evidence shows that violence has become normalized in the areas where Colombia Transforma operates. Both treatment and control areas have developed coping strategies to manage insecurity based on avoiding threats; the state is not seen as being able to provide security. In terms of resilience, communities have worked to address conflict among residents through communication and dialogue, yet in cases of external aggression communities flee and hide, returning once it is safe. The majority of community members in both treatment and control areas continue to equate civic/political engagement with elections (rather than other forms of civic or political engagement). While results suggest the tripartite model did not have a measurable effect on perceptions of levels of civic/political engagement, this could be due to the narrow interpretation of the term during interviews and focus groups. CROSS-CUTTING FINDINGS Respondents in treatment areas and municipal staff consistently praised Colombia Transforma, identifying several benefits of the program and its infrastructure projects. Community praise did not always extend to the tripartite model requirement to work with government, even though improvements observed by x | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV this evaluation seem to stem from the model. Program and municipal staff in Norte de Santander suggest convenios solidarios are an opportunity to increase the tripartite model’s impact in improving state￾society relations and communities’ capacity, empowerment, and ownership. Community respondents in three of the four treatment sites that had used a convenio solidario reported they were familiar with this tool (Retrospective 2 was the only study arm where this was applicable). These community leaders did not give much detail, although they seemed satisfied with results. CONTEXT Across all municipalities, most community members and leaders indicated security threats have not substantially changed in recent years but perceived that insecurity had a minimal impact on project implementation. This contrasted with views of program staff, which noted that the community played a fundamental enabling role in allowing the infrastructure projects to be carried out securely. COVID-19 affected project implementation schedules due to common disruptions such as supply chain issues and difficulties in meetings. Communities emphasized delays in project start up while program staff emphasized that despite challenges, overall projects mostly finished on time. Remote techniques are viable alternatives for monitoring project completion and municipalities used remote monitoring successfully. However, remote techniques are a poorer substitute for coaching, mentorship, and ongoing support to communities. Respondents across communities and municipal staff could not establish an impact of COVID-19 on security issues. However, there is some evidence that COVID-19 restrictions may have added challenges to already dire security conditions. CONCLUSIONS AND RECOMMENDATIONS Findings indicate that the tripartite model results in positive impacts on state-society and intra-society relations. These impacts are most apparent in trust and JAC-municipal relations, somewhat apparent in perceptions of transparency and understanding of the peace accord, but not apparent in public service provision. Results seem to be more positive in areas with stronger doses of the intervention, as indicated by a pattern of stronger results in retrospective treatment sites. Perceptions of trust, transparency, and JAC-municipal relations seem to be closely tied. Trust and transparency both seem closely tied to knowing the mayor and having repeated interactions with the municipality. The relationship with the municipality is also heavily dependent on the individual mayor, as compared to the institution, and that person’s ability to deliver on campaign promises. Repeated interactions do seem to build understanding among community members of the resource constraints municipalities face. Findings clearly suggest that the tripartite model’s strongest impacts are on community capacity, empowerment, and ownership of the local development projects implemented through Colombia Transforma. While communities themselves do not always articulate the change that stems from participation in the tripartite model, this study identifies these changes through comparison of treatment to control areas, which clearly captures the treatment communities’ heightened ambition in future project plans, fundraising, ability to manage projects, and lobbying of local government for support. Colombia Transforma hypothesizes that any link between the tripartite model and community resilience, adaptability, and civic engagement would be indirect, and findings confirm that this link is difficult to detect. The fact that data did not indicate an effect on community resilience or adaptability could reflect ongoing challenges—or even backsliding—in the security context, the possibility that the tripartite model or even communities are fundamentally unable to do much in the face of the security threats or the possibility that any indirect effects require more time or critical mass in order to be seen. xi | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Overall, it appears that areas where the program has been implemented for the longest, achieved critical mass, and used convenio solidarios show the highest results. The longer duration of Colombia Transforma presence in Retrospective 2 areas in Catatumbo coincides with additional communication, dialogue, and collaboration among stakeholders. However, attributing this effect to critical mass or use of a convenio solidario is complicated by other differences between the study arms, which offer competing explanations. First, the baseline/endline study arm had a compressed implementation timeframe for projects and treatment sites only experienced a single tripartite project; consequently, difference could also be due to dosage effects. Second, evidence also shows that while the Retrospective 2 treatment communities demonstrated higher absolute ratings on several dimensions, these dimensions were also higher in the Retrospective 2 control communities than those of other study arms; thus, the amount of relative difference between treatment and control is comparable to the other study arms. This is to say, while the Retrospective 2 treatment communities show a higher level of dimensions like trust, transparency, and so on, the amount of difference between treatment and control is consistent with the other study arms because the Retrospective 2 control communities show a higher level than the baseline/endline control communities on these same dimensions. Finally, the longer history of programming, critical mass, and use of a convenio solidario cannot explain why Retrospective 1 communities showed higher assessments than baseline/endline communities. These caveats suggest that there are likely a variety of factors behind these differences. Colombia Transforma operates in an extremely challenging context in which violence is normalized, communities are largely left to fend for themselves, and organized crime shapes many aspects of daily life. COVID-19 appeared to be a more disruptive contextual factor during the first year of the pandemic, but now appears to be receding; while it can affect timelines and supplies, communities and municipalities have mostly found practical workarounds. The depth of interactions communities and municipalities experience during project implementation is an important exception. RECOMMENDATIONS TO OTI COLOMBIA TRANSFORMA 3 · Build on the tripartite model’s powerful applied approach to capacity strengthening by providing communities with short accompanying tools that target specific functions such as administrative topics and managing security. When designing the tools, consider that virtual techniques are still challenging for community members. · Consider programming strategies to extend the successes observed in Norte de Santander by teaching communities about concrete municipal administration processes such as petitioning the local government. Focus on processes that are tangible to community development projects. · To support spread of the tripartite model, consider targeted communication products that describe how to initiate a project using the model in lay terms. The convenio solidario tool may also merit additional communication products. · For the third phase of Colombia Transforma, ensure expectations are reasonable given the scope of community infrastructure projects in the context of decades of war, mistrust between communities and municipal government, and municipal government resources. Evaluation evidence shows that the tripartite model can have measurable effects, but state-society relations are starting from a low baseline. The tripartite model can help start the process of improvement, but the process will require sustained engagement over many years to counteract long-standing conflict and the rural sense of abandonment. · Consider better documenting or specifying approaches to initiating contact and communication between actors at the start of a potential tripartite model project xii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · Recognize that building state-society relations between communities and municipalities is a long￾term process and build on evidence that suggests concrete mechanisms to leverage success from a first project for additional trust building. · To further support the relationship between communities and municipalities, look for more avenues that allow for building trust through collaboration. This may involve continuing to identify ways to reduce the prominence of OTI’s role so that communities see the municipality as the primary partner rather than Colombia Transforma. · The link between the small infrastructure projects themselves and the peace accord seems better understood than the link between the collaborative implementation model and the peace accord. Colombia Transforma and its partners should ensure this latter link is emphasized. · Continue to promote the convenio solidario as a tool within the broader tripartite model and other collaborative models. · As the convenio solidario mechanism builds momentum and Colombia Transforma generates more critical mass, there may be other innovations that can build on success of the base tripartite model, possibly through involving private enterprises or other charitable or civil society organizations. RECOMMENDATIONS TO USAID/COLOMBIA AND OTHER DEVELOPMENT ACTORS · Programs that seek to build trust in the state should take note of the tripartite model’s success. The model is a practical means to bring community leaders and local government together to collaborate on tangible projects that all parties can see. These first steps in building trust are central to longer-term development processes in Colombia’s most vulnerable communities. · Other small infrastructure projects can replicate the tripartite model to ensure that projects result in positive social-political developments in addition to the completion of the infrastructure itself. Successful replication of the model can also be leveraged to help increase community-level demand for future collaborative development projects with local government. · Given the apparent added value of critical mass, word-of-mouth diffusion of the tripartite model, repeated small projects, extended engagement, and convenios solidarios, USAID/Colombia should consider how micro-targeting projects geographically and longer-term development strategies can build on momentum in the specific geographies of CT 1 and CT 2. · Colombia Transforma had been working for almost 6.5 years by the time Phase 2 of this evaluation was collecting data. New programs will take time to build trust with community and municipal government actors, particularly in areas where there has traditionally been mistrust of the U.S. Government. New programs should work with established trusted actors when entering new areas and should moderate expectations for results. · The speed of project implementation was something communities greatly appreciated in partnership with Colombia Transforma. If USAID/Colombia is to conduct other infrastructure or community development programs in these areas, it should proactively consider how to design projects for efficiency while achieving compliance objectives. · OTI branding is less prominent than that of other USAID programs. In spite of this, it continues to take great effort to position the municipal government as the communities’ long-term partner. Other programs will need to take care to ensure they are supporting the role of local government and not eclipsing it. xiii | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV RECOMMENDATIONS TO OTHER ACTORS · The Government of Colombia at all levels should involve JACs and other community organizations in the execution of community projects, both to strengthen these organizations’ capacity and to strengthen state-society relationships. Existing mechanisms to do this, such as convenios solidarios, have proven successful, so this would be a proven strategy to utilize. · Government of Colombia programs can learn lessons in how to operationalize collaborative programs, interventions, planning, and other processes from the tripartite model. This collaboration can be time-consuming, but the investment has the ability to strengthen relations. · To the greatest extent possible, utilize cooperative models for implementing the peace accord’s rural development initiatives in order to deliver not only on the letter of the agreements but the broader spirit of building durable peace. · Convenios solidarios are a practical way to achieve more community development projects in the context of limited municipal staffing by devolving more of the project oversight to communities, while also building greater trust. Find ways to promulgate and diffuse this mechanism. · Municipal governments should return to in-person monitoring and visitation to communities as the COVID-19 pandemic continues to normalize. Remote monitoring techniques have ensured work continues, but do not carry the same value to communities that have felt abandoned. · Actors should seek to forge connections with other actors—government at all levels, civic and faith-based, private, and international development—and to communities in order to strengthen networks working in collaboration toward development goals. · Future evaluations should take note of the aspects of this evaluation’s design that made it possible to detect outcomes in a complex environment. Two specific factors were particularly important in this evaluation’s design: (1) use of matched control sites that were comparable to intervention sites on several key socio-political factors, and (2) robust use of open-ended questions. · The evaluation team recommends the following for any future evaluation of the Colombia Transforma program: (1) for results that will be measured quantitatively, community leaders will be the most efficient level of measurement to assess and detect results, (2) a full probability sample could be achieved among community leaders, as a discrete sampling frame could be compiled for probability-based sampling at the second stage, while trying to achieve a full probability sample among community members would likely require a challenging household listing exercise prior to data collection, (3) by extension, the evaluation team recommends that the value of data provided by community members is strongest with qualitative methods, (4) a sequential mixed-methods design, whereby quantitative data are collected first and followed by a separate second round of qualitative data collection, could be a powerful design that yields even richer qualitative data. 1 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV INTRODUCTION For more than half a century, Colombia has experienced intractable armed conflict against insurgent and criminal groups, resulting in 220,000 deaths and at least 7 million registered victims.1 The conflict has been driven by state weakness and structural inequities that brought about “two Colombias”—a modern, urban one and a poor, neglected, rural one, where most of the conflict has taken place.2 After four years of negotiations, on November 30, 2016 the Colombian government ratified a final peace accord with the largest guerrilla group, the Revolutionary Armed Forces of Colombia (FARC-EP). FARC-EP has since demobilized, while the accord commits the government to wide-ranging reforms related to rural development, inclusive political participation, transitional justice, and reduction of illicit crops. Full implementation will take more than a decade; the initial years may be the most critical. The long-term success of the accord will depend on the extent to which Colombians—particularly those who have suffered most from the conflict—benefit from peace and buy into the process. In July 2015, the United States Agency for International Development (USAID) Office of Transition Initiatives (OTI) awarded a task order to MSI to implement the Colombia Transforma program. This first iteration of Colombia Transforma, known as CT 1, ran from July 2015 to July 2019. CT 1 aimed to support the critical first 36 months of peace accord implementation by assisting government entities at all levels to deliver quickly on the accord’s promises, particularly its first chapter focused on rural reform and development. In support of this goal, CT 1 implemented almost 500 small infrastructure projects, along with other interventions. Through trial and error, CT 1 developed a methodology for implementing these infrastructure projects, known as the tripartite model. In this model, OTI, the community, and municipality each contribute resources and share responsibility for the project. An external performance evaluation indicated that this model yielded positive programmatic and operational results.3 The evaluation’s scope and design, however, limited the depth of evidence specific to the tripartite model (the evaluation covered the entire Colombia Transforma program, including its other interventions). The successes of CT 1 led to the renewal of Colombia Transforma for a second iteration, referred to as CT 2, which ran from July 2019 to February 2022. The small infrastructure project component of CT 2 explicitly built on lessons learned from CT 1 by systematically using the tripartite model as the programming approach. EVALUATION PURPOSE AND QUESTIONS OTI seeks rigorous independent evidence on the tripartite model’s impacts on state-society and intra￾societal relations, and communities’ capacity to manage local development. OTI is also interested to learn how the tripartite model’s first-level impacts may in turn contribute to greater community resilience as an indirect result, how and under what circumstances the tripartite model is most effective, and how contextual factors influence results. Specifically, the Colombia Transforma team is interested in seeing whether the tripartite model fostered: · State-society and intra-society relationships · Community and community leaders’ capacity, empowerment, and ownership 1 Centro Nacional de Memoria Histórica. 2013. ¡Basta Ya! Colombia: Memorias de Guerra y Dignidad. Bogotá. 2 Romero, Simon. 2008. “Two Colombias, at War and at Peace.” The New York Times, September 7, 2009. 3 Colombia Transforma Program Mid-term Performance Evaluation, Social Impact. May 2018. 2 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · Community resilience and adaptability (as an indirect outcome) This evaluation achieves this purpose by addressing five key questions, which were developed through a participatory process facilitated by the evaluation team with OTI Colombia during the initial design of the evaluation. As part of the validation of the design for the endline evaluation phase, these questions were refined based on learning during the baseline. The questions are presented here in their final form. Evaluation question one: State-society and intra-society relationships 1. To what extent and through what mechanisms does the tripartite model create or strengthen relationships between local governments and community members? 1.1. To what extent and through what mechanisms does the tripartite model facilitate local government transparency? 1.2. To what extent does the tripartite model increase community trust in the local government’s ability to respond to community priorities? What are the key pathways to change? 1.3. To what extent does the tripartite model create avenues for citizen engagement with the local government? Are these avenues perceived as effective? 1.4. To what extent does participation in the tripartite model generate support for the peace accord among municipal staff, community leaders, and community members and enhance their understanding of the connection between the accord and Colombia Transforma? Evaluation questions two and three: Community capacity, empowerment, and ownership 2. To what extent does participation in the tripartite model empower community members and leaders? 2.1. To what extent and through what mechanisms does participation in the tripartite model increase ownership of infrastructure projects (e.g., indicated by willingness of communities to manage and maintain the projects and protect them from armed groups)? 2.2. To what extent does participation in the tripartite model increase community members’ and leaders’ willingness to engage in local development and peacebuilding to improve community well-being? What are the key pathways to change? Due to the COVID-19 outbreak, how has this willingness and/or behaviors changed? 3. To what extent does participation in the tripartite model increase community members’ and leaders’ capacity to manage local development processes? 3.1. In what ways and to what extent does the tripartite model strengthen the organizational capacity of community members, groups (specifically juntas de acción comunal (community action boards, or JACs) and consejos comunitarios), and leaders to manage, sustain, or formalize local development and peace-building processes (including their ability to articulate their needs and advocate for and manage development)? Evaluation question four: Community Resilience and Adaptability 4. To what extent does participation in the tripartite model increase communities’ resilience and adaptability as an indirect program outcome? 4.1. To what extent and in what ways does participation in the tripartite model increase community members’ and leaders’ capacity to protect their communities and projects from threats such as armed groups (violence prevention) and COVID-19? 3 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV 4.2. To what extent and in what ways does the participation in the tripartite model affect communities’ resilience (adaptation to and recovery from) vis-à-vis negative events such as external violence and COVID-19? What are the key pathways to change? Evaluation question five: Context 5. How has the context affected the implementation of the tripartite model? 5.1. How and to what extent has COVID-19 affected the willingness and capacity of municipal staff, community members, and community leaders to participate in activities related to the tripartite model? 5.2. How has COVID-19 affected the capacity and willingness of community members and leaders to engage in local development and peace-building processes? To address these questions, OTI contracted EnCompass LLC to conduct a two-phase evaluation of the tripartite model used in CT 1 and CT 2. Evaluation results will inform adjustments to the third phase of Colombia Transforma and other USAID/Colombia projects; the results could also support strategy in other OTI country programs engaged in community development projects. The primary evaluation audience is OTI/Colombia and USAID/Colombia. Secondary audiences include other OTI country programs, the Government of Colombia and its agencies, and other development actors in Colombia. PROGRAM CONTEXT Colombia Transforma worked in areas that have experienced some of the most severe conflict conditions in the country. While these areas were initially spared the COVID-19 infection rates seen in urban areas, later variant-driven waves were punishing and economic effects were severe in these areas, which are already among the most impoverished in the county. Exhibit 1 presents security and COVID-19 indicators for the nine municipalities sampled for this evaluation. Although the evaluation team was not able to ask in interviews about the presence or activities of particular armed actors because of security concerns, 2019 survey data from USAID’s Regional Governance Activity endline evaluation provide information on five of the municipalities sampled for this evaluation (Caucasia, Tarazá, Teorama, Tibú, and Tumaco). These data indicate that armed groups were able to move freely in all five municipalities. Teorama and Tibú show the most contestation, with average reports of three different armed actors reportedly able to move freely at the end of 2019, followed by Tumaco with an average of two armed groups able to move freely. OTI regional staff confirmed the ongoing presence of armed groups in all municipalities as of 2021. According to the information available to OTI staff, multiple armed groups dispute territorial control in all municipalities, except Caucasia and Tarazá, where a single armed group was able to control most territory after years of dispute. In fact, in 2018, the International Committee of the Red Cross (ICRC) established there are at least five non-international armed conflicts occurring in Colombia in the municipalities sampled for this evaluation; ICRC reiterated this finding in 2019, 2020, and 2021.4 Furthermore, Tarazá and Tibú exhibit extraordinarily high homicide rates, at over 150 killings per 100,000 residents per year; most municipalities sampled have experienced an increase in homicide rates since 2015. Finally, Tarazá, Tibú, and Tumaco have all seen high numbers of killings of social leaders. This evidence suggests that five years after FARC-EP handed over its arms, security in the regions covered in this evaluation has gotten worse. 4 ICRC, Armed conflict in Colombia: a pain that doesn’t go away. 4 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Exhibit 1: Context indicators by evaluation sample municipalities Indicator Caucasia Tarazá Tibú Teorama Sardinata El Carmen Tumaco Valle del Guamuez Orito COVID-19 [1] (deaths/cases) 279/6,348 29/1,132 86/1,897 7/163 38/807 12/122 254/5,687 79/1,506 94/2,106 Social leader killings 2015– 2021 [2] 4 21 12 1 3 2 34 4 6 Armed group presence 2021[3] Control of one armed group Control of one armed group Dispute between armed groups Dispute between armed groups Dispute between armed groups Dispute between armed groups Dispute between armed groups Dispute between armed groups Dispute between armed groups Homicides 2015 [4] (cases) 37 22 23 8 11 1 130 16 6 Homicides 2021 [4] (cases) 60 64 76 8 10 4 173 22 24 Population [5] (total/rural) 126,161/ 20,673 48,926/ 16,704 37,455/ 22,965 23,864/ 20,904 22,570/ 13,248 12,933/ 10,746 221,469/ 94,687 54,819/ 33,800 59,283/ 31,448 [1] https://www.ins.gov.co/Noticias/Paginas/coronavirus-filtro.aspx. [2] http://indepaz.org.co/lideres-sociales-y-defensores-de-derechos-humanos-asesinados-en-2021 and annual reports of Somos Defensores https://somosdefensores.org/informe-anual-1 [3] Report from OTI regional staff [4] https://www.policia.gov.co/grupo-informacion-criminalidad/estadistica-delictiva [5] http://www.dane.gov.co/files/investigaciones/poblacion/proyepobla06_20/Municipal_area_1985-2020.xls Beyond the challenges posed by the public health and security context in the localities where Colombia Transforma works, the national context has also presented successes and challenges. The breakthrough in negotiations that led to the initial peace accord between FARC-EP and the Colombian government in September 2016 opened a window of opportunity after 50 years of conflict. This breakthrough was quickly followed by a setback with the accord’s narrow defeat in a national plebiscite in October. A quick renegotiation of key provisions allowed the government to get the deal ratified by Congress in November, but as observers have written, “the plebiscite’s failure unsettled the deal.” 5 The peace accord became a defining element in the 2018 presidential election, with the victor, Iván Duque, campaigning against the accord. In 2019, President Duque’s administration redesigned government institutions tasked with implementing the peace accord; the Special Presidential Advisor for Peace became the Special Presidential Advisor for Stabilization and Consolidation, and the Ministry for Post￾conflict that had been created as a special presidential office under the former administration was eliminated. Despite the adoption of regional development plans, in a context in which the rural reforms were always going to be difficult, government delays in these reforms have produced cycles of expectation and disappointment. Critiques related to the accord’s rural development chapter implementation have included a government preference for choosing its own priorities over those of communities. Critiques to implementation of the chapters on illicit crops and political participation have included failure or slow payment to farmers who uprooted coca crops and delays in implementing political reforms. 6 Meanwhile, 5 A Fight by Other Means: Keeping the Peace with Colombia’s FARC. International Crisis Group, November 2021: 2. 6 Ibid, 16-17. 5 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV the 2019 announcement that at least two FARC-EP negotiators and some former commanders had returned to arms left citizens increasingly skeptical of the accord.7 Exhibit 2: Key events in Colombia’s peace accord implementation process that relate to Colombia Transforma COLOMBIA TRANSFORMA AND THE TRIPARTITE MODEL Colombia Transforma’s main objectives are to: · Increase government entities’ commitment to peace accord implementation (at all levels of government) · Support bottom-up accord implementation by ensuring that communities and local stakeholders are actively involved in and sharing responsibility for activity design and implementation Through small infrastructure projects and other interventions, Colombia Transforma seeks to support the implementation of the peace accord, including the Programas de Desarrollo con Enfoque Territorial (PDETs, a Government of Colombia program aimed at addressing some of the inequities between urban and rural Colombia through investments in rural development in the 170 municipalities most affected by the conflict). The program seeks to increase state presence in target areas, foster collaboration between stakeholders without prior relationships, and serve as a model for how the Government of Colombia can carry out rapid assistance. On the municipal side, it seeks to encourage municipal governments to more closely engage with communities. For communities, the program seeks to reinvigorate community spirit, educate residents about their roles and rights under the peace accord, and develop leadership, 7 Fundación Ideas para la Paz, La segunda marquetalia. Disidentes, rearmados y un futuro incierto. 6 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV problem-solving, technical, and administrative skills. According to the theory of change, effective implementation of small infrastructure projects will promote a sense of community collaboration and co-responsibility for accord implementation alongside government authorities. Although the central Government of Colombia has well-developed institutions and capable bureaucrats working in Bogotá, its highly centralized and bureaucratic processes hamper provision of public services and conditions for the rule of law in the most conflict-affected areas. Most of these areas are in poor municipalities that face challenges in independently raising public funds to improve service delivery. In this context, Colombia Transforma supports small-scale infrastructure projects, many of which are implemented using the tripartite model. In the model, the municipal government, community members, and Colombia Transforma collaborate in the planning, design, and management of the projects, reflecting a shared-responsibility relationship. Each partner contributes necessary resources: the community volunteers manual labor, the municipal government provides skilled labor, and Colombia Transforma provides in-kind materials and equipment. Typical projects include rehabilitation of community centers, schools, sports facilities, and tertiary roads. CT 1 operated in the Departments of Arauca and Putumayo and the Catatumbo region of Norte de Santander. USAID/Colombia and OTI prioritized these areas because of their long histories of armed conflict, presence of armed groups, high presence of illicit crops (to a lesser degree in Arauca), lack of historical USAID program coverage, and status as complex, insecure border zones. CT 1 helped expand the geographic footprint of USAID/Colombia into these areas that would be critical for peace accord implementation. USAID designed the CT 2 program to consolidate programming across key PDET geographies by prioritizing the Bajo Cauca, Nariño Pacífico, and Catatumbo regions. Across the CT 1 and CT 2 regions, the program has and has operated in virtually all municipalities. Exhibit 3: Number of Colombia Transforma infrastructure projects by phase Project Type CT 1 CT 2 Total Small infrastructure projects 447 284 731 Projects that used the tripartite model 153 of 447 125 of 284 278 of 731 Tripartite model projects that used a convenio solidario n/a 23 of 125 23 of 278 As is typical for an OTI program, small grant activities are designed and developed on a rolling basis. Each regional Colombia Transforma team prioritizes micro-targeted areas of municipalities for interventions based on a variety of factors, including strategic corridors where communities have been greatly affected by conflict, illegal actors remain present, and local community and civilian actors are interested in partnering with the program and other government stakeholders. Colombia Transforma used OTI’s Act, Assess, Adapt model throughout implementation. This model includes daily context analysis, learning from partners, maintaining the political objective, and strong teamwork to effect change. Colombia Transforma uses a “one team” approach to implementation, involving the participation of the Senior Management Team (including both OTI and implementer staff), the management committee, and the procurement, grants, finance, security, administration, programs, and monitoring and evaluation areas in collaboration with national and territorial partners and allies. 7 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV EVALUATION METHODOLOGY DESIGN The evaluation consists of two phases (Phase 1 and Phase 2) and two design components: a pre/post component and a retrospective component. Both components feature comparisons between treatment and comparison sites, which helps isolate the effects of Colombia Transforma’s tripartite model. The retrospective component consists of two parts, one at each evaluation phase. The two retrospective phases have separate samples and consequently are separate study arms, referred to as Retrospective 1 (conducted at Phase 1) and Retrospective 2 (conducted at Phase 2). Exhibit 4 depicts this visually. Retrospective 1 provides evidence from CT 1 programming; this evidence assists the evaluation in understanding potential differences in impact between CT 1 and CT 2 (for example, due to changes in the context from the pandemic or otherwise, due to the compressed implementation timeline for CT 2, or other factors). Retrospective 2 focuses on CT 2 veredas that have utilized a convenio solidario or took place in areas where the program believes it has achieved critical mass, thereby generating evidence regarding these evolutions in the tripartite model; these veredas are all located in Catatumbo, Norte de Santander, the only department with programming since 2015. The retrospective study arms represent an ex-post, treatment/comparison design. The pre/post study arm generates evidence related to CT 2, with two rounds of data collection consisting of baseline and endline. This represents a pre/post, treatment/control design. The evaluation questions are the same across the study arms. Exhibit 4: Evaluation design by the three study arms Data for both evaluation phases are primarily qualitative. Qualitative methods provide a narrative description of how the tripartite model has been implemented across projects representing different geographical areas and how the program approach, strategy, and theory of change may be affected by contextual factors such as the COVID-19 pandemic. The evaluation phases are further described below. 8 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV PHASE 1: BASELINE AND RETROSPECTIVE I Phase 1 data collection occurred December 2020–March 2021 and included the baseline and Retrospective 1 components. The baseline collected data in areas where CT 2 was expected to be implemented (treatment areas) and in a set of similar veredas that would not receive interventions (comparison veredas). Data collection took place about a year and a half into CT 2 implementation, but before implementation had started in the treatment veredas. In contrast, for Retrospective 1 this timing meant projects had been concluded approximately two years prior in treatment sites; comparison sites had received no intervention at the time of data collection. In Phase 1, both study arms featured an in￾depth focus on how the COVID-19 context was affecting implementation. Phase 1 rendered relevant learning for the evaluation’s implementation, which was incorporated into the design during Phase 2. PHASE 2: ENDLINE AND RETROSPECTIVE 2 Phase 2 data collection occurred November–December 2021, at the end of CT 2, after projects concluded in the treatment sample sites. The sample sites for the endline component were the same as baseline. The sample sites for Retrospective 2 differ, in order to maximize the total evaluation sample size and to specifically focus on CT 2 project sites that utilized a convenio solidario or reflected critical mass. Phase 2 respondent groups were consistent with evaluation Phase 1 across all study arms. SAMPLING Sampling reflects a two-stage design. At the first sampling stage, treatment veredas are selected randomly from the sampling frame consisting of the universe of relevant potential intervention sites. Comparison sites are then selected based on matching techniques to construct a counterfactual. While first-stage sampling is random, assignment (determination of intervention) is not, representing a quasi-experimental design. At the second stage, respondents are selected purposively, based on pre-established criteria and respondent types. The two stages are described in detail below. SITE SELECTION (STAGE 1 SAMPLING) For baseline/endline, the evaluation team collected data in 12 sample veredas (six CT 2 treatment and six control) across five municipalities. The OTI program team refrained from implementing programming in the comparison areas over the course of CT 2 to preserve these veredas as control sites. Retrospective 1 collected data in eight sample veredas (four CT 1 treatment and four comparison) across four municipalities, including two municipalities that also had a separate set of baseline/endline sites (Teorama and Tibú). Retrospective 2 collected data in eight sample veredas (four CT 2 treatment and four comparison) across four municipalities, including two municipalities that also included baseline/endline sites (Teorama and Tibú). The sample sites and municipalities are specified in Exhibit 14 of Annex B: Detailed Methodology. Exhibit 5: Data collection sites by study arm 9 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Sampled veredas represent about five percent of the overall number of treatment sites where the tripartite model was implemented. Across study arms, all matched pairs of treatment and comparison veredas come from the same municipality. Because the tripartite model emphasizes interactions between communities and municipal offices, this pairing holds constant a potential source of variation. RESPONDENT SELECTION AND RECRUITMENT (STAGE 2 SAMPLING) The evaluation used purposeful sampling across both phases to select respondents within the evaluation sites. Purposeful sampling is a form of non-probability sampling where the evaluation team identifies individuals based on specific selection criteria. At both phases, the evaluation team identified the main community leaders involved in public works decision-making for each vereda. The team then sampled a small number of these leaders as well as other residents. Relevant staff within the municipal governments (alcaldías) were identified based on function. Focus groups sought to convene a selection of community members representative of their vereda. DATA COLLECTION METHODS The evaluation team used participatory and appreciative approaches to collect data through the following complementary methods. Interview and focus group guides sought to maintain consistency across evaluation phases 1 and 2 to allow for comparison of data, while incorporating adjustments based on lessons learned and other small improvements to flow and accuracy. Retrospective 2 guides include a focus on the convenios solidarios specific to that phase. · Document review: The team reviewed OTI activity database documents, which include activity notes and final evaluation reports, as well as a midterm performance evaluation of CT 1, relevant program documents, and reports from third parties. Review of the OTI database allowed the team to draft a short summary for each treatment vereda describing key aspects of the intervention; these summaries enhanced data collection by informing enumerators’ probing. · Key Informant Interviews (KIIs): In-depth, semi-structured interviews were conducted face￾to-face and—for program and some municipal staff—remotely. KIIs used a mix of open- and closed-ended questions tailored for each respondent group. KII guides facilitated triangulation by asking similar questions to different respondent groups. The closed-ended, survey-style questions additionally facilitated triangulation and systematic comparison across study arms. · Focus Group Discussions (FGDs): Moderated, semi-structured group discussions were facilitated with six to ten community members to explore key topics in depth. FGD guides facilitated interaction across respondents in order to generate rich discussion. The following respondent groups provided data through these KIIs and FGDs: · Program staff (KII), including OTI and MSI (implementing partner) staff · Municipal government officials (KII), primarily secretaries for planning, secretaries of government, and PDET focal points; some interviews included other officials for responsibility for public services or public works projects and select mayors · Community Leaders (KII), including local government officials, local civil society organizations, community-based organizations, and community (JAC) leaders 10 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · Community members (KII and FGD) data collection sought to include a broad sample of those who have participated in CT activities and those who have not, as well as achieving broad representation by sex, age, socioeconomic status, and members of vulnerable groups Exhibit 6 presents the final achieved sample; the municipal and program staff are presented by evaluation phase rather than study arm, as these respondents covered municipalities with sample veredas in both study arms in each evaluation phase.8 Exhibit 6: Final achieved sample Phase 1 Phase 2 Baseline Retrospective 1 Endline Retrospective 2 Total Community Leaders 34 17 35 24 110 Community Members: KIIs 43 24 48 32 147 Community Members: FGDs (participants) 9 (94) 1 (10) 12 (130) 8 (73) 30 (307) Municipal Staff 12 Included in baseline 9 Included in endline 21 ART Staff 4 n/a n/a n/a 4 Program Staff 9 Included in baseline 8 Included in endline 17 Total: 111 (196 respondents) 41 (51 respondents) 112 (230 respondents) 64 (129 respondents) 329 (606 respondents) ANALYSIS Interviews and focus groups were recorded and transcribed to ensure accuracy of data; following transcript cleaning and verification, recordings were destroyed. Answers to closed-ended questions were extracted and compiled in an Excel database. For qualitative data, the evaluation team used Dedoose software, a collaborative qualitative coding platform that enables cross-checking and assessment of inter-coder reliability. The team consolidated data from both evaluation phases and all study arms into a single encrypted Dedoose (qualitative) and Excel (quantitative) database to facilitate comparison across phases and treatment/control groups. Qualitative analysis followed a two-step procedure that included both inductive and deductive approaches. In the first step, the team used a codebook based on the evaluation questions that specified inclusion and exclusion criteria for each code. The codebook used in the final evaluation was based on the Phase 1 codebook, with refinements to strengthen the process. To pilot the codebook, all analysts coded the same sample of transcripts; the team then met to discuss inter-coder reliability and make codebook refinements. The final codebook was then applied to all transcripts during the first stage of coding, reflecting a deductive approach. In the second stage, the evaluation team analyzed each code to generate emergent themes through an inductive process; inductive analysis avoids presupposing hypotheses regarding respondents’ experiences and, thus, allows unexpected results to surface based on the most salient points. Emergent themes represented broad patterns of attitudes and perceptions, and 8 All program and municipal staff KIIs in Phase 2 were conducted remotely. Community-based KIIs and FGDs were all conducted in-person. Some individuals were interviewed at both Phase 1 and Phase 2; thus, the total 606 respondents includes some repeat interviewees. Respondents 11 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV included disaggregation to capture variation by geography and respondent types. Quantitative data were analyzed in Excel using descriptive statistics, in line with the sampling design. Based on this analysis, the evaluation team came together for a data analysis, interpretation, and synthesis workshop, an internal process through which the team developed preliminary findings by comparing the results from each individual study arm. The comparison allowed findings to take into account differences in program intensity by the three study arms. This collaborative process allowed for collective sense-making, checking potential biases, and consistent comparison across each study arm’s treatment versus control groups (as well as over time, for the baseline/endline study arm). The evaluation team validated these preliminary findings, revised conclusions, and co-created recommendations with OTI during a participatory validation workshop on February 2, 2021 that included approximately 40 participants. This validation workshop helped ensure findings and conclusions are context-relevant, evaluation data are accurately interpreted, and recommendations are feasible. Similar workshop processes (internal and with OTI) were used at the conclusion of evaluation Phase 1. EVALUATION STRENGTHS AND LIMITATIONS This evaluation used collaborative and participatory approaches throughout to ensure the final results met the needs of the program’s key stakeholders. Engagement with program staff during evaluation design ensured a rigorous approach to sampling and targeted evaluation questions. Coordination during data collection helped ensure participants’ security and that adjustments to the sample and tools aligned with the evaluation’s purpose. During reporting, the evaluation team intentionally structured internal data synthesis processes to check potential biases. Collaboration with the Colombia Transforma team, meanwhile, helped validate findings and co-create conclusions and recommendations while the evaluation team ensured that results stayed true to the data; this process aided accurate data interpretation, contextual awareness, and relevancy to stakeholders. The evaluation benefited from deep local expertise of data collectors based in sampled regions and a Colombian expert in the peace accord. Sampled veredas represent just over five percent of tripartite model implementation sites and data collection reached over 600 respondents, relatively robust for an evaluation based on qualitative methodologies. Matched comparison veredas and the use of a baseline further bolsters the design and enables attribution in cases where clear patterns emerge across treatment and comparison sites. Data collection instruments contain diverse types of questions, including some closed-ended response formats that yield quantitative indicators for triangulation with qualitative analysis. Finally, the evaluation reached diverse participants to capture distinct viewpoints, including community members, community leaders, mayoral staff, and program staff. The evaluation takes a broad approach to the topics covered and phrased questions to elicit insights not just about indicators but also about social processes. This evaluation is limited by the following factors, which are discussed in further detail in Annex B: Detailed Methodology · Selection bias due to purposive respondent sampling means the selected respondents may not be representative of the larger universe, thus results cannot be generalized to all tripartite model projects; the large overall sample and treatment/comparison design mitigate this limitation. Because participation in the tripartite model is premised upon mayors’ and communities’ willingness to engage in a participatory community development process, there may also be a self-selection bias that influences who opts in to the program. Selection of control veredas from the same municipality mitigates this potential source of selection bias. 12 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · Recall bias may have inhibited respondents’ ability to remember previous events; several techniques helped mitigate this risk, but could not remove it. · Response bias and social desirability bias in particular could have led respondents to feel pressure to give answers that are socially acceptable. Participatory approaches help mitigate this, and Phase 1 results showed that most respondents were candid and did not hold back. · Limited access to some stakeholders: ultimately the evaluation team was able to reach the majority of targeted respondents by extending the timeline for data collection and offering flexible accommodations for the setting (e.g., remote and options outside the vereda); qualitative appeared to reach data saturation, indicating that additional data collection may not have yielded additional nuance. · Evaluation challenges related to the COVID-19 pandemic: These challenges were primarily logistical. However, the delay of baseline data collection due to the evaluation’s redesign meant that though baseline data were collected prior to CT 2 implementation in baseline/endline veredas, the interval between baseline and endline data collection was compressed. The compressed timeline also meant that CT 2 only completed one project in each treatment vereda between the baseline and endline as opposed to grouping projects within neighboring veredas, as is standard. The risk is an underestimation of results in the baseline/endline study arm. · Changed programming context due to COVID-19: The pandemic and associated public health policy responses have greatly hobbled development processes. Person-to-person communication is fundamental for executing projects using the tripartite model. At a minimum, the switch to remote management methods by municipalities decreased the amount of interaction and collaboration with communities. The pandemic conditions therefore required several adaptations to implementation procedures. This departure from the preferred tripartite model calls for a moderation of program expectations. The Retrospective 1 study arm (focused on CT 1) helps mitigate this challenge, but the overall risk remains of an underestimation of the tripartite model’s full potential. On the other hand, the pandemic provides an opportunity to observe how the model withstands the tremendous pressures of a once-in-a-century pandemic. 13 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV FINDINGS STATE-SOCIETY AND INTRA-STATE RELATIONS FINDING 1: Evidence suggests that the tripartite model improves community trust in local government. Overall, the level of trust ranged from low to medium; program and municipal staff perceived the level of trust to be higher than community members did. Trust in municipal government Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher ▲ Treatment higher ▲ Treatment higher With trust in local government higher in treatment than control areas in all study arms, evidence suggests that the tripartite model results in some improvement in community trust in local government. Evaluation data show a variation in perceptions of trust in local governments across stakeholders, with program and municipal staff crediting the tripartite model with greater improvements in trust than indicated by community members. While trust in local government was higher in treatment than control areas in all study arms, this difference was not large enough to qualify as trusting, even in treatment areas; overall, community leaders and members remain distrustful of local government. Respondents did not state directly that perceptions of municipal transparency had changed; rather this finding is observed through comparison of responses between treatment and control areas. Among community respondents from endline treatment veredas, a minority indicated that trust in the local government was better than two years ago; a small subset of this minority noted Colombia Transforma as a specific reason for the improvement. Most felt that trust had not changed. In contrast, in the endline control areas, perception of trust had deteriorated further since baseline. Thus, the evidence from the baseline/endline study arms suggest that while the tripartite model did not appear to substantially improve trust in local government, the model prevented community trust from deteriorating further. Similarly, both retrospective study arms suggested similar slight positive effects of the tripartite model: Trust in local government was rated by respondents as medium-high in treatment veredas for both Retrospective 1 and Retrospective 2, higher than in control veredas. At the same time, trust was higher in the control veredas for the two retrospective study arms than it was in even the treatment veredas for baseline/endline; evaluation data did not offer an explanation for why this might be. In addition, select Retrospective 2 treatment community leaders in each municipality credited Colombia Transforma with lessons learned around collaboration and communication with local government. Respondent 1: We need to acknowledge one important thing that [the municipality] has done, they are things that we as community leaders and members need to value what they do, however small… The project was implemented by CT, ART, and the municipality. The community contributed the manual labor. Trust increased because the project was completed. 14 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Respondent 2: I think during those two years of the project the mayor won our trust, which is what we want—a mayor who will help us. With his help and others, we can do more things. – FGD Endline treatment community, Norte de Santander While community leaders and members generally reported continuing to feel abandoned, some respondents in Retrospective 2 treatment veredas reported improvements in local government presence in veredas and acknowledged that the presence of Colombia Transforma helped build confidence with the local government. In Norte de Santander, program staff noted that the small projects have been small steps to show communities that they can participate directly in their development and that the municipality has their interests at heart, thus changing community mindsets that municipalities are just out to rob their constituents. Community trust in the local government seems to be closely tied to personal relationships, visits by government officials and other repeated interactions, and mayors’ ability to deliver on campaign promises. These associations point to some of the barriers that may be impeding further improvements in perceptions of trust. Program staff, for example, reported that while the project has improved trust in local governments, much of that trust also hinges on the dedication and quality of specific mayors and the extent to which they are perceived as being closely tied with their constituents. Meanwhile, the community members in endline treatment veredas who reported the lowest trust in the local government also indicated that local authorities make promises during electoral campaigns and then do not follow through on their promises. These respondents also noted that local government authorities rarely or never visit their veredas after the electoral campaign is over. Perceptions of clientelism are another barrier, with several community members indicating that the mayor provides benefits only to their inner circle. When elections come, they list so many things that they are going to promise you, but they don’t help at all. They just want your vote. They come in here in their cars and when elections are over you can forget about them coming back. We ask for help and by God, they won’t give it. – Community Leader Endline Treatment, Nariño Across all municipalities, however, municipal staff considered that communities had moderate to significant levels of trust in local governments, with staff also acknowledging that more work is left to be done. In Norte de Santander, one municipal staff indicated communities are now more trusting of the local government because of their participation in convenios solidarios where their feedback on needs is being taken into consideration. Program staff also generally agreed that community trust in municipal governments is improving as a result of the project, and there is more work to be done. We won’t lie and say that trust is now at maximum levels and that communities now believe in the local government… there’s still a long road and much work to do, but we are on a good path. When we arrived, people said, “I won’t work with that mayor.” Now we have seen that the community thanks the local government, they approach them more easily and speak with them with more confidence. – Program staff FINDING 2: Perceptions of municipal transparency ranged from low to moderate.There is some limited evidence suggesting that the full tripartite model can improve community perceptions of municipal transparency, with both retrospective study arms showing some effects. Municipal transparency 15 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▬ No difference ▲ Treatment higher ▲ Treatment higher When asked about local government transparency, community members and leaders understood transparency in different ways; some considered transparency to be about communication, whereas for others it was related to fulfilling promises. Where community members felt there are government efforts to communicate, transparency was perceived to be higher. - How transparent is the mayor? - He’s been very good to our community. - Why do you think he’s been good, and transparent? - Because we have asked him for things, and he helps where he can—when he’s made promises, he’s delivered. - How does the local government communicate with your community? - They communicate well, they come to give us information during the project. They are very mindful. – Community leader, Retrospective 2 treatment, Norte de Santander As seen in Exhibit 7, the tripartite model’s effects on perceptions of transparency were largely limited to the retrospective areas that experienced full implementation of the model; effects were not seen in the baseline/endline study arm that featured the abbreviated tripartite model. In three of four Retrospective 2 treatment communities, community leaders and members reported more positive or mixed perceptions of municipal transparency than in their control community counterparts. Perceptions of municipal transparency were higher among community leaders than community members. Similarly, respondents from Retrospective 1 treatment communities indicated medium-high levels of perceptions of municipal transparency, which was higher than the matched control sites. Local governments who maintain communication with communities were seen as being more transparent, with examples of communication including phone, WhatsApp, meetings, or communication through mechanisms such as the concejales and the association of juntas. 16 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Exhibit 7: Perceptions of municipal transparency by study arm In contrast, in endline treatment areas, there was limited evidence of a positive view of local government transparency. Several community members indicated they did not know enough about the mayor or transparency to have an opinion. I wouldn’t say they are transparent. When you tell lies to one person, you are lying to the whole community, to the Junta, so they are not transparent, because when one is transparent you do what you say you’re going to do and you are focused on what’s going to be done. – Community leader, Endline treatment, Antioquia Municipal and program staff identified several actions and techniques they have used to improve communication, some of which have been linked to COVID-19 restrictions. Program staff noted the importance of social media, WhatsApp groups, and webpages as a means of promoting government transparency remained consistent across baseline and endline. There appears to be a change in how local government officials communicated with communities from baseline to endline. Local government officials at endline reported utilization of print media, radio, television, social media, and official websites as a means of informing citizens and soliciting community feedback, whereas government staff at baseline were less likely to use social media and more prone to mention suggestion boxes. It is not clear if this is due to Colombia Transforma or to the COVID-19 pandemic. Despite these changes, these mechanisms of communication differed from those most frequently cited by vereda-level respondents above, which emphasized more personal means of communication. 17 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV FINDING 3: There is some evidence that the tripartite model improves community￾municipal relations, though this occurs within a broader context of an overall sense of abandonment in rural areas. JAC-municipal relations and municipal outreach to communities Baseline Endline Retrospective 1 Retrospective 2 n/a* ▲ Treatment higher ▲ Treatment higher ▲ Treatment higher *As seen in Annex A, this domain was not assessed in the baseline in the same way as other study arms. Evidence from all study arms suggest that the tripartite model improves community-municipal relations. While qualitative data indicate the strongest impact in the Retrospective 2 arm, there is also some evidence in the baseline/endline and Retrospective 1 arms. In all three arms, there were responses in each treatment site that indicated improvement, even though each site also had many respondents who did not express improvement. Leaders in treatment communities in all study arms were more likely to report meeting with local government than leaders in control communities (Exhibit 8). Exhibit 8: Percent of community leaders who indicated they meet with the local government Note: no data are available for baseline and Retrospective 1. In contrast, responses were consistently negative in the control communities. The municipality now, recently, is paying a lot of attention to the communities. Now, the local government is starting to take on directly social projects alongside communities, like this one, which is community￾government. So, it’s there that communities enter to form part of the municipality, or rather how you can see the importance of community participation. – Community leader, Retrospective 2 treatment, Norte de Santander When asked about their opinions on Colombia Transforma, community respondents generally mentioned the local government as a stakeholder in Colombia Transforma programming, but did not dwell on contributions from local government. Community respondents did not necessarily see each party of the tripartite model as equal partners, viewing the local government’s role as less important. For example, while community members in endline communities largely reported that Colombia Transforma’s reliability and follow-through was what set it apart from other community development projects, this trust in the program was not credited to the local government, but specifically to Colombia Transforma. - What entities participated? - The community is one of the entities because it’s a key foundation of our process 18 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV - Just CT, the community, and the municipality - What did each entity contribute? - The municipality, nothing more. The biggest part of the work was done by CT and the community. - The municipality was left very small – FGD Retrospective 2 treatment, Norte de Santander It appears that much of the tripartite model’s effect on community-municipal relations is achieved simply by being the first experience in which the two entities have collaborated. Colombia has long history of critical need for economic development in rural areas, particularly as rural areas have been the most affected by the armed conflict. Most recently, the 2014 Mission for the Transformation of Rurality in Colombia pointed to the absence of state agencies and state investment in rural areas. 9 As such, the signing of the 2016 Peace Accord and fostering community-municipal collaboration to implement it remain opportunities to close the gap between urban and rural development. Overall, there is a pervasive feeling of abandonment across all communities, both treatment and control; this sentiment is not solely directed to the local government. In addition to the tripartite model seeming to improve relations with the local government, treatment veredas felt more generally that Colombia Transforma eased the sense of abandonment, though this change was attached more to the program than to interaction with local government. Program and municipal staff were more positive about Colombia Transforma’s effects on relations than community members. Interviews with program staff suggest that the ability of local governments to enter veredas with security concerns improved as a result of the program, and that this change alone is a key shift in the relations between municipalities and communities, as it lays the groundwork for future programming. Staff referred to this shift at endline with reference to the baseline, as well as with reference to both retrospective arms. For example, at baseline program staff spoke of challenges associated with in-person visits due to security concerns, particularly in Bajo Cauca and Nariño. While security concerns remain, under Colombia Transforma local governments reported working closely with community juntas to enter veredas that often felt abandoned due to security issues. The program was very assertive, it has been very assertive in this area, even reaching very remote areas so that the communities do not feel alone, do not feel unprotected or abandoned, but rather on the contrary always accompanied. They see that presence of the municipality because that is our aim; not so much to make the program visible, we arrived with the municipality, we are contributing to generate that trust so that the municipality and the community can work in coordination, contrary to that image that was formerly had of “The municipality does not come to my community, we are abandoned, we are relegated by the municipality, they only come during election season.” – Program staff All respondent groups saw communication as critically important for positive community-local government relationships; as in other findings, perceptions of the relationship seem closely tied to communication and interaction, particularly in-person interaction. Program staff at endline noted the importance of in-person visits from local governments to veredas as a means of building trust and transparency. While not always possible due to COVID-19 restrictions and the sheer size of the territory that must be covered (as noted in the case of Nariño), program staff reported that local governments have made strides when it comes to improving in-person presence and are showing up to plan alongside communities rather than showing up for appearances’ sake. Some program staff, however, 9 Misión para la Transformación del Campo Colombiano 19 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV noted that local governments need to prioritize in-person monitoring and collaboration with communities if communities are to consider local governments (rather than Colombia Transforma) as drivers of local change. The administrative level of the planning office prevents the mayor's office from being much more visible in the communities and from being able to follow up on the projects. This is super important so that the communities not only identify Colombia Transforma as the leader of the activity, but also identify the municipal mayor's office as a leader. I believe that this element should be improved. Of course, there are some municipalities where a higher level of approval of the municipal administration was observed than others and unquestionably [REDACTED] was one of the municipalities where it was not. I don’t know if it is the volume of work or what, but the municipal administration had great difficulties in being able to carry out a rigorous follow-up with the activities of the tripartite model. – Program staff In a shift from evaluation Phase 1, during Phase 2 municipal staff resoundingly mentioned communities as being the ones best placed to understand their needs. While some municipalities at baseline indicated they did not consult with communities for projects, municipalities at endline all commented on the importance of working with communities to understand their needs, particularly in communities with Colombia Transforma presence. For Colombia Transforma projects, community members in endline treatment veredas did not describe the process of taking their request to the local government as they normally do with their community needs, but acknowledged that the community needs were solicited by the project and that the local government does not implement projects without consulting the community. It leaves me with many lessons, especially working as a team with Colombia Transforma and the community, understanding the community more, supporting them, listening to them… On one of the roads we were building, the money ran out—or, at least what had been projected—but we realized that in the future the road could be damaged because it would not have a culvert, so they came and together we looked for a solution and worked as a team, we managed to carry the project forward, so I also think it is how to listen to them, know what their needs are, put ourselves in their shoes. They also put themselves in our shoes because many presidents told us that now that they are doing this work they had to organize the stipends and payments which was a bit difficult but they said they learned and that they understood that for that people from the municipality who come to do this type of it is also difficult and now we are going to collaborate with them more. So, I think that this makes us all put ourselves in everyone's shoes and we can carry out a project. – Municipal staff FINDING 4: Communities perceive public service provision as being poor; this does not vary between treatment and control communities. Public service provision Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▬ No difference ▬ No difference ▬ No difference Across all study arms, evidence suggests that community members generally perceive the provision of public services to be poor. Community members in treatment and control communities alike expressed a need for potable water, reliable and affordable electricity, accessible health services, and better road maintenance. Some community members also reported difficult access (remote and insecure locations) to communities as part of the rationale behind the lack of services. The majority of community members 20 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV reported a lack of basic services, and, in some cases, when services were available (such as energy), the inability to pay the current price of the service. There is some limited evidence that community members who participated in the full Colombia Transforma model in Retrospective 1 and 2 areas have more sympathetic views of the barriers limiting municipal service provision than community members in control areas, even though all coincide in their assessment of the poor quality of services. In control communities, for example, respondents pointed to barriers such as negligence, lack of interest (except during elections), and lack of capacity. Some leaders and community members mentioned problems in management and a few community members went so far as to use the word corruption. Community members from endline treatment communities shared many of the same negative views of the obstacles related to the government’s provision of services, and did not express the sympathy shown by retrospective treatment areas. - What about education, health, housing, water, sewage, and energy services? - No, not at all. Negative to everything! In every sense, we are talking about the village because it is a rural area and they do not take us into account at all. Why is it that for a rural village you are acknowledged during election time; they see everyone everywhere and everyone greets you. When someone wins, no matter who wins, we are already invisible to everyone. – FGD Endline Control community, Antioquia - As far as obstacles, I sincerely think that it is the lack of political will because the conditions are given for us to have good services, but the local government abandonment is evident, so I think that this is the main obstacle that there is. – Community Leader, Endline treatment, Antioquia In Retrospective 1 and 2 treatment areas, however, community members ascribed poor service delivery more to a lack of resources at the local government level, though many still mentioned the factors noted in control areas as well. Among community leaders and members there is the overall perception that public services are the responsibility of the local government, but that the government seems to be unable to deliver. As such, public service provision is perceived as very inefficient. - In terms of public services in the community, it is very different from the urban area, because rural communities basically just have the sewage system, but the administration has served us with energy; in the matter of connectivity, we are making a great effort with the Internet issue, and work has also been achieved, so let's say, in the aspect of the urban to the rural it is a difference but yes... - What are the main obstacles that are presented to the municipality to provide a better service, in terms of public services? - Very complex, because public services in Colombia became privatized and it is very difficult for an administration to contract with a private entity, because it becomes very expensive for the community and it is something very complex that Colombia is facing, because it is very hard for an administration to obtain resources when in Colombia there is little for the public sector. – Community leader, Retrospective 2 treatment, Norte de Santander Municipal staff across municipalities reported working with communities to provide services, and are generally positive about their ability to provide services. Municipal staff noted, however, considerable challenges to service provision, including limited funding, difficulties accessing remote communities, and in Norte de Santander, a lack of sufficient qualified personnel. I think that since we are sixth category municipalities it is a bit difficult to obtain resources because they are very limited resources, many times we want to do more but it is not possible, so I would think that it could be improved if we had more cooperation, which will help us to be able to execute these projects in the long run are for the good of everyone; we would also do better if we had structured teams for the municipality's 21 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV resources. There is a lack of organizational strengthening so we can’t always contract with the people we’d like to develop projects. – Municipal staff FINDING 5: Evidence suggests that the tripartite model generally improves community members’ understanding of the connection between local development projects and the peace accord. With one exception, local governments generally feel under-resourced to implement the accords. Peace accord understanding Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher ▬ No difference ▲ Treatment higher Evidence suggests that while community leaders and community members do not always understand how the government is implementing the peace accord, the tripartite model is successful in improving this understanding. This understanding was found predominantly in Retrospective 2 communities and was especially prominent among community leaders. While some community leaders and members in treatment communities mentioned PDETs, the vast majority of respondents spoke about Colombia Transforma, specifically when asked about linkages of local development projects to the peace accord. Retrospective 1 treatment, as well as control communities in all study arms, did not relate the peace accord with any of the development projects in their veredas and had scant information about PDETs. Right now, through the PDET, a project was executed for a school and a school classroom and a health unit, they are also executing an exercise to legalize land … also improved tertiary roads related to PDET are also under consideration. – Community leader, Retrospective 2 treatment, Norte de Santander At endline, community respondents generally remained unaware of the local government’s involvement in peace accord or believed the local government had done nothing to implement the peace accord. Respondents in treatment communities, however, had more positive views of municipal support for the peace accord than their counterparts in comparison communities, as seen in Exhibit 9 (which also shows this effect for Retrospective 2). Many respondents at endline mentioned coca eradication efforts as part of the peace accord implementation. How has the municipality contributed to the implementation of the peace accord? R1: No, they have not participated in anything, at any point in time. R2: We are orphans here. – Focus Group, Endline treatment community, Nariño 22 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Exhibit 9: Percent of community leaders and members indicating the local government supported peace accord implementation Note: no data are available for baseline and Retrospective 1. Program and municipal staff agree that even when local governments have the willingness, there remain limited funds for peace accord implementation. Municipal staff generally feel the national government does not provide the necessary resources to implement the peace accord and for this reason appreciate Colombia Transforma as their primary mechanism to fund the local development aspects of the accord. Based on municipal responses, national funding seems to vary. At Phase 2, government staff from four municipalities indicated that they are receiving government funding for implementation of the peace accord; of these municipalities, one municipal staff also shared that their municipality has been prioritized for PDET funding because of tourism to the area which is considered to be a cultural municipality. Three municipal officials reported it is still challenging to secure funding, with one official reporting there is a strong sense among government officials that the national government is reluctant to provide funding for peace accords in areas with armed groups. Colombia Transforma has been one of the most important, that is the PDET investment that we have had in the municipality, the national government has not entered [this] area, it has not entered to collaborate in the way that it should be done so that a municipality can have the help and cooperation to be able to have several structured projects in all the sectors. We have had bottleneck difficulties in the approval of those projects. The municipality is a sixth10 category, the human resources are insufficient for us to structure the amount that we would like, but even so with those projects that we have already structured to obtain national resources… – Municipal staff COMMUNITY CAPACITY, EMPOWERMENT, AND OWNERSHIP FINDING 6: There is strong evidence that the tripartite model increases the capacity of JACs to manage more complex and ambitious projects. Community project management Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher ▲ Treatment higher ▲ Treatment higher 10 Please see Categories for Districts and Municipalities in the Glossary 23 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Comparison between control and treatment areas at baseline and endline as well as evidence from treatment areas in both retrospective study arms all suggest the tripartite model increases the project management capacity of JACs. Community members and leaders in treatment areas described projects, development priorities, and community needs with more detail than participants from control areas. They were also more precise about what they needed to implement and sustain projects. While Exhibit 9 shows that community respondents in both treatment and control sites across all study arms rated JAC performance more positively than negatively, there were slightly more positive ratings in treatment sites. The biggest contrast in rating is observed in the baseline/endline arm. At baseline, community members and leaders from control and treatment areas mentioned JACs are responsible for managing local development projects, including by coordinating informational meetings, conducting outreach to gather input on project implementation, and carrying out fundraising. Community members also mentioned that the JAC informs the community about projects and manages projects. Responses related to who is responsible for implementing public projects varied by respondent, with no clear pattern. Several respondents shared that it should be up to the municipal government to implement such projects, while others noted that the responsibility should fall on the JAC (and specific JAC members or committees) or the community more broadly. At endline, community members and leaders across treatment veredas similarly answered affirmatively regarding their ability to implement community projects. Exhibit 10: Community leaders’ views on how well JACs manage projects However, respondents then added substantial detail about how their previous experience implementing specific projects was an asset. Community members provided many examples of having come together 24 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV over the years to implement and maintain projects for their communities. Community members and leaders shared that they carry out these projects with their own resources or with the help of various organizations, including the private sector. They described the prioritization and internal decision￾making process at the community level. They provided many examples of successful projects implemented and financed through local fundraising efforts. In contrast, most respondents in control areas answered they were capable of implementing projects because their community could work together, but did not give specific details regarding their previous experience. Participants in control veredas that mentioned specific infrastructure projects had limited information about how they were implemented. Although this nuance is subtle and there were no responses from community members and leaders in treatment areas that explicitly attributed their project implementation capacity to Colombia Transforma or the tripartite model, comparison within sites indicated improvement between baseline and endline in treatment sites, and no change in control sites. - What projects did you complete or try to implement recently? - The bridge was one; fixing access roads was another; the cemetery another; the communal house. All the projects you see in the area, the sport courts, the park, road pavement: all projects are implemented by the community, we are involved in almost everything. – Focus group, Endline treatment community, Norte de Santander - Which are the main obstacles the community have to implement projects? One, that we do not have sufficient knowledge to implement projects and two that we do not have the right person that can help us with that. – Community leader, Endline control, Nariño Similar patterns exist in both retrospective study arms. Respondents in treatment veredas reported higher levels of confidence in their ability to carry out new local development projects, which some attributed to their communities’ prior work with Colombia Transforma and subsequent increased capacity to carry out future projects. In Retrospective 2, where implementation had been more recent, most leaders and some community members were able to describe in detail a number of projects successfully implemented in the past year. Responses included budgets, implementation phases, and how certain products could be connected with future projects. When asked about meeting with the municipality, respondents answered that meetings occurred periodically and that these were important for project implementation and sustainability, giving examples of petitions for additional budget. Some Retrospective 2 communities specifically attributed learning better ways to organize themselves to working with Colombia Transforma. One leader in Teorama compared the JAC with a small company. Retrospective 2 treatment areas also demonstrated deeper knowledge and experience in navigating obstacles in project implementation. Several respondents gave specific examples of difficulties that they overcame, such as clarifying property titles prior to starting project construction. Only a few responses criticized JAC performance or were vague regarding project implementation. In fact, when asked to rate JAC performance in project implementation, 58 percent of respondents in treatment areas gave the highest score and there were no responses with the lowest two scores, compared to 33 percent of responses in control areas giving the lowest scores. We are in the capacity of implementing new projects. We are registered, we are a small company and we can take on local projects or with international or national aid. We are ready. – Community leader, Retrospective 2 treatment, Norte de Santander 25 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV The evidence provided by community members’ and leaders’ responses is confirmed by municipal and program staff. At endline, municipal government officials reported that as a result of Colombia Transforma tripartite model, community leaders have increased technical skills and are now better equipped to take care of projects in the future. FINDING 7: All respondent types view the JAC as playing a central role in communities, particularly with regards to managing local development projects. This is true in both treatment and control communities. Role of JAC Baseline Endline Retrospective 1 Retrospective 2 n/a* ▲ Treatment higher ▲ Treatment higher ▲ Treatment higher As seen in Annex A, this domain was not assessed in the baseline in the same way as other study arms. Community members usually gave positive feedback regarding the JACs’ work and commended leaders’ dedication. Data show this pattern across all study arms, both in treatment and control areas. Community leaders across all veredas reported similar JAC structures, with some committee-level variance, such as the presence of a conflict mediation committee. Most descriptions included a committee in charge of work and implementing projects. None of these organizational differences varied by region. All respondent categories emphasized that JACs are very important to the community and help address internal community matters and common priorities, such as project implementation. Community leaders shared that their role as leaders primarily included organizing community initiatives and serving as a resource for community members. Community leaders in all study arms and areas reported undertaking a wide range of activities, such as public works projects, informing community members of the goings￾on, and providing financial and material aid to community members in need; community members were less detailed in their descriptions but confirmed similar roles. One distinction data show between treatment and control areas is that respondents in control areas more frequently perceived there were not enough projects to implement. - What is your opinion on the leadership and work of the JAC? - The work they do, their leadership focuses on developing constructions, projects. They are very good. – Community member, Retrospective 2 treatment. Norte de Santander At endline, municipal and program staff agreed JACs are key for community organization and emphasized their importance vouching for projects and staff to armed groups. Staff focused on the important project implementation role of JACs and linked this to appreciation for the tripartite model providing further training and empowering communities in project implementation. Across all regions, but particularly in Norte de Santander, municipal officials at endline reported that communities and community leaders involved in the tripartite model are now well-versed in municipal processes. Municipal staff saw capacity building for JACs as a major strength of the model, as community groups are now able to solicit and manage their own projects, manage funds, and take care of contracting. Program staff in Nariño and in the Norte de Santander indicated that the municipality and the community were largely seen as equal partners. In Norte de Santander, however, program staff reported that it was thanks to the leadership of local JAC that the projects were able to push forward, 26 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV as the local government had a limited number of professionals who were able to do this work. In Antioquia, program staff at endline noted that local government was unable to provide 100 percent of what they had committed due to security and governability challenges, whereas the community came through with 100 percent of its contribution. Some leaders, particularly in treatment sites at endline and Retrospective 2, did not necessarily see each party of the tripartite model as equal partners, viewing the local government’s role as less important. They mentioned local governments were less present in project implementation and introduced obstacles. Some suggested Colombia Transforma could facilitate a more open dialogue with the alcaldías. FINDING 8: Treatment and control communities across study arms consistently feel they have insufficient access to funds for all the projects required to meet community needs. However, communities also have multiple creative strategies for small local fundraisers they have used to collect community match resources; fundraising strategies appear somewhat more ambitious in treatment communities. Project funding Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher n/a* ▲ Treatment higher *As seen in Annex A, this domain was not assessed in Retrospective 1 in the same way as other study arms. Across all study arms and both treatment and control, community members and leaders described multiple fundraising strategies they have used over the years, including bingos, sports tournaments, parties, raffles, and paying dues. With these funds community members have built and maintained roads, aqueducts, cemeteries, and police posts. Community members provided cash, leadership, coordination, advocacy, and manual labor to contribute to these projects. Despite this initiative, communities feel they have insufficient access to funds for more complex and ambitious development projects. Retrospective I communities (both treatment and control) mentioned barriers to undertaking local development projects include funding limitations and budgetary restrictions. Control veredas at endline and Retrospective 2 argued communities were too poor to implement projects on their own, and added they were willing to contribute with manual labor and small amounts of money. Treatment areas at endline and Retrospective 2 mentioned pressing health, education, and communication needs that remained without funding. Some leaders and community members warned these projects had to be funded externally. However, treatment communities mentioned several creative and more ambitious strategies to collect community match resources. Some leaders in treatment areas at endline, Retrospective 1 and 2 explained fundraising activities to add phases or start development projects. - How does the JAC fundraise for projects? - Well, in JAC meetings they ask for a contribution to each farm and to each business and we all contribute our grain of sand and we find out how much was collected and how much is the project. - Have you ever done collective fundraising activities, such as mingas, bazares? - Yes sir. We do bazares frequently. 27 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV - How do they work? - They work well because we are very responsible. Each person is in charge of something: one of the beer, another of the food. So everyone has to be responsible. - If the community does not have enough funds or manual labor, what do you do? - We do raffles and again we collect contributions from all the community and everyone contributes. - If someone does not pay, are there sanctions? - There are sanctions or fines. There is a fine and a sanction that the person has to attend or pay the day that the person is not in the project. – Community leader, Retrospective 2 treatment, Norte de Santander Municipal staff at endline mentioned communities came to the alcaldías to propose joint projects with community funds. Responses did not differentiate between communities that had participated in Colombia Transforma and those that had not. For example, in El Carmen municipality, municipal staff reported that communities had gotten creative to raise funds (such as through tolls) and had approached the local government with requests for support to match the funds they had already raised. According to program staff, some communities are now taking on their own projects with only community funding. Some communities come to the office and tell us that they create tolls in tertiary roads, meaning, they put a rope every time a car passes by and ask for 1,000, 2,000 pesos [25 and 50 cents of a U.S. dollar, respectively]. With these they start a common fund and come here and tell us: Look we have raised 4,000,000 million [1,000 USD] to complete a public work: How much can you add? – Municipal staff Program staff at endline in both Antioquia and Nariño noted a need to further clarify roles and responsibilities in the tripartite model. In many cases, there was not always a common understanding among the community and local government of manual labor and what that entailed, resulting in confusion. Some program staff added it was important to clarify roles and responsibilities for the tripartite model while taking into consideration the realities of the poverty of some of the communities the project served—in some cases (as noted in Nariño), asking communities to contribute manual labor is a significant burden because it can take them away from their livelihood. FINDING 9: There is strong evidence that the tripartite model results in high degrees of project ownership by communities. Project ownership Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher n/a* ▲ Treatment higher *As seen in Annex A, this domain was not assessed in Retrospective 1 in the same way as other study arms. Across treatment veredas in all study arms, community members and leaders are proud of their unity and their efforts to develop projects. Community members are proud of their community leadership and expressed that their principal challenge to working on development projects is not internal, but rather the lack of funds and support from external organizations. Most community members and leaders indicate that their communities are ready to implement new projects because they have rallied together through the years, violence, and difficulties. Respondents from treatment veredas pointed to their previous work with the Colombia Transforma program as evidence of their capacity to take on new 28 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV projects in the future. Community leaders and members also described how the community manages projects in a transparent and direct manner. Both leaders and members consider themselves to be very united and collaborative and are proud to respond to their residents’ needs. In the case of one endline treatment vereda, some community members mentioned that men were not willing to do the manual labor, so the women did the work, resulting in reported increased feelings of empowerment and pride as a result. - What are the things that make you think that you can undertake projects? - The unity we have. We are always very close. Despite the fact that we are a community so affected by violence, we have always been very united. – Community member, Endline treatment, Antioquia While respondents from control veredas reported confidence in their abilities to implement projects, data indicate this belief is more aspirational and less based on prior experience and successes. Control vereda respondents reported preferring infrastructure projects, yet this type of project is considered to be outside the reach of JACs as a result of limited resources. Participants in control veredas that mentioned specific infrastructure projects had limited information about how they were implemented. Local government authorities and program staff reported that the participation of local communities in the tripartite model was critical to secure community ownership. According to program staff, many communities accustomed to receiving a great deal of external support in the past were empowered to contribute towards their own local development to improve community well-being. Program staff agree that when the community participates, they are more likely to take care of local development projects in the long run. In almost all cases, all municipal staff reported that communities came through or exceeded their allotted contributions to projects. Because the projects come from the communities, and because communities have the ability to monitor projects themselves, municipal staff reported that communities are empowered because they are holders of all the information. We see a difference in the projects we carry out—we have all the documentation, but in this case they [the community] have all the documentation in their hands and can carry out the respective review so that these resources and everything within the project is done properly. They physically have the documents there to do the reviews and monitoring, they come to supervise the project, besides just us. – Municipal staff FINDING 10: Evaluation evidence suggests the tripartite model leads to greater community initiative for local development projects, particularly vis-à-vis interactions with the local government. JAC project initiative Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▲ Treatment higher n/a* ▲ Treatment higher *As seen in Annex A, this domain was not assessed in Retrospective 1 in the same way as other study arms. Evaluation evidence show that treatment communities in endline and Retrospective 2 are more empowered to make requests to their local governments than control areas, even if they are not always answered. While this theme did not emerge inductively from data in Retrospective 1, minor differences in questionnaires and coding could explain this absence as a design limitation for that study arm. Community respondents in these areas spoke of the importance of community unity and strong community leaders who advocate for projects at local levels. While treatment areas didn’t necessarily 29 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV state directly that they take more initiative as a result of Colombia Transforma, comparison of treatment and control responses showed consistent patterns in treatment areas that reflected this greater initiative, and the baseline/endline study arm detected a change in this dimension after implementation of the tripartite model. - Do you think your community can undertake new projects? - Well, yes…we think at least with all that about the implementation of the peace agreements, we want to do a project every year with the help of God, because at least we know that peace—at least that one of the implementations of the peace agreements—is not going to be brought to us in a sack. We have to manage peace ourselves here in our community, and how do we have peace? By having everything you need; roads, health, education, electrification. That is, for me that is peace because… one doesn't wait for it to come. - And what makes you think or what are the main things that make you think that the community can undertake new projects? - Seeing that at least that we have one project, and we have very good leaders, you know? When you have good leaders who manage, who care about the needs of the community, you know that you can achieve many things. – Community member, Retrospective 2 treatment, Norte de Santander In Retrospective 2 treatment communities, some respondents also mentioned the importance of the local JAC securing proper legal registration, or strengthening their internal organizational capacity to ensure they are able to effectively advocate for and manage development projects. - Do you feel that the community can undertake new projects? - We would say that we have the capacity, we are registered, at this moment we are in a microenterprise, we are capable, the community, for any project, whether it is local or with international cooperation, or with national ones, we are prepared. - What are the main obstacles to undertaking community projects? - I would say that starting with the organization of the juntas, because a junta is like a citizen, if you don't have the documents, you can't go out freely. For the juntas, that is one of the most difficult obstacles… that is an obstacle that must begin to be broken in order to have development. – Community leader, Retrospective 2 treatment, Norte de Santander Program and municipal staff both pointed to community groups’ improved working knowledge of local government processes in treatment areas. Municipal staff attributed this directly to Colombia Transforma. Across all regions, but particularly in Norte de Santander, municipal officials at endline reported that communities and community leaders are now well-versed in municipal processes. Community groups are more formalized and understand the requirements to take on projects with the municipality. Program staff in some regions noted that community leaders and members took particular interest in understanding how to submit petitions to the government; they are now actively and regularly going to the municipality with petitions. As noted in Findings 6 and 8, municipal staff pointed to capacity building in project management as a major strength of the tripartite model, and also indicated that some treatment communities have gotten more creative in fundraising after their Colombia Transforma project. Municipal staff connected these improvements to the parallel growth seen in community initiative and project ownership. While some community members in endline and Retrospective 2 treatment communities expressed they had felt some degree of self-doubt at the start of the Colombia Transforma project, respondents shared how they 30 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV persevered, worked together, and were able to complete what they set out to do, thus encouraging themselves to take on future projects. Program staff confirmed this, reporting in some cases that some communities were initially skeptical of the tripartite model or their ability to contribute. As a result of the tripartite model, however, communities were encouraged to participate and understood the value of their contributions to their infrastructure development projects. - What are the main obstacles that arise when carrying out community projects? - I said it once and I repeat it, the lack of ownership; we do not appropriate what is ours, people look for what they are going to be given, how much money I am going to receive? As right now, the tripartite model is working with Colombia Transforma, the mayor's office and the community. The people were difficult, very difficult, but at the same time we gained a lot because we learned to work as a team as a community; some, not all. Children and adults…participated in the project, we were many. That is why I say, a project so desired, so longed for and it arrived thanks to God, the mayor's office, Colombia Transforma, Zona Futuro and all the entities that joined this cause. Before, as a community we were focused on what they were going to give us, what we were going to receive, how much is my payment? It is difficult but with the community members that we had at the helm, we are completely convinced that public projects can come, whatever projects there are, we are going to achieve it. – Community member, Endline treatment, Nariño COMMUNITY RESILIENCE AND ADAPTABILITY FINDING 11: Violence has become normalized and both treatment and control areas have developed coping strategies based on avoiding threats.The state is not seen as being able to provide security. Ability to manage insecurity Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▬ No difference ▬ No difference ▲ Treatment higher Community perceptions regarding the ability to manage insecurity vary by municipality, but with the potential exception of Retrospective 2 these perceptions do not vary by treatment and control within study arm. There is some variation across study arms, with Retrospective 1 communities showing more positive perceptions than baseline/endline and Retrospective 2; this variation across study arms may be explained by the distinct contextual differences in municipalities across the sample. Generally, in all control as well as endline treatment communities, community leaders from more peaceful veredas feel they can protect their residents, whereas community leaders in veredas where significant external threats exist do not feel capable of protecting residents. The vast majority of community members and leaders in Retrospective 2 treatment communities, however, reported feeling able to protect their residents from threats. Given the known presence of armed groups during data collection in some areas, however, this finding has significant limitations as it is possible that respondents were nervous to answer openly, thereby providing a positive bias in the data. Yes, the community gathers a lot and does not let any violence happen. – Community member, Retrospective 1 31 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Communities largely feel that they are the only ones who respond to security threats and that local government and police are absent. Within these overall perceptions, data suggest Retrospective 2 is a small exception to lack of variation within study arms, with treatment communities describing slightly higher ability to manage insecurity than control communities. While most community members in endline treatment communities reported that the security situation has not changed and has been good in the last two years, some community members expressed that the situation has either deteriorated or improved. In contrast to the approach of threat avoidance and perception of state absence observed in baseline/endline and Retrospective 2, community respondents in retrospective treatment 1 communities suggested that municipal governments are effective in managing security conditions. Several community members also noted that they have confidence that their community leaders would be able and willing to organize a response to a threat. While these data appear to be an outlier compared to other study arms, this could be due to data limitations; several respondents in Orito and Valle del Guamuez were unwilling to respond to these questions, and community members in Teorama expressed views in line with the baseline/endline and Retrospective 2 communities. Community members from endline and Retrospective 2 communities mainly rely on the JAC as an authority for security incidents. Many respondents state that there are no other security forces available to protect them in their communities. Many community members also describe not being able to rely on government security, or even not wanting to. Municipal staff and program staff alike consistently reported the importance of working closely with community leaders to manage security challenges in veredas. In both evaluation phases, program staff reported consistent examples of the importance of community groups in mitigating security issues to move Colombia Transforma programming forward. Municipal and program staff see JACs in particular as key for managing security issues, noting they have the authority to address security situations in veredas and armed groups generally respect their decisions. In order for outsiders like Colombia Transforma and municipal governments to safely enter communities, it is essential to liaise with community leaders who understand security concerns within the veredas and can advise on how to access veredas securely. Data collection teams in both evaluation phases also experienced this phenomenon, relying heavily on the JAC leadership to facilitate access to all veredas. We truly count on the presidents of the JACs who are really the people in charge in the veredas because even when there are those illegal groups, the moment that the community rises up, the JAC rises up, they don’t allow the project to fall, they don’t permit them and we can then enter the vereda. – Municipal staff Program and municipal staff also see community leaders as key to ensuring the success of the projects and protection from external security threats. In two regions, program staff reported that community leaders were able to successfully stop extortion efforts. According to one program staff member, due to the presence of armed groups, some local governments almost see community leaders’ role protecting and maintaining the projects after construction as more important than their contribution during the construction itself. In one case where security concerns kept the local government from entering a community, municipal authorities reported that the junta almost singlehandedly carried out the entire project, taking possession of the Colombia Transforma material at the main road, carrying it into the community, and working with the municipality to monitor the project virtually. Program staff spoke of the importance the project has had on removing stigmas of “dangerous” veredas. By working with local government in one municipality to enter a barrio that had a history of armed groups, staff were able to start entering without community escorts and send a message that individuals involved in the project were not coming to get the armed groups, but to provide services that everyone could benefit from, 32 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV which is key to the success of the project. According to program staff, the trust built by the project in these communities resulted in increased trust in local governments associated with the project. To protect those people who come here, the community always protects the people who come here to do any interviews or whatever activity that arrives. Here the community grants whatever security is needed. – Community member, Endline treatment, Nariño FINDING 12: Communities have worked to address conflict among residents through communication and dialogue, yet in cases of external aggression communities flee and hide, later returning to their communities once it is safe. Resilience Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▬ No difference ▬ No difference ▬ No difference Community leaders and members described very different responses to security from one community to the next. These responses seemed to vary in terms of the security situation within that municipality, and not by study arm or treatment/control distinction. Several community leaders and members commented on the importance of community dialogue as a means of addressing security issues and conflict within communities. Several respondents indicated that safety decisions are usually made at the family level; however, some communities hold meetings to discuss family-level strategies or provide advice to one another, such as not allowing youth out on the streets after a certain time, not wearing all black (which is an armed group uniform), and advising youth to study. As a peasant and as civilians, I think it is very hard for us to face conflict situations. Of course, with good words and speaking it is can be possible to dialogue, but when [armed groups] get too upset, one has to lower one's head. – Community member, Retrospective 1 Across all communities in all study arms, community respondents indicated that the community is prepared to address conflict among residents, but that the community cannot protect residents from external aggression. In such cases, community members are left no choice but to flee and hide. In the case of one control vereda, residents had only recently returned after being displaced due to violence in their community. What does the community do if there is an act of violence? We hide. In those cases, what do people do to carry on with their day to day lives? Carry on like nothing happened is quite difficult, don’t you think? – Community member, Endline treatment, Antioquia FINDING 13: The majority of community members in both treatment and control areas continue to equate civic engagement with elections (rather than other forms of civic engagement), indicating the tripartite model did not have a measurable effect on perceptions of levels of civic engagement. Civic engagement 33 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Baseline Endline Retrospective 1 Retrospective 2 ▬ No difference ▬ No difference ▬ No difference ▬ No difference Across all study arms, community respondents primarily viewed civic engagement in terms of voting; this did not vary by treatment and control areas. This view of civic engagement may be in part due to the formulation of this line of questioning in the tools, which referred to “political participation” in both phases of the evaluation; however, during most interviews in Phase 2 data collectors clarified “political participation” in terms that encompassed civic engagement. Community respondents expressed voting as a way to implement changes in communities and said that they are motivated to vote in the hopes that elected officials can implement improvements in their personal and community situations. Elections are seen as the most direct way to advocate for community needs, even though promises are not always kept. While generally community respondents felt that it is relatively easy to participate in voting, some community members mentioned poor roads and limited transportation as obstacles to voting, in addition to the fact that some veredas must first come to a common agreement regarding which political candidate to support, which can take time to work through differences. According to data collected by the Electoral Observation Mission in the 2019 mayoral elections, the turnout rates for the evaluated municipalities varied. While 43 percent of the electoral census voted in Tarazá, in Teorama 72 percent of the electoral census voted. Exhibit 11: Voter participation rates in 2019 municipal elections Political participation, or the business of electing and being elected… Here the people like voting in elections because they always represent themselves with their vote. – Community Leader, Retrospective 2 treatment, Nariño Community members and community leaders also shared that an important aspect of civic engagement is being able to know and express what the community’s needs are. It is representing the community and advocating for community projects. Respondents expressing this view of civic engagement saw political engagement as effective when their petitions resulted in the desired projects. There is general disappointment in politicians' ability to deliver on campaign promises. Community members in some endline treatment communities indicated they would be motivated to participate in civic engagement activities if their petitions were answered by the local government. Most respondents, however, indicated that their petitions to the local government were ignored. Other avenues for citizen engagement mentioned include petitions (see Finding 10)—though they noted these petitions are often ignored—participation in local development plans, engagement with local 34 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV authorities through JAC presidents, and direct engagement with the local government via phone or WhatsApp (Retrospective 1). Endline control communities mentioned protests as a specific avenue of participation for youth. Program staff commented on local governments’ outreach to communities as a means of promoting civic participation, speaking particularly about the benefits of mesas de incidencia y dialogo as a means of engaging in dialogue with communities. Mesas de incidencia are seen by staff as important to soliciting local input; given that there are various mesas for specific groups (youth, women, the elderly, etc.), mesas are seen as an effective way to solicit feedback from specific populations. Municipal staff overwhelmingly reported that the increased involvement of local government in communities through Colombia Transforma has resulted in increased citizen engagement. According to municipal staff, communities are more motivated to participate now that they have knowledge of how to petition governments and convenios solidarios (see Finding 15), and have interacted more with local government. Some municipal staff also noted that previous administrations had failed to take initiative, and that citizen engagement is also increasing as a result of increased municipal communication and sensitization efforts such as through websites, Facebook pages, and information campaigns. When I started at the municipality the communities were very isolated, but since these types of processes have come about, the community members have been open to speaking with the mayor, to come to our offices to solicit projects without fear. They now know they can carry out convenios solidarios—they come here saying “You can’t tell me no because I have all my junta documents” so I think they are now very clear on that front. For that reason, I say there is participation because they approach us—we don’t have to go seek them out. They are very willing, they bring their requests themselves, and we see more that they are there, present and communicating their needs and advocating for themselves – Municipal staff CROSS-CUTTING FINDINGS FINDING 14: Respondents in treatment areas and municipal staff consistently praised Colombia Transforma, identifying several benefits of the program and its infrastructure projects. Community praise did not always extend to the tripartite model requirement to work with government, even though improvements observed by this evaluation seem to stem from the model. Views of Colombia Transforma Community members and leaders from treatment areas across all three study arms mentioned direct benefits of Colombia Transforma projects and the tripartite model implementation process. Most community respondents described direct benefits from the infrastructure, including wider and safer recreation spaces, more hygienic schools and lunch spaces, and better communication between veredas due to placa huellas, projects focused on road pavement. Leaders and some community members referred specifically to benefits from the implementation process, such as increased trust and unity within communities, people who had not previously engaged in community work, and increased motivation due to projects’ size and quick completion timeline. Leaders and community members in some control veredas at endline mentioned hearing about Colombia Transforma in neighboring veredas. In Tumaco, participants referred to a polideportivo built in Imbili through the tripartite model. In Caucasia, community leaders mentioned tripartite model projects implemented in Cacerí. In both Tumaco and Caucasia, these control community respondents were 35 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV impressed by the way the projects had been implemented, said they benefited directly from the projects because they used the resulting infrastructure, and hoped the same model could be implemented in their own veredas. Some respondents in Norte de Santander control communities also mentioned having heard of Colombia Transforma, or having participated in tripartite model projects themselves. The data did not offer additional clarity, but this could reflect the density of interventions in this region over the course of CT 1 and CT 2. In some endline treatment communities, some respondents mentioned delays in municipalities’ participation in CT projects and suggested Colombia Transforma should work directly with communities, without municipalities. In Retrospective 1 and 2, community members and leaders did not hint at excluding municipalities. Program staff also mentioned certain delays from municipalities, but did not suggest excluding municipalities. Program staff from Norte de Santander suggested that convenios solidarios were helpful to address municipal staff shortage and delays, by devolving more of the municipal role to communities themselves. Municipal staff did not mention these shortages and delays when asked about lessons learned from Colombia Transforma. As mentioned in Finding 9, during evaluation Phase 2 municipal staff commended community participation in the tripartite model as a key factor in the success of the program. Program staff further described this participation across all regions during Phase 2, as well as with reference to Retrospective 1 areas during Phase 1. Program staff had mixed perceptions about the extent to which local government, communities, and the Colombia Transforma project were equal partners in the tripartite model. In Nariño and in the Norte de Santander, the municipality and community were largely seen as equal partners. In Norte de Santander, however, program staff reported that it was thanks to the leadership of local JACs that the projects were able to push forward, as the local government has a limited number of professionals who are able to do this work. In Antioquia, program staff noted that in some municipalities, the local government was unable to provide all of what they had committed due to security and governance challenges, whereas the community did come through with their whole contribution. Some program staff recognized the success of the tripartite model but raised concern that despite the program’s efforts to reduce its own visibility and defer credit to the other partners involved, communities and municipalities kept recognizing the program’s commitment and processes and not the other partners’ participation. Overall, all types of participants across study arms had positive views of Colombia Transforma projects and mentioned specific benefits related to the tripartite model. FINDING 15: Program and municipal staff in Norte de Santander suggest convenios solidarios are an opportunity to increase the tripartite model’s impact in improving state-society relations and communities’ capacity, empowerment, and ownership. Convenio solidario The four treatment sites sampled for the Retrospective 2 study arm in Norte de Santander all used a convenio solidario; this sampling had the explicit intention to provide evidence related to use of convenios solidarios. Program and municipal staff from Norte de Santander explained convenios solidarios between JACs and municipalities started to be used as part of the tripartite model during CT 2 in select veredas. Overall, these respondents detailed very positive results of using this tool. 36 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Program staff agreed convenios solidarios are a powerful tool to address community demands and can further improve communities’ trust in local government. They added that convenios solidarios increased JACs’ project implementation capacity, as municipalities delegated contract obligations and direct budget management. Given that the budgetary limits for a convenio solidario will increase when a new legal reform is completed, program staff in Norte de Santander also spoke of the importance of continuing to support local JACs or groups to legally register themselves and build their capacity, especially to establish savings accounts, so that they can qualify for a convenio solidario. Some municipal and program staff added that many JACs—estimated as high as 60 percent—are not formalized, and this is a requirement for signing a convenio. Convenios solidarios are a good tool because they allow JACs to manage resources directly given by the local government and that gives them experience in managing public budgets. Strengthened JACs with organizational, administrative, and financial capacities. JACs formalized. In Catatumbo JACs are strong but they are not formalized, they did not have documents, tax registry, books…Through convenios solidarios, now we have JACs that now have contracts with various institutions. – Program staff Municipal staff from Tibú, Teorama, and Sardinata mentioned convenios solidarios were used recently to implement Colombia Transforma projects as well as other development projects. They celebrated JACs’ increased capacity to implement projects and spend budgets. Municipal staff in Sardinata noted that the municipal government recently changed fund disbursement procedures and now requires JACs to show progress against milestones before disbursing 100 percent of funds. The municipal government made this change after observing that JACs reported spending the entire budget before finishing projects. While the evaluation did not explicitly ask about misuse of convenio solidario funds, no respondents indicated that fund misuse was a challenge with this mechanism. In Retrospective 2, three of the four treatment sites that had used a convenio solidario reported they were familiar with this tool, and mentioned that it had been used in implementing their Colombia Transforma project (Retrospective 2 was the only study arm where this was applicable). In each of these three communities, at least some of the community leaders mentioned convenios solidarios were used for Colombia Transforma. These community leaders did not give much detail about how they worked, although they seemed satisfied with results. Only one respondent in a focus group had a negative view of the tool, mentioning that resources given through convenio solidarios were insufficient and that there were many taxes that made convenios solidarios ineffective. Overall, leaders reported having learned the power of convenio solidarios through the Colombia Transforma program, citing the importance of unity across all actors to achieve great things. Community members were generally not familiar with the convenio solidario mechanism. This process has been one of the best, because we learned and experienced and as a result we have done great things. One of the biggest achievements is that we united under a convenio solidario and we realized that it was a total success that we were able to achieve with Colombia Transforma and the municipality, an excellent team. – Community Leader, Retrospective 2 treatment, Norte de Santander Communities in the Retrospective 1 and baseline/endline study arms were not familiar with convenios solidarios, which aligns with program information indicating their projects had not used them. The evaluation did not find evidence of comparison veredas that had started self-organizing to approach the municipality or OTI, though as noted in Findings 6 and 10, some treatment communities did appear to self-organize for future projects after participating in the tripartite model. 37 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV CONTEXTUAL FINDINGS The findings in this section discuss community members’ and leaders’, as well as municipal and program staff’s, perceptions of security challenges and the impacts of COVID-19 on project implementation. FINDING 16: Across all municipalities, most community members and leaders indicated security threats have not substantially changed in recent years and perceived that insecurity had a minimal impact on project implementation. Security context Community members and leaders from all municipalities shared perceptions of ongoing security threats from armed groups and considered the security situation remained unchanged in recent years. In Tarazá and Caucasia, most respondents indicated there had been little to no change and that even though threats persisted, in the past six months there had been fewer security incidents. In fact, leaders from one vereda in Tarazá explained that community members were returning after being forcibly displaced several months ago. This contrasted with perceptions of community members and leaders in Tibú, Teorama, El Carmen and Sardinata (all municipalities of Norte de Santander), who reported in December 2021 that violence had increased. However, during evaluation Phase 1 (baseline), some evidence for these same municipalities suggested improvement of security conditions in the prior year (despite this trend contradicting official statistics). In Tumaco responses were mixed and most respondents did not mention changes in the security situation. However, in all municipalities just a couple of participants admitted the JAC had received threats in the past year. In Retrospective 1, respondents in Valle del Guamuez felt security had improved. In endline municipalities, where Phase 1 previously collected data close to a year earlier, perceptions of security were more positive in Caucasia and Tarazá and were worse in Norte de Santander, while in Tumaco remained the same when compared to baseline. Exhibit 12: Community leaders’ and members’ views regarding change in security situation over past two years During data collection in Norte de Santander, interviewers observed that respondents were uncomfortable answering security questions. In some veredas—including both treatment and control— interviewers mentioned armed groups were present around data collection sites and perceived this affected the interviews and focus groups. Despite perceptions of insecurity, respondents in treatment areas agreed overall that there were no direct security incidents related to Colombia Transforma project implementation. This contrasted with program staff evidence mentioned in Finding 11; the staff described two incidents where community 38 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV leaders stopped threats of extortion to Colombia Transforma projects. Community members’ and leaders' responses in treatment areas also diverged from responses in some control areas which mentioned incidents of extortion in implementation of non-Colombia Transforma community development projects. - How is safety in the vereda? Have there been changes in the security situation in the municipality in the last two years?) - Horrible. - Is it worse? - It is worse. - How have security threats affected the Colombia Transforma tripartite model? - It has not affected because thank God nothing ever happened and things worked out well and they were done with no difficulties. – Community leader, Retrospective 2 treatment, Norte de Santander Security perceptions are mostly corroborated by homicide rates and the number of social leaders killed in the past years. As Exhibit 13 shows, except for Valle del Guamuez in 2020, homicides rates were higher than the national rate in 2020 and 2021 for all nine municipalities sampled. OTI staff with precise information on security threats in treatment areas reported presence of more than one armed group in all, except in Tarazá and Antioquia, where only one armed group holds control over the territory. Exhibit 13: Homicide rates per 100,000 by municipality per year, 2015–2021 Source: Cases and rates for 2015–2019 were reported by Instituto Nacional de Medicina Legal, Forensis and cases and rates for 2020 and 2021 were calculated based on Police report of cases and DANE report of population. 39 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV FINDING 17: COVID-19 affected project implementation schedules due to common disruptions such as supply chain issues and difficulties in meetings. Communities emphasized delays in project start up while program staff emphasized that despite challenges, overall projects mostly finished on time. COVID-19 impact on project implementation Community members, leaders, and municipal staff across all study arms agreed COVID-19 negatively affected economic conditions through loss of income, price increases, transportation disruptions, loss of access to markets, materials and supply chain disruptions, and reduced workforce. Thus, communities and municipal staff reported delays and even cancelling of some development projects, which did not include Colombia Transforma projects. Colombia Transforma staff described that even though COVID-19 restrictions imposed challenges for project implementation, communities helped Colombia Transforma projects continue and communicated on a regular basis. - From what you know, how has COVID-19 affected development projects in the vereda? - Well, many projects stopped, only few continued. The one that we wanted the most: the construction of the sport park and the children park started with the pandemic and it was delayed because the budget was fixed and with COVID, supplies’ prices increased and for a while the project did not move. Then things started going back to normal and the project started again and thank God the project was finished. – Community member, Endline treatment, Nariño - Fortunately, communities did well. We had several constructions in place and communities decided not to stop, they agreed that projects should continue. They put checkpoints in the road, but gave way to trucks transporting CT materials. When municipalities saw that communities were responding, they responded as well and so did the Program. Overall, the pandemic for us was sort of a success because despite challenges projects were not cancelled. – Program staff In contrast, in control areas at endline, community members and leaders perceived that COVID-19’s impacts were severe on development projects. They reported complete cancellation of most projects. Control areas’ evidence of severe delays and general cancelling of projects suggests other development projects outside Colombia Transforma programming may have been more affected by COVID-19. Information from the National Institute of Health suggests confirmed cases and deaths related to COVID-19 were not worse in the municipalities visited for the evaluation than in other parts of Colombia with similar populations (see Exhibit 1). Across both control and treatment areas and all study arms, leaders and community members described following health protocols and adapting to the new reality. As such, they perceived COVID-19 as diminishing their capacity to meet and work together. This perception contrasted with program staff’s view that COVID-19 was an opportunity for greater community leadership and engagement. By evaluation Phase 2, community members and leaders in treatment and control areas covering both endline and Retrospective 2 stated that JAC meetings and some project implementation meetings had been reinstated. It is important to note that Phase 2 data collection finished before Colombia’s fourth COVID-19 peak. 40 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV FINDING 18: Remote techniques are viable alternatives for monitoring project completion and municipalities used remote monitoring successfully. However, remote techniques are a poorer substitute for coaching, mentorship, and ongoing support to communities. Remote monitoring Despite COVID-19-related challenges, in evaluation Phase 2, program and municipal staff acknowledged virtual monitoring had become a viable alternative and was successful for monitoring the completion of projects. Respondents mentioned virtual monitoring started when quarantine restrictions were imposed and did not mention if this had stopped. However, both types of participants perceived those virtual techniques were not suitable for ongoing monitoring and support to communities during project implementation. Initially, I could not leave my home, we had to work from our homes, there were more restrictions, and we could not be here (referring to the office) to assist and support communities and there was no alternative to accompany communities' processes. Now, when restrictions are lifted, we are reaching communities most in need. – Municipal staff Due to quarantine measures, government officials reported that initially, in-person visits to communities were put on standby, thus affecting their ability to connect with communities. Once quarantine restrictions lifted, however, government officials reported that their ability to visit communities had improved. Community members and leaders mentioned virtual meetings with municipal and program staff as alternatives to in-person monitoring, but were less enthusiastic about it. Connectivity gaps are pertinent; when asked about provision of public services, in most areas, participants responded that access to the Internet was limited. Also, some community members and leaders acknowledged they had problems adapting to remote techniques. - From what you know, how has COVID-19 affected development projects in the vereda? - Virtual meetings were a hazard. We thought they were very complicated and still do. – Community leader, Retrospective 2 treatment, Norte de Santander FINDING 19: Respondents across communities and municipal staff could not establish an impact of COVID-19 on security issues. However, there is some evidence that COVID-19 restrictions may have added challenges to already dire security conditions. COVID-19 and security During strict quarantine measures and specifically at the beginning of the pandemic, participants across study arms mentioned restrictions in movement were imposed in most veredas, such as roadblocks for goods and materials as well as for visitors and locals who had to exit and enter periodically. When asked if they believed COVID-19 restrictions had affected security, overall, respondents were not able to establish a connection. Some participants initially responded in terms of health security and the fear they had to meet with neighbors, but did not refer to physical threats. In a very small number of cases, respondents mentioned that security conditions had slightly improved due to quarantine. One program staff member mentioned observing that the cultivation of illicit crops increased during quarantine, but was not explicit about the effect on security threats. 41 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV However, at the validation workshop some program staff were surprised about these responses and explained that in certain regions, like Bajo Cauca and Pacífico Nariñense, armed groups had installed their own measures to maintain public health, child recruitment had increased, and overall armed groups had expanded their control. As noted above, asking about the security situation repeatedly proved difficult; communities could have omitted this type of information out of fear. Program staff’s perception is in part corroborated by organizations like Human Rights Watch, which in June 2020 published a report on violent measures imposed by armed groups in various rural areas. 11 Likewise, the ICRC established in a recent publication that COVID-19 could have increased insecurity perceptions in rural communities due to the stricter and expanded control of armed groups.12 FINDING 20: COVID-19 affected education across municipalities. Limited access to technology impeded remote alternatives and there was a consistent perception that the pandemic had affected education quality. COVID-19’s impact on education All participants agreed COVID-19 limited education because remote solutions were not available for all students. As mentioned in Finding 8, treatment and control areas reported poor or no access to Internet services. The guides (printed copies of classes) that were provided as an alternative to virtual and in-person education were not effective either. Government officials across all regions at endline agreed that education was severely affected by the COVID-19 pandemic due to the shift to virtual school coupled with the fact that many communities had limited or no Internet access, and students did not have appropriate technology to access virtual school. Officials in Norte de Santander also noted that parents did not have sufficient schooling themselves to support their children in lessons, so families needed to find someone to explain lessons to their children. In addition to limited or no access, community members and leaders across all study arms mentioned COVID-19 confinements and the absence of in-class teaching affected school quality, student engagement, and in some cases increased drop-out rates. Participants indicated children were not receiving basic school instruction. Printed guides of classes were not an adequate alternative, as they did not provide nearly enough interaction between children and teachers and parents or other adults were unable to explain the guides. Overall, community members and leaders that were parents considered their children had not learned in the past two years. Some mentioned their children had to repeat the school year, or that they had dropped out completely. - How did COVID-19 affect education to children and young people in your community? - It affected them greatly because the guides that substitute classes are very difficult, myself as a mother I cannot understand them. It is very different when they can be in the classroom and a teacher can explain. I cannot explain those guides at home. – Community leader, Retrospective 2 treatment, Norte de Santander Parents’ productivity—particularly mothers—was affected by children staying home for school. Some community members and leaders described having to play multiple and simultaneous roles as workers, caretakers, and teachers. 11 Human Rights Watch, “Colombia: Brutales medidas de grupos armados contra COVID-19.” 12 ICRC, “Colombia: aislada por la pandemia y el conflicto.” 42 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV CONCLUSIONS AND RECOMMENDATIONS This section presents the evaluation’s conclusions and recommendations. The conclusions synthesize the evaluation findings with information from other available sources; they are organized by the same domains as the evaluation questions. The recommendations are based on findings and conclusions and reflect input from a virtual validation workshop with Colombia Transforma staff; they are organized by the actor corresponding to the recommendation. CONCLUSIONS State-society and intra-society relations 1. Findings indicate that the tripartite model results in positive impacts on state-society and intra￾society relations. These impacts are not universal across different dimensions; they are most apparent in trust and JAC-municipal relations, somewhat apparent in perceptions of transparency and understanding of the peace accord, but not in public service provision. Results seem to be more positive in areas with stronger doses of the intervention (longer project implementation periods, repeated projects in a community, regional-level critical mass, or a longer history of Colombia Transforma in the broader region), as indicated by a pattern of stronger results in retrospective treatment sites and more modest positive results in endline treatment sites. These effects are logical given the long history of mistrust and may indicate that in this context, repeated collaboration will be necessary in order to achieve the progress needed in these relations. 2. Perceptions of trust, transparency, and JAC-municipal relations seem to be closely tied, and these are not necessarily distinct concepts for community members. Trust and transparency both seem closely tied to knowing the mayor and having repeated interactions with the municipality. The relationship with the municipality is also heavily dependent on the individual mayor, as compared to the institution, and that person’s ability to deliver on campaign promises. Repeated interactions do seem to build understanding among community members of the resource constraints municipalities face, at least in some cases. Community capacity, empowerment, and ownership 3. Findings clearly suggest that the tripartite model’s strongest impacts are on community capacity, empowerment, and ownership of the local development projects implemented in the course of the Colombia Transforma program. While communities themselves do not always articulate the change that stems from participation in the tripartite model, this study identifies these changes clearly through comparison of treatment to control areas, which clearly captures the communities’ and JACs’ heightened scope of ambition in future project plans, fundraising, ability to manage projects, and lobbying of local government for support in treatment area communities. Community resilience and adaptability 4. Colombia Transforma hypothesizes that any link between the tripartite model and community resilience, adaptability, and civic engagement would be indirect, and findings confirm that this link 43 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV is difficult to detect. The fact that data did not indicate an effect on community resilience or adaptability is detectable at this point in time could reflect ongoing challenges—or even backsliding—in the security context, the possibility that the tripartite model or even communities are fundamentally unable to do much in the face of the security threats that these regions face, or the possibility that any indirect effects require more time or critical mass in order to be seen. Cross-cutting 5. While this evaluation shows the Colombia Transforma tripartite model to be effective in achieving a number of outcomes and many of these effects are seen across study arms, it also appears that areas where the program has been implemented for the longest, has achieved critical mass, and has used convenio solidarios show the highest results. Specifically, the longer duration of Colombia Transforma presence in Retrospective 2 areas in Catatumbo, Norte de Santander (as compared to treatment areas in other study arms) coincides with additional communication, dialogue, and collaboration among stakeholders, resulting in stronger success than other areas of Colombia Transforma implementation. However, attributing this effect to critical mass or use of a convenio solidario is complicated by other differences between the study arms, which offer competing explanations. First, the baseline/endline study arm had a compressed implementation timeframe for projects (less than a year) and treatment sites only experienced a single tripartite project (as compared to multiple projects grouped in neighboring veredas in retrospective communities); consequently, difference could also be due to dosage effects. Second, evidence also shows that while the Retrospective 2 treatment communities demonstrated higher absolute ratings on several dimensions, these dimensions were also higher in the Retrospective 2 control communities than those of other study arms; thus, the amount of relative difference between treatment and control is comparable to the other study arms, even though the Retrospective 2 communities appear more positive overall. This is to say, while the Retrospective 2 treatment communities show a higher level of dimensions like trust, transparency, and so on, the amount of difference between treatment and control is consistent with the other study arms because the Retrospective 2 control communities show a higher level than the baseline/endline control communities on these same dimensions. Finally, the longer history of programming, critical mass, and use of a convenio solidario cannot explain those instances in which Retrospective 1 communities showed higher assessments than baseline/endline communities. These caveats suggest that there are likely a variety of factors behind these differences; this may be a productive line of inquiry for future evaluations with larger samples stratified by tripartite model variations. Context 6. Colombia Transforma operates in an extremely challenging context in which violence is normalized, communities are largely left to fend for themselves, and organized crime shapes many aspects of daily life. COVID-19 appeared to be a more disruptive contextual factor during the first year of the pandemic, but now appears to be receding; while it can affect timelines and supplies, communities and municipalities have mostly found practical workarounds. Two exceptions seem to be in education and in the depth of interactions communities and municipalities experience during project implementation. 44 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV RECOMMENDATIONS RECOMMENDATIONS TO OTI COLOMBIA TRANSFORMA 3 (CT 3) · Build on the tripartite model’s powerful applied approach to capacity strengthening by providing communities with short accompanying tools that target specific functions such as administrative topics and managing security. These tools could help communities better appreciate the vital role that they are already playing, roles that are valued by other partners as essential to implementing community development projects. When designing the tools, consider that virtual techniques are still challenging for community members. · Consider programming strategies to extend the successes observed in Norte de Santander by teaching communities about concrete municipal administration processes such as petitioning the local government. Focus on processes that are tangible to community development projects in order to help communities better navigate the government apparatus. · To support spread of the tripartite model and build on momentum already observed by word￾of-mouth communication, consider targeted communication products that describe how to initiate a project using the model in lay terms. Audiences for these products may include communities, other local development actors such as civil society organizations and churches, and municipalities; the convenio solidario tool may also merit additional communication products. · For the third phase of Colombia Transforma (CT 3), ensure expectations are reasonable given the scope of community infrastructure projects in the context of decades of war, mistrust between communities and municipal government, and municipal government resources. Evaluation evidence shows that the tripartite model can have measurable effects, but state-society relations are starting from a low baseline. The tripartite model can help start the process of improvement, but the process will require sustained engagement over many years to counteract long-standing conflict and the rural sense of abandonment. · Consider better documenting or specifying approaches to initiating contact and communication between actors at the start of a potential tripartite model project · Recognize that building state-society relations between communities and municipalities is a long￾term process and build on evidence that suggests concrete mechanisms to leverage success from a first project for additional trust building. Evidence from the two retrospective study arms in this evaluation suggests that tripartite model projects that include longer time frames, repeated projects, concentration of projects within critical mass zones, or use of convenios solidarios result in more improvement in state-society relations than the condensed, one-off tripartite model that was necessary to use in the baseline/endline study arm. Sample sizes within the retrospective study arms were too small to distill which of these more in-depth elements may be most important and whether there are interaction effects across them, and the programming context is likely too complex to allow for a single “recipe for success”, but overall, these elements seem to be productive in extending gains. · To further support the relationship between communities and municipalities, look for more avenues that allow for building trust through collaboration. This may involve continuing to identify ways to reduce the prominence of OTI’s role so that communities see the municipality as the primary partner rather than Colombia Transforma. It could also involve adding planning and implementation activities that allow for a direct dialogue between communities and municipalities. 45 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · The program logic makes an explicit link between the peace accord and small infrastructure projects. The link between the infrastructure projects themselves and the peace accord seems better understood than the link between the collaborative implementation model and the peace accord. Colombia Transforma and its partners should ensure this latter link—the importance of the tripartite process and not just the product—is emphasized during implementation in order to achieve maximum value of these projects toward the longer-term goal of supporting a durable peace. · Continue to promote the convenio solidario as a tool within the broader collaborative tripartite model, including looking for ways to scale up the use of this mechanism and extend it to new communities. Given that use of these convenios has been limited to date (based on number of total tripartite model projects), CT 3 should monitor any scale up closely to track how benefits, risks, and results evolve with wider use of the tool. · As the convenio solidario mechanism builds momentum and Colombia Transforma generates more critical mass, there may be other innovations that can build on success of the base tripartite model, possibly through involving private enterprises or other charitable or civil society organizations. RECOMMENDATIONS TO USAID/COLOMBIA AND OTHER DEVELOPMENT ACTORS · Programs that seek to build trust in the state or governance structures should take note of the magnitude of success achieved by the tripartite model. The model has found a practical means to bring community leaders and local government officials together to collaborate on tangible projects that all parties can see. Positive first steps in building trust are of central importance to beginning longer-term sustainable development processes in Colombia’s most vulnerable communities; other projects can build on this model. · Other small infrastructure projects can replicate the tripartite model to ensure that projects result in positive social-political developments (such as trust or capacity to collaborate with the government), in addition to the completion of the infrastructure itself. Successful replication of the model can also be leveraged to help increase community-level demand for future collaborative development projects with local government. · Given the apparent added value of critical mass, word-of-mouth diffusion of the tripartite model, repeated small projects, extended engagement, and convenios solidarios, USAID/Colombia should consider how micro-targeting projects geographically and longer-term development strategies can build on momentum in the specific geographies of CT 1 and CT 2. · Colombia Transforma had been working for almost 6.5 years by the time Phase 2 of this evaluation was collecting data. New programs will take time to build trust with community and municipal government actors, particularly in areas where there has traditionally been mistrust of the U.S. Government. New programs should work with established trusted actors when entering new areas and should moderate expectations for results; a new program may not be able to achieve in its first years the results that Colombia Transforma has achieved in its most recent years. · At the same time, the speed of project implementation was something communities greatly appreciated in partnership with Colombia Transforma. If USAID/Colombia is to conduct other infrastructure or community development programs in these areas, it should proactively consider how to design projects for efficiency while achieving compliance objectives. 46 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · OTI branding is less prominent than that of other USAID programs. In spite of this, it continues to take great effort to position the municipal government as the communities’ long-term partner. Other programs will need to take care to ensure they are supporting the role of local government and not eclipsing it. RECOMMENDATIONS TO OTHER ACTORS · The Government of Colombia at all levels (national, departmental, and municipal) should involve JACs and other community organizations in the execution of community projects, both to strengthen these organizations’ capacity and to strengthen state-society relationships. Existing mechanisms to do this, such as convenios solidarios, have proven successful and would be a proven strategy to utilize. · Government of Colombia programs can learn lessons in how to operationalize collaborative programs, interventions, planning, and other processes from the tripartite model. This collaboration can be time-consuming, but the investment has the ability to strengthen relations. · To the greatest extent possible, utilize cooperative models for implementing the peace accord’s rural development initiatives in order to deliver not only on the letter of the agreements but the broader spirit of building durable peace. · Convenios solidarios are a practical way to achieve more community development projects within the context of limited municipal government staffing by devolving more of the project oversight to communities, while also building greater trust. Find ways to promulgate and diffuse this mechanism by ensuring municipalities are aware of how to initiate and implement a convenio solidario and ensuring cross-municipal sharing of lessons learned. · Municipal governments should return to in-person monitoring and visitation to communities as the COVID-19 pandemic continues to normalize. Remote monitoring techniques have provided practical means to ensure work continues, but do not carry the same value to communities that have felt abandoned. · Actors should seek to forge connections with other actors—government at all levels, civic and faith-based, private, and international development—and to communities in order to strengthen networks working in collaboration toward development goals. · Future evaluations should take note of the aspects of this evaluation’s design that made it possible to detect outcomes in a complex environment typified by security concerns, a history of mistrust, and where community respondents themselves do not necessarily point explicitly to patterns of change attributed to Colombia Transform and its tripartite model. Two specific factors were particularly important in this evaluation’s design, and without these, many of the results above would not have been detectable: o Use of matched control sites that were comparable to intervention sites on several key socio-political factors, including being within the same municipality. o Robust use of qualitative and open-ended questions. The terminology of the democratization, rights, and governance field is not a perfect overlap with the terms in which communities think about the peace accord and community development. For example, the concepts of trust and transparency were closely linked for community respondents. Qualitative techniques helped this evaluation build robust thick understanding of meaning across actors from different backgrounds, and capture substantial nuance that closed-ended questions could miss. This is not to suggest that 47 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV quantitative methods cannot be used successfully in future evaluations, but they should be intentionally paired with qualitative tools and any survey work seeking to generalize quantitative results at population-levels will require care and extensive instrument validation processes. · The evaluation team recommends the following for any future evaluation of the Colombia Transforma program: o For results that will be measured quantitatively, community leaders will be the most efficient level of measurement to assess and detect results; that is, quantitative tools should target community leaders, as this is where detectable results are clearest and where quantitative tools are most likely to achieve strong reliability and validity. o A full probability sample could be achieved among community leaders, as a discrete sampling frame could be compiled for probability-based sampling at the second stage. This would require developing a comprehensive list of JAC leaders for each community sampled at the first stage, which the evaluation team feels is feasible based on its experience here. Quantitative data from this evaluation could be used to establish an intra-class correlation measure, for purposes of establishing efficient first and second stage sampling allocation. In contrast, trying to achieve a full probability sample among community members would likely require a household listing exercise prior to data collection, which would be costly and may prove infeasible in the security context. o By extension, the evaluation team recommends that the value of data provided by community members is strongest with qualitative methods where interviewers can probe, and extended back-and-forth allows for construction of thick understanding. Focus groups also show the points of contrast and consensus between community members. o A sequential mixed-methods design, whereby quantitative data are collected first and followed by a separate second round of qualitative data collection, could be a powerful design that yields even richer qualitative data. Such sequential designs allow open-ended questions that probe emergent findings suggested by quantitative analysis. 48 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV REFERENCES Centro Nacional de Memoria Histórica. 2013. ¡Basta Ya! Colombia: Memorias de Guerra y Dignidad. Bogotá. “A Fight by Other Means: Keeping the Peace with Colombia’s FARC.” International Crisis Group, (November 2021): 2, 16–17. Human Rights Watch. “Colombia: Brutales medidas de grupos armados contra COVID-19.” ICRC. “Armed Conflict in Colombia: a Pain that Doesn’t Go Away.” International Committee of the Red Cross. ICRC. “Colombia: aislada por la pandemia y el conflicto.” International Committee of the Red Cross. Fundación Ideas para la Paz. Romero, Simon. 2008. “Two Colombias, at War and at Peace.” The New York Times, September 7, 2009. Social Impact. “Colombia Transforma Program Mid-term Performance Evaluation.” May 2018. 49 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEXES 50 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX A: SUMMARY OF RESULTS BY STUDY ARM Baseline Endline Retrospective 1 Retrospective 2 Domain Dimension Baseline Control Baseline Treatment Baseline Treatment - Control Balance Endline Control Endline Treatment Endline Treatment - Control Difference Retro 1 (Both Treatment and Control) Retro 1 Treatment - Control Difference Retro 2 Control Retro 2 Treatment Retro 2 Treatment - Control Difference State-society and Intra￾society Relations 1. Trust in municipal government Low Low No difference Very low Low Treatment higher Medium￾high Treatment higher Medium Medium￾High Treatment higher 2. Municipal transparency Low Low No difference Low Low No difference Medium￾high Treatment higher Medium Medium￾High Treatment higher 3. JAC-municipal relations and municipal outreach to communities Very low Low Treatment higher Medium￾High Treatment higher Low Medium￾High Treatment higher 4. Public service provision Very low Very low No difference Very low Very low No difference Low No difference Low Low No difference 5. Peace accord understanding Low Low No difference Low Medium Treatment higher Low No difference Low Medium Treatment higher Community Capacity, Empowerment, and Ownership 6. Community project management Medium￾High Medium￾High No difference Medium Medium￾High Treatment higher Medium￾High Treatment higher Medium￾High High Treatment higher 7. Role of the JAC Medium Medium￾High Treatment higher Medium￾High Treatment higher Medium High Treatment higher 8. Project funding Medium￾High Medium￾High No difference Medium Medium￾High Treatment higher Low Medium￾High Treatment higher 9. Project ownership Low Low No difference Low Medium￾High Treatment higher Low Medium￾High Treatment higher 10. JAC Project Initiative Medium￾High Medium￾High No difference Low Medium￾High Treatment higher Medium￾High High Treatment higher Community Resilience and Adaptability 11. Ability to manage insecurity Varied Varied No difference Low Low No difference Medium￾High No difference Low Medium Treatment higher 12. Resilience Varied Varied No difference Medium Medium No difference Medium￾High No difference Medium Medium No difference 13. Civic engagement Limited (voting) Limited (voting) No difference Limited (Voting) Limited (Voting) No difference Limited (voting) No difference Limited (Voting) Limited (Voting) No difference 51 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX B: DETAILED METHODOLOGY This annex provides additional detail regarding the evaluation methodology. EVOLUTION OF THE EVALUATION DESIGN The initial concept for this evaluation emerged from a monitoring and evaluation field visit early in the implementation of CT 2, in August-September 2019. The report notes that the tripartite model provides a unique opportunity for rigorous evaluation within the OTI context where complex environments mean programming models must constantly adapt to incorporate emergent practices. Based on this motivation, OTI initially developed a scope for a full quasi-experimental evaluation that would be based on three measurement phases (baseline, midline, and endline). This quasi-experimental evaluation was to be based on matched treatment and control sites and quantitative tools, enabling a difference-in￾difference model to estimate the tripartite model’s impact. Based on this scope, OTI commissioned EnCompass LLC with subcontractor Social Impact to conduct the evaluation. EnCompass led a participatory design process to develop a full design based on the scope, while Social Impact supported development of the sample by testing four propensity score matching and coarsened exact matching models for pairing a balanced sample of treatment and control veredas. This work had progressed substantially by March 2020, with completion of a balanced stage 1 sample covering 64 veredas (32 treatment and 32 control). At this point, however, the novel COVID-19 virus became a global pandemic. The ensuing widespread closures, travel-restrictions, and transition to remote collaboration to maintain social distancing were particularly pronounced in the pandemic’s early months and presented insurmountable obstacles to conducting a broad population-based quasi￾experimental evaluation. In May, 2020, OTI and EnCompass decided to redesign the study to ensure some form of evaluation could occur. The most fundamental revision was a substantial reduction in the sample size, from 64 veredas to just 20 (during evaluation Phase 2, this would later be increased to the final stage 1 sample size of 28 veredas). Data collection methods were also converted to more qualitative techniques. A retrospective study arm—focused on CT 1—was added to complement the baseline/endline study arm and would help provide data from the pre-COVID-19 era. Finally, the midline phase was removed from EnCompass’ scope and converted to a much smaller internal evaluation. Several aspects of the original design were maintained and can be seen in design elements discussed in detail below. Most importantly, the pairing of treatment and comparison veredas helped maintain a counterfactual, albeit at a smaller scale. The coarsened exact matching model used to identify the original 64 stage 1 sample sites provided the final selection of the baseline/endline sample sites (all 12 sample veredas were matched pairs from the original sample of 64). Respondent categories remained consistent, while the stage 2 sampling was relaxed to a purposive approach. Ultimately, this redesign took substantial time, meaning the evaluation Phase 1 data collection only began in December 2020, a year and a half into CT 2 implementation. Particularly during the first phase, logistics for in-person data collection were time-consuming, requiring public health permissions for each municipality visited. The effects of these challenges are discussed in detail in the limitations section below. Throughout the rest of this report, the methodology presented refers to the final achieved methodology. 52 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV SAMPLING STAGE 1 SAMPLING FRAME (SITE SELECTION) Across all components and phases, the sampling frame for intervention sites was restricted to communities where the tripartite model was used (or planned to be used in the case of baseline). The sampling frame of comparison sites for each evaluation component consisted of all non-intervention veredas in the same municipality as the intervention site to which the comparison site was matched; this held the municipal government constant between treatment and comparison sites, which was important given the role of the municipal government in the tripartite model. The universe of relevant intervention sites that made up the sampling frame for each study arm varied by study arm. For the baseline/endline study arm, the sampling frame consisted of planned CT 2 intervention veredas where infrastructure projects had not yet begun; this enabled baseline measurement to occur pre-intervention. For Retrospective 1, the sampling frame consisted of CT 1 veredas that had used the tripartite model to implement their small infrastructure projects (bearing in mind that the model emerged through trial and error during CT 1 and thus a smaller subset of sites had used the model); prior to Retrospective I data collection, OTI confirmed that all sampled treatment sites received the tripartite model (particularly in Catatumbo and Putumayo). For Retrospective 2, the sampling frame consisted of CT 2 veredas that utilized a convenio solidario or fell within critical mass zones identified by OTI. In the final Retrospective 2 sample, all four treatment sites had used a convenio solidario and two fell within critical mass zones. For all three study arms, OTI provided the evaluation team lists of intervention sites, restricted as appropriate based on these sampling frames; the evaluation team then selected the sample sites (veredas). Due to security realities and the need to maintain comparison sites as non-intervention zones, the OTI implementation team was aware of the sites included in this evaluation. DETERMINATION OF CT 2 INTERVENTION SITES (BASELINE/ENDLINE STUDY ARM) Prior to evaluation Phase 1, program teams in each of the three CT 2 departments identified their likely geographic targets (veredas where they had not worked before, but that were included in their regional strategies). The program teams coded potential sites one through four, with 1 as the most probable to have an intervention. While utilization of the tripartite model requires willingness of local actors to use the shared implementation approach, OTI targets communities based on need—those most affected by violence and conflict. Thus, while use of the tripartite model involves some degree of self-selection, participation in CT programming overall involves “hardest case” scenarios, sometimes referred to as a Sinatra test (if it can work here, it can work anywhere). MATCHING COMPARISON SITES (COUNTERFACTUAL CONSTRUCTION) Baseline/endline study arm: Following the evaluation team’s selection of a random set of intervention sample sites from the sampling frame for the baseline/endline study arm, statistical matching identified a comparable set of control veredas. The control veredas would not receive CT programming and were as similar as possible on average (“balanced”) across key characteristics to the treatment veredas. Because random assignment of the CT program was not feasible, the evaluation team used a quasi-experimental approach, constructing the counterfactual group using coarsened exact matching (CEM). As part of this process, the evaluation team tested two CEM models and two propensity score matching models to determine which model produced the most balanced set of sites; the CEM 53 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV technique produced the most balanced model. Models drew on a total of 21 variables, which were selected based on (a) their ability to predict treatment status or selection into the treatment group and (b) the extent to which they were significant predictors of the outcomes of interest. Although there is limited data available at the vereda level in Colombia, the team was able to compile a substantial number of key vereda indicators to conduct the matching exercise. The variables used included: demographic, community participation, and historical victimization (displacement) indicators from the 2013–2014 Agricultural Census (DANE); coca-related indicators from UNODC; past USAID/OTI programming; Zonas Futuro designations; and geographic indicators (IGAC). For Tumaco, a qualitative matching approach was used since data limitations impeded the application of a statistical matching algorithm. As a result, the evaluation team manually selected matched veredas pairs across different geographic sub-regions in Tumaco for greater representativeness of the municipality. This statistical approach was designed in consultation with the program team to incorporate the key decision criteria and variables described above that were used to select the intervention sites. In one case where security conditions inhibited site visits, the evaluation team replaced the selected site with a new control vereda from the master list of pairs produced by the model; see Exhibit 14 for detail. FINAL SAMPLE SITES BY STUDY ARM The following table provides the final set of data collection sites, with additional detail for each. The table notes where substitutions were made in the course of data collection, noting the reason for each. In each case of substitution, the evaluation team confirmed the substitution did not affect suitability as a treatment or control site before the substitution was made; in the case of treatment sites, substitutions were restricted to the original sampling frame. 54 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Exhibit 14: Data collection sites Region, department (Evaluation Phase) Municipality Vereda Treatment/control Putumayo (Retrospective 1) Orito Arauca Treatment Orito Acea Control Valle del Guamuez El Cairo Treatment Valle del Guamuez Nueva Palestina Control Catatumbo, Norte de Santander (Retrospective 1) Teorama Asserío Treatment Teorama Bejuco Control Tibú Km 60 Treatment Tibú Pueblo Nuevo Control Catatumbo, Norte de Santander (Retrospective 2) Teorama El Diviso13 Treatment Teorama San Jose de las Vegas Control Tibú Campo Raya Bajo Treatment Tibú Vetas Central14 Control Sardinata Jerico Treatment Sardinata Santa Maria Control El Carmen Santa Ines Treatment El Carmen Culebrita15 Control Catatumbo, Norte de Santander (Baseline and Endline) Teorama San Juancito Treatment Teorama Mata de Tilo Control Tibú Santa Cruz Treatment Tibú San Marcos Control Bajo Cauca, Antioquia (Baseline and Endline) Tarazá Purí Treatment Tarazá El Rayo / San Antonio16 Control Caucasia Cacerí Treatment Caucasia La Corcobada Control Nariño (Baseline and Endline) Tumaco Pital Piragua Treatment Tumaco Chimbuzal Control Tumaco Inguapí del Guadual Treatment Tumaco Brisas del Acueducto Control MIDLINE REVIEW In addition to evaluation Phases 1 and 2 described in the body of the report above, the OTI program team conducted a midline review in-house between July and August 2021. The midline targeted a small sample of program staff, local authorities, and community partners, with a total sample of 19 interviews 13 Community leaders in the original selection declined to participate. 14 The original selection, Caño Indio, was replaced based on security. 15 The original selection, Culebra, was replaced when local authorities provided contact details for Culebrita. 16 El Rayo was sampled at baseline. Due to security concerns, at endline El Rayo was replaced with San Antonio. San Antonio was selected from the original list of matched control veredas used at baseline (produced via the statistical procedure described above), thus seeking to maintain the sample’s balance. There were three potential sites in the matched list from baseline; San Antonio was manually selected as the closest match based on the same 21 variables used in the original statistical CEM model. 55 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV plus one working group session. The data collection focused on the changes to the tripartite model during the COVID-19 pandemic, challenges and achievements, the use of a convenio solidario in tripartite model implementation, and improvement opportunities. The aim of the midline review was to assess whether and how the implementation model had varied as a result of the pandemic. The primary data from the midline review were not analyzed by the evaluation team that produced this report and as such these data does not directly support findings above. However, the results from the midline review supported minor design refinements before endline and interpretation during the validation workshop for this evaluation by providing context on how implementation of the program has progressed in different settings. DATA COLLECTION PROCEDURES,TRAINING, AND ETHICS EnCompass contracted Cifras y Conceptos, a Colombian research and evaluation firm, for the primary source data collection. Before data collection, the evaluation team worked with Cifras y Conceptos to develop enumerator training, a fieldwork manual, and associated data collection protocols; the evaluation team and Cifras y Conceptos also collaborated to conduct pilots of the data collection instruments prior to both evaluation phases. These steps helped ensure data collection staff were thoroughly trained on the details of each instrument, ethical conduct appropriate for the study population, and data collection best practices, as well as validate the tools. The team lead and evaluation specialists engaged with Cifras y Conceptos extensively in the instrument refining; Cifras y Conceptos also advised regarding the sampling design, particularly the security dynamics of and reachability of each site. The Colombia-based subject matter expert accompanied interviewers on a site visit to pilot the instruments and procedures at evaluation Phase 1 and participated in the data collector training for both phases to provide additional in-person support and ensure the quality of the interviews and focus groups. During fieldwork for both phases, the evaluation team provided ongoing quality assurance and technical support to Cifras y Conceptos to ensure adherence to protocols, including multiple data quality reviews during data collection. All sample adjustments described in Exhibit 14 were made directly by the evaluation team in consultation with Cifras y Conceptos and in coordination with Colombia Transforma. HUMAN RESEARCH SUBJECT ETHICS The evaluation team applied the highest ethical standards to ensure no individuals experienced unnecessary risk or undue harm as a result of their involvement in data collection processes; the evaluation team achieved these standards by using a multi-pronged strategy. Approval from Institutional Review Board (IRB): During Phase 1, the evaluation team submitted the evaluation design to EnCompass’ IRB to ensure the application of strong methodological and ethical standards. The evaluation team submitted the updated methodology, field guide, data collection instruments, informed consent procedures, security protocols, and other relevant documents to the IRB committee for review prior to the start of evaluation Phase 2 data collection, consistent with the committee’s requirement for annual review of all ongoing studies and anytime protocols are updated. In the dynamic security and COVID-19 context in which data collection occurred, the evaluation team utilized incident reporting procedures and maintained continuous contact with the IRB chair to assess changes in risk to evaluation participants. Initial contact and sampling for respondents: The evaluation team worked closely with OTI program staff and Cifras y Conceptos to establish appropriate and effective procedures for reaching out 56 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV to potential data collection participants. In any case where there was concern that data collection could put a particular participant at a higher than minimal risk that could not be mitigated, the evaluation team coordinated with OTI to determine another method for collecting this information or select another individual to interview. This dynamic, iterative process of assessing and mitigating risk occurred in parallel to broader coordination to mitigate security concerns and will continue throughout data collection. Data collectors’ safety: Protocols included procedures for the personal information data collectors share with participants, biosecurity, and physical security; enumerators had frequent, agreed-upon times for check-ins and easy access to SOS contact information. Procedures specified that in any potential safety or security incident (including missing scheduled check-ins), the data collection coordinators contacted the EnCompass headquarters evaluation team immediately, which would activate EnCompass’ Emergency Action Committee as appropriate. The evaluation team and Cifras y Conceptos also developed biosecurity protocols specific to COVID-19 to ensure data collectors and participants’ health and to mitigate the risk of the spread of COVID-19. Safety plans included clear communication of the risk of the spread of COVID-19 and took into account local regulations and as well as the evolving recommendations of the US Centers for Disease Control and World Health Organization. Secure data management: The evaluation team put into place the following data security protocols at the onset evaluation Phase 1 in line with EnCompass procedures and in coordination with the IRB: · Restricting data access to Cifras y Conceptos and the direct evaluation team (not sharing any raw data); following EnCompass’ confirmation of data receipt, Cifras y Conceptos destroyed all records of data. EnCompass maintains primary source data in secure spaces with access restricted to direct evaluation team members.17 · Removing identifying information and storing respondents’ information in a separate locked file prior to coding qualitative data. · Using secure communication channels as requested by the respondent; these channels were subject to the respondent’s preferences. For virtual interviews, this involved secure communication platforms; for in-person interviews, this involved allowing respondents alternate locations for conducting interviews and focus groups. · Using encryption for all data-transfer, analysis (coding), and storage. All raw or sensitive information is maintained on an encrypted server with restricted access. · Destroying all evaluation data three years after the completion of the evaluation. When permitted by the interviewee(s), data from KIIs and FGDs were recorded on local (offline) audio files using unique numeric identifiers that do not include personally identifiable information; audio files allowed for accurate transcription of responses after data collection. Where the interviewee did not 17 As agreed with OTI, during evaluation Phase 2 EnCompass shared a small sample of anonymized excerpts to each questionnaire item with the OTI Monitoring and Evaluation Advisor to validate the findings; the anonymization process protected the confidentiality of evaluation participants’ responses. This sample covered 10 percent of total interviews and was shared through a secure platform. The purpose of this sharing was to enrich the analysis process, allowing the Monitoring and Evaluation Advisor to identify any themes that were not apparent to the evaluation team; ultimately, this process did not identify additional themes. These data were formatted as a compilation of responses per question to minimize possible identification of individual respondents. 57 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV permit audio recording, notes were to be taken in a local (offline) file; in the end, there were no respondents who withheld consent to audio record in either Phase. The data collection firm transcribed data using Word templates. Completed and cleaned transcriptions and audio files were stored in an encrypted folder managed by EnCompass on a secure platform until checked by headquarters’ evaluation team, then moved to a restricted SharePoint folder at EnCompass for analysis. Files on the shared space were then deleted. The data collection firm also destroyed all hard and electronic copies of notes, transcriptions, and audio files after EnCompass confirmed receipt of these files. All transcripts were assigned unique numeric identifiers in place of personally identifying information on the forms. The evaluation team maintained a master key in an encrypted file at EnCompass. Consultants who worked on this evaluation will not be able to access this folder at any point after the close of the evaluation; EnCompass staff evaluation team members will be able to access the data only if justifiable need arises. Data access and restrictions are administered by EnCompass’ IT team. If further data disposal measures are necessary, the evaluation team will develop them in consultation with OTI.   The above processes are important to ensure that respondents feel comfortable to offer clear and honest responses given the sensitive nature of the subject, and to ensure that risks to participants associated with data breaches are minimized to the fullest extent possible. The evaluation report itself uses broad categories to attribute quotes; these categories were designed to give the reader context of the respondent’s position vis-à-vis the program while being broad enough to prevent any potential identification of the respondent or even the sample site. In limited cases where perspectives attributed to a specific regional OTI program team could be identifying, the evaluation team checked the report’s language with OTI to ensure potential association of the response with an individual respondent would not place that individual at risk. EVALUATION STRENGTHS AND LIMITATIONS STRENGTHS This evaluation used collaborative and participatory approaches throughout to ensure the final results met the needs of the program’s key stakeholders. Engagement with program staff during evaluation design ensured an ethical, safe, and rigorous approach to sampling and targeted evaluation questions. Coordination during data collection helped ensure participants’ security, efficient execution of data collection, and that adjustments to the sample and tools aligned with the evaluation’s purpose. During reporting, the evaluation team intentionally structured internal data synthesis processes to check potential biases. Collaboration with the Colombia Transforma team, meanwhile, helped validate findings and co-create conclusions and recommendations while the evaluation team ensured that findings, conclusions, and recommendations stayed true to the data; this process ensured accurate data interpretation, contextual awareness, and relevancy to stakeholders. The evaluation benefited from deep local expertise, as data collectors were based in the regions of the sample sites and were familiar with local linguistic and cultural nuances. This helped foster trust among participants during data collection. Additionally, a Colombia subject matter expert in the peace accord process who was also familiar with the sampled regions was involved in each step, from tool design to analysis and reporting. 58 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV While the evaluation sample is relatively small, having reached just over 5 percent of tripartite model projects and over 600 respondents, the sample is relatively robust for an evaluation based primarily on qualitative methodologies. Matched comparison veredas and the use of a baseline before intervention further bolsters the evaluation design and enables attribution in cases where clear patterns emerge across treatment and comparison sites. Data collection instruments contain diverse types of questions, including some closed-ended response formats that yield quantitative indicators that are able to capture clearly trends over time and between treatment and comparison groups. Through triangulation with qualitative results produced using both deductive and inductive analysis techniques, the mixed-methods approach provides multiple checks that help verify results. Finally, the evaluation reached diverse participants to capture distinct viewpoints, including community members, community leaders, mayoral staff, and program staff. The evaluation takes a broad approach to the topics covered—ranging from collective action issues to pandemic conditions to attitudes about the peace process—and phrased questions to elicit insights not just about indicators but also about social and political processes. LIMITATIONS AND RISKS As with all evaluations, this evaluation has risks and limitations. Most risks related to the safety and well￾being of beneficiaries, participants, and other stakeholder and are discussed above. In addition to the security risks, there are also methodological limitations; the primary limitations, along with practical implications for the findings and mitigations measures follow. Selection bias due to purposive respondent sampling (stage 2 sampling): As with all purposeful sampling approaches, there is always a risk that the selected respondents may not be representative of the larger universe, thus making generalizability of the results a challenge. While a participatory approach can promote trust and garner insider perspectives, it can also be subject to possible selection bias. Use of random sampling for the sites in the first sampling stage and rigorously defined comparison sites both help mitigate selection bias, while the large overall sample (over 300 interview and focus group events across both phases reaching over 600 respondents) mitigates this risk. Because participation in the tripartite model is premised upon mayors’ and communities’ willingness to engage in a participatory community development process, there may also be a self-selection bias that influences who opts in to the program. Selection of control veredas from the same municipality mitigates this potential source of selection bias. Finally, accessibility of veredas due to geographical and security constraints conditioned the stage 1 (vereda) sampling. As documented in the footnotes to Exhibit 14, ultimately four of 28 randomly selected sites needed to be replaced over the course of both evaluation phases. While this 14 percent replacement rate is somewhat higher than desirable, three of the four replacements occurred in comparison sites where a comparable match was identified. This also means that the replacement rate among treatment sites was a more modest seven percent. Recall bias: Some of the data collection tools ask respondents about past events (e.g., projects that have ended). There is a possibility of recall bias where the respondents may not remember previous experiences or omit details. The evaluation team tried to mitigate recall bias by attempting to identify those respondents who self-select regarding their ability to accurately recall their involvement with the relevant projects. In addition, the team probed participants by referencing commonly remembered events in the data collection instruments, where appropriate. The risk of recall bias is most pronounced with the retrospective study arms; for the baseline/endline study arm, Phase 2 data collection occurred just weeks after the conclusion of the OTI-supported small infrastructure project, which helped mitigate this potential bias in that study arm. 59 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV Response bias: The evaluation team anticipated that social desirability bias could be a challenge for this study. Response bias occurs when respondents feel pressure to give answers that are socially acceptable. The use of participatory approaches helped mitigate the potential for response bias because a comfortable and secure interview environment should facilitate responses that are as candid as possible under the circumstances. This potential bias is further mitigated through the use of data triangulation and independent verification of project implementation results where possible. At both evaluation phases, results showed that most respondents were candid and did not hold back from criticizing public officials, local conditions, or past programming. Limited access to key stakeholders: The evaluation team anticipated that access to key stakeholders, including beneficiaries, who would provide information to address evaluation questions could also be a challenge. Limited access could have arisen due to lack of, security conditions, or challenges related to COVID-19. To facilitate access, the evaluation team collaborated with OTI and established trust with stakeholder groups through local partners and leaders. Other security measures, including data security and comprehensive consent statements in Spanish, also helped mitigate these challenges. For the baseline/endline study arm, repeated interaction with a consistent data collection team across the two study phases further helped build trust. Finally, data collection was flexible with modes of data collection (virtual and in-person, including deferring to respondents in terms of platform or location). Ultimately, the evaluation team did not find access limitations to be a limitation, but cannot know the perspectives of individuals and veredas not sampled. Evaluation challenges related to the COVID-19 pandemic: The COVID-19 crisis posed significant challenges throughout this evaluation, severely curtailing both international and domestic travel and forcing an adjustment to the original design. Although restricted travel prevented evaluation team members based outside Colombia from conducting in-person training with the data collection team, the evaluation team adjusted the approach to allow for remote and virtual training and monitoring of the data collection activities, supplemented by an in-country consultant who supported data collection monitoring activities. The evaluation team also supplemented in-country data collection with virtual meetings. EnCompass brought 20 years of best practices working in virtual spaces and a state-of￾the-art virtual instructor-led training practice that supported these adjustments, including conducting highly participatory workshops during the design, finding validation, and results sharing stages. Additionally, the delay of baseline data collection due to the evaluation’s redesign (necessitated by the onset of the pandemic) meant that though baseline data were collected prior to CT 2 implementation in sample veredas for the baseline/endline study arm, the interval between baseline and endline data collection was compressed. This may have limited the ability to detect program effects following implementation. Additionally, the compressed timeline meant that CT 2 only completed one infrastructure project in each of the treatment veredas between the baseline and endline as opposed to more than one project in the specific vereda or vicinity, as is standard in the model.18 The risk is an underestimation of CT 2 outcomes. Changed programming context due to COVID-19: The pandemic and associated public health policy responses have greatly hobbled development processes in some communities. Communities have generally faced worsening economic conditions and greater challenges in cooperating on projects and acting collectively. In some locations, both mayors and municipal staff have been impacted by COVID-19 18 CT 2 is implementing numerous projects per site in other areas in Narino, Bajo Cauca, and Catatumbo from 2019–2021 (outside treatment and control areas), in comparison to the single-project intervention studied in the baseline and endline sample. 60 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV and reduced their interactions with veredas. JAC leaders in some veredas have also stepped up to play important roles in managing COVID-19 risks for their communities. Person-to-person communication is fundamental for planning and implementing development projects. More difficult communication and possible reductions in available donations and resources to dedicate to projects could dampen citizen buy-in or tarnish perceptions of local government effectiveness. The pandemic conditions therefore required several adaptations to Colombia Transforma program planning and implementation procedures. These challenges and changes also mean that the CT 2 procedures are a departure from the standard tripartite methods that were used in CT 1 and call for a moderation of expectations for tripartite model outcomes. The inclusion of the Retrospective 1 study arm helps mitigate this challenge by detecting changes that have occurred since the onset of COVID-19, but the overall risk remains of an underestimation of the tripartite model’s full benefits. On the other hand, the pandemic provides the opportunity of a real-life laboratory to observe how the model withstands the tremendous social￾economic and health pressures of a once-in-a-century pandemic. 61 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX C: BACKGROUND ON EVALUATION TEAM Zachariah Falconer-Stout, Team Lead: Zachariah Falconer-Stout is an international development professional with over 12 years of experience, primarily leading monitoring, evaluation, and learning to assist government, non-profit, and foundation clients in making evidence-informed decisions. He has been lead author on over 20 evaluation reports and technical resources, facilitated over 700 hours of workshops in English, Spanish, and Romanian, and conducted over 200 focus groups and interviews, with audiences ranging from village collectives to boards of directors. Zachariah has worked extensively in Africa, Asia, Europe, and Latin America, and since 2015 has lived in Central and South America. His technical focus is in democratization and governance as well as youth and education, with additional experience in health and gender; his methodological training and experience covers both quantitative and qualitative techniques, with a specialty in participatory and mixed-methods evaluations. Cecilia Papariello, Senior Evaluation Specialist: Cecilia Banks Papariello is a Senior MEL Specialist with the EnCompass Technical Assistance and Evaluation team, leading and supporting evaluation design, management, and implementation. A seasoned development practitioner with over 15 years of experience, Cecilia brings extensive monitoring, evaluation, learning, and project management expertise with a particular focus on human rights and democracy and governance. In addition to serving as a youth and families Peace Corps volunteer in Ecuador, Cecilia has worked as an evaluator, project manager, and MEL specialist with NGOs and consulting firms on projects and assignments in LAC, MENA, Sub￾Saharan Africa, and Asia. She holds a MS in Development Management from American University and received her BA in Psychology from the College of William and Mary. Cecilia speaks Spanish and French. Gina Cabarcas, Subject Matter Expert: Gina Cabarcas is the Co-Founder and senior investigator of Laboratorio de Justicia y Política Criminal. Expert in criminal justice and social research of grave human rights and humanitarian law violations. Gina was part of Colombia’s Historical Memory Group and led multiple investigation teams in Colombia’s Prosecution Office and the Unit for the Search of Disappeared Persons. Gina was a member of Releg (Latin American Network for Gender-based Strategic Litigation) and of Justice Rapid Response roster of experts in international criminal investigations. Ghazia Aslam, Technical Advisor: Ghazia Aslam has more than 12 years of experience designing, conducting, and leading qualitative, quantitative, and mixed-methods research and evaluation. She has deep expertise in conflict, fragility, and political transitions, and has led evaluations, fragility assessments, and conflict analyses in Indonesia, Pakistan, Somalia, and Yemen. She is a co-author of the flagship report, Societal Dynamics and Fragility, which forms the basis of the World Bank’s analytical work in fragile and conflict-affected contexts and continues to provide guidance to the World Bank’s Fragility, Conflict and Violence team. Dr. Aslam has conducted and supported evaluations of foreign assistance programs in several restrictive environments, including in Egypt, Indonesia, Nigeria, Pakistan, and Yemen on issues regarding fragility, conflict, marginalized populations, and political transitions. She also provided technical assistance and advisory services in design of development programs in post-conflict contexts to strengthen democratic governance and build peace. She taught at Lahore University of Management Sciences, Pakistan and George Washington University in Washington, D.C. Cifras y Conceptos, Data Collection Firm: Cifras & Conceptos, S.A. is a research consulting firm founded in February 2007 that offers advisory services in monitoring, data collection, information, statistics and indicators. Through consulting and technical assistance, Cifras & Conceptos seeks to improve the use of information by organizations. Cifras & Conceptos central purpose is to convert data into intelligent information that supports decision-making, facilitates the control, monitoring and evaluation of policies and processes, optimizing the products of each organization. The firm has a qualified group of professionals from specializing in various areas that make up the team. 62 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX D: STATEMENTS OF DIFFERENCE There were no differences of opinion among the evaluation team members, funder, or implementer. 63 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX E: CONFLICT OF INTEREST DISCLOSURES Disclosure Form: Real or Potential Conflict of Interest for USAID Activities Evaluations, assessments, monitoring, or learning activities related to USAID projects and/or activities will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest. For the types of activities noted in sentence one, all team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project work to be completed. Evaluators, monitoring specialists, and learning specialists of USAID projects/activities have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the team or require recusal by that individual from conducting certain aspects of the project(s). In addition, if team members gain access to proprietary information of other companies in the process of conducting the work, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated, assessed, reviewed, or monitored or the implementing organization(s) whose project(s) are being engaged. 2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated, assessed, reviewed, or monitored or in the outcome of the work. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, assessed, reviewed, or monitored, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the work or the implementing organization(s) whose project(s) are being evaluated, assessed, reviewed, or monitored. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated, assessed, reviewed, or monitored. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated, assessed, reviewed, or monitored that could bias the work. 64 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV 65 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV 66 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV 67 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV 68 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX F: EVALUATION SCOPE OF WORK Colombia Transforma Tripartite Model: Retrospective Analysis, Baseline and Endline Data Collection SOW DRAFT: This SOW has been updated to include all phases of the Colombia Tripartite Model Evaluation, led by Encompass under TO 7200AA19F0017. OTI plans to issue a separate activity covering the endline and final report, however this scope of work is inclusive of both activities and updates to the tasks and deliverables under Activity 1 as well as new tasks and deliverables tied to Activity 2. Each activity will have separate budgets. Activity 1: Retrospective Evaluation and Baseline: · Phase I - Inception phase · Phase II - Data collection for the Retrospective Evaluation and the Baseline · Phase III - Documentation and Reporting of Retrospective Evaluation and Baseline Activity 2: Endline and Final Report: · Phase IV – Endline planning design refinement phase · Phase V – Endline Data Collection · Phase VI – Documentation and Reporting of Endline and Final Report) PERIOD OF PERFORMANCE: January 2020–February 2022 BACKGROUND For over half a century, Colombia has experienced an intractable armed conflict against insurgency groups, resulting in 220,000 deaths and over 7 million registered victims. At the conflict’s core are state weakness and structural inequities that result in two Colombias: a modern, urban one and a poor, neglected rural Colombia where most of the conflict has taken place. After four years of negotiations, on Nov. 30, 2016, the Colombian government ratified a final peace accord with the largest guerilla insurgency group, the Revolutionary Armed Forces of Colombia (FARC-EP). The FARC-EP has demobilized, and the government is enacting wide-ranging reforms related to rural development, inclusive political participation, transitional justice for victims, and reduction of illicit crops, like coca. It will take a decade or more to fully implement these reforms. However, the initial years of implementation may be the most critical. The long-term success of the accords will depend on the extent to which Colombians begin to experience immediate changes brought by peace, particularly those who have suffered the most from the conflict. In July 2015, USAID/OTI awarded a task order to Management Systems International (MSI) for the implementation of the first phase of the Colombia Transforma (CT) project with the goal of supporting the critical first 36 months of peace accord implementation. The program carried out more than 600 small activities all over the country with the goal of supporting government entities in quickly delivering on the peace accord’s promises, creating connections between government at all levels and communities, and working with community-level organizations to increase “bottom-up” capacity to create peace. In support of these objectives, the program implemented over 500 small infrastructure projects (to include rehabilitation of community centers, schools, sports facilities and improvements to critical tertiary road infrastructure). Through trial and error, CT developed a methodology for implementation that came to be known as the ‘tripartite model’ and has, through qualitative analysis, yielded positive 69 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV programmatic and operational impacts. According to one Mayor from the Catatumbo, a deeply insecure and conflict-affected area of Norte de Santander, “[The tripartite model] creates a sense of belonging and does away with the tendency to ‘ask and ask’…people value more when they’re involved in putting it together” (see attachment I for more information on the tripartite model). As the program enters its next phase (July 2019–February 2022), it plans to continue use of the tripartite model to implement some of its infrastructure activities, and plan to use the learning from phase I to inform the design of an evaluation to test its hypotheses about the tripartite model. With this in mind, working closely with M&E Cadre advisor Amalia Prado, CT developed an initial plan for gathering data in an effort to evaluate the model’s impacts (see attachment II, “Figure 1. Transforma Tripartite Model). However, overall conditions in Colombia due to COVID-19 have since minimized access to the program sites, and significantly shifted the team's thinking on an appropriate evaluation mechanism. While the program team has been optimistic about their ability to pursue the initial plan for a quantitative evaluation of the tripartite model despite COVID-19, it seems highly likely that collaboration between local authorities and communities will be challenged as long as the pandemic continues. This could have an effect on the relationships that CT has been able to build and strengthen. Also, a key variable leading to the success of tripartite model activities is community ownership, created initially through the selection and planning of projects in close collaboration with communities. In areas where in-person and even virtual or phone meetings are a challenge, this ability will be eroded. However, the program team still believes that there may be impacts to measure, but that they might be less significant than in a pre￾COVID-19 world. PURPOSE Over the course of the last four years, Colombia Transforma has developed a methodology for a participatory, community driven approach to infrastructure that it believes yields positive impacts on several levels. The purpose of the Tripartite Model Evaluation is to test those program hypotheses and provide answers to research questions, lending credibility to those conclusions. The evaluation will investigate the implementation and outcomes from past tripartite model activities (aka “retrospective evaluation” under Activity 1). It will also explore how the model varies across regions given the significant political, cultural, and socioeconomic differences. Pathways to change will be explored. It is hoped that the evaluation results showcase the effectiveness of the model applied in different contexts and its subsequent impacts on the stakeholders. Retrospective Evaluation (Activity 1 - EnCompass) The retrospective component (Sep 2020–May 2021) of the evaluation should capture some impacts and lessons learned from past activities judged to be both successful and not successful, which will help to shape the analysis of tripartite model activities implemented under CT 2. (For example; which mechanisms are key pathways to create the impact through the tripartite model?) Although the evaluation methodology shifted from a quantitative to a qualitative design due to COVID-19 delays, it includes similar respondent groups and data collection tools. Baseline Component (Activity 1 - EnCompass) The Baseline component of the evaluation began in earnest in December 2020 with a pilot to test the methodology and interview protocol in Taraza, Bajo Cauca and is expected to be completed in all 70 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV treatment and control veredas by the end of February 2021.The final report of the baseline evaluation is expected to be submitted by mid-May. Midline Study (OTI Internal) OTI intends to internally conduct a Midline Study around June 2021. The midline will not necessarily follow the same methodology, vereda sampling, or interview protocol as the Encompass baseline or endline. It will be used as a more informal check-in with program staff and partners to pause and reflect on any changes to implementation of the tripartite model due to the COVID-19 pandemic and other external variables. The midline results will also be shared with the Encompass evaluation team to help inform the endline. A separate SOW for the Midline Study will be prepared. Endline Component and Final Report (Activity 2 - EnCompass) Add Dates: June 2021–February 2022 At endline, the program team will assess the overall outcomes and impacts of Colombia Transforma I & II, having worked with the M&E POC (and external evaluators) to design and implement an appropriate evaluation methodology. (Note: Colombia Transforma I impact will be assessed only through retrospective study and include reflections from the CT 2 baseline and endline evaluations as well as reflections from OTI’s internal midline evaluation). Evaluation results will be used in the short-term by Colombia Transforma and OTI and—where relevant—shared with the USAID Mission, other Embassy stakeholders, Colombian government, and other international donors to better understand community dynamics around small infrastructure projects in Colombia in the longer term, it will hopefully serve as a comparison, illustrating changes that have taken place as a result of program activities. Ultimately, the information gleaned by this and similar future studies will contribute to the body of knowledge around community empowerment, ownership, and participation in implementation of the peace accord through infrastructure projects in Colombia. AUDIENCE USAID/OTI; USAID Colombia; Colombian government entities responsible for peace accord implementation such as the Agency for Territorial Renovation LINES OF INQUIRY · Does participation in the tripartite model increase ownership of infrastructure projects? Does it increase a sense of empowerment on the part of community-level participants? · Does participation in the tripartite model increase organization and ability of community groups to better advocate for and implement development resources? In what conditions? · Does use of the tripartite model create and/or strengthen relationships between Government of Colombia authorities and community members? · To what extent does the participation in the tripartite model increase community members’ and leaders’ willingness to engage in local development and peace-building to improve community well-being? · Due to the COVID-19 outbreak, how has this willingness and/or behaviors changed? 71 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · What were the key pathways to change? How did the unforeseen circumstances affect the pathways to change, if any? METHODOLOGY The overall evaluation (Activities 1 & 2 above) comprised of retrospective, baseline and endline studies will apply qualitative research methods (Annex 1) to respond to the lines of inquiry above for the retrospective and baseline phases of the study. The findings from baseline and retrospective will help refine the lines of inquiry for the endline and inform potential methodological tweaks, if deemed necessary. The pre (or baseline) phase includes qualitative data collection within the program’s target regions to establish a baseline of current community level relationships and trust in the local government, which will then be compared to post implementation data. The evaluation team is also collecting data from comparison veredas before and after the implementation of the program in an attempt to isolate the impacts of the program. The analysis focuses on how and if the program outcomes differ by context, and examine how those differences in implementation then affect the achievement of outcomes. The post (or endline) phase of data collection will serve to identify key programmatic outcomes, particularly as they relate to changes in community trust, engagement, and resilience. This phase will also identify lessons learned, and recommendations for follow-on programming. Although the endline data collection could involve mixed-methods (qualitative and quantitative), contextual conditions will likely prevent this and OTI assumes the endline will follow a similar qualitative methodology to the baseline. Encompass will share updated methodology for the endline (informed by the baseline and midline studies) with OTI for review and discussion. Given the challenges surrounding the COVID-19 pandemic, the evaluators may need to adjust methodology over the course of the evaluation and will need to develop contingency plans. For instance, they may need to make use of more remote data collection tools if Covid-19 biosecurity protocols prevent in person interviews and focus groups. A detailed contingency plan will be submitted with the endline methodology report. Once OTI and Encompass have selected control and treatment veredas for the baseline, OTI will endeavor not to change the treatment and control list, however, some flexibility may be required if conditions change on the ground. Methodological specifics will be further refined by the evaluators and OTI/Colombia team during the evaluation design. The evaluators shall analyze data that is disaggregated and analyzed by sex, age, ethnicity, and location, whenever such data is available. OTI’s activity database is a rich source of information on individual activities and should be utilized by the evaluation team. TEAM COMPOSITION Required qualifications of the evaluation team leader are outlined below, but the rest of the evaluation team should be comprised as appropriate to respond to the needs articulated by the SOW. Team members should be familiar with USAID/OTI programming approaches and be familiar with political transitions. EnCompass, in consultation with the team members, should propose the final structure of the team and the division of roles and responsibilities and LOE among them, informing OTI/Colombia no later than the start of fieldwork. OTI also welcomes the potential subcontracting of enhanced data 72 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV collection capacity and/or logistics support via a local or regional evaluation firm, if needed. The positions to be filled by EnCompass require USAID/OTI concurrence of the proposed candidates. Lead Evaluator Qualifications · Experience designing complex evaluations on issues related to conflict, violence, and/or social cohesion. · Experience managing Colombian evaluators or evaluation firms to conduct complex evaluations. · 5–10 years of experience conducting field research in Colombia preferred. · Fluent Spanish required. · Experience publishing articles, books, or scholarly journals on issues related to conflict, violence, and/or social cohesion in Colombia. TASKS AND DELIVERABLES The evaluation team, under the direction of the Team Leader, is responsible for completing the following Tasks and submitting the following Deliverables: Tasks · Refined or clarified evaluation questions in collaboration with USAID/OTI; (For Retrospective, Baseline and Endline). · Thoroughly review program documentation, including activity documents via OTI’s activity database; (Ongoing at all phases of the evaluation). · Facilitate a virtual design workshop with OTI and Colombia Transforma staff to understand the implications of COVID-19 on the tripartite model. This process should result in a modified evaluation methodology developed through a participatory process, and communicated among stakeholders (Complete). · Facilitate a virtual data validation/design workshop after baseline data collection and after the OTI internal midline is completed to share findings and refine endline methodology (approximately July 2021). · Support the selection of local evaluators (or a local firm) who will undertake data collection (Complete: Cifras y Conceptos). · Propose preliminary activities in advance of the planning, design and implementation of retrospective, baseline, midpoint and endline studies planned later in the period of performance. (Complete.) See SOW Annex 2 with vereda selections as of January 2021. · Participate in and manage data collection (ongoing). · Aggregate data on both independent variables and processes (collected by CT) and information collected on main dependent variables to provide an analysis of the impacts of the tripartite model. · Brief OTI/Colombia and Colombia Transforma staff (and others as appropriate) on preliminary findings from baseline data collection, retrospective data collection and endline data collection. These spaces should be used for data validation to create feedback loops between the program and the evaluators. 73 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV · Participate in periodic check-in calls with OTI, Colombia Transforma as requested, and keep the team abreast of any updates and challenges to the process. · Brief OTI/Colombia and Colombia Transforma staff (and others as appropriate) on final evaluation findings. Deliverables · Initial Evaluation Plan focused primarily on the baseline and retrospective components to include the following: (Complete). o Summary of proposed data collection methods, sampling plan, and analytical framework o Draft data collection tools o Management framework, including divisions of roles and responsibilities among the Evaluation Team members and points of contact for liaising with OTI, and MSI o Updated timeline with applicable LOE · Proposed Activity 1 budget for the retrospective study and baseline data collection (Phases 1-3). (Complete) · Revised Evaluation Plan focused on Endline, due after the baseline, retrospective and midline components have been completed, (July 30, 2021) to include the following: (New) o Summary of proposed data collection methods, sampling plan, and analytical framework o Draft data collection tools o Management framework, including divisions of roles and responsibilities among the Evaluation Team members and points of contact for liaising with OTI, and MSI o Updated timeline with applicable LOE o Updated Cifras y Conceptos data collection plan (New) o Covid-19 Contingency Plan Annex (NEW) that outlines Encompass’ plan if field visits are not possible due to continued lockdowns · Proposed Activity 2 budget for the endline data collection and final full evaluation report (Phases 4-6). (New) · Memo outlining findings of retrospective study and baseline data collection to include any relevant suggestions for activity design and implementation to improve evaluation outcomes and lessons learned to inform endline data collection: May 15, 2021 in English and Spanish. · Executive summary or memo outlining key findings to be included in the Final Evaluation Report in Spanish and English due December 15, 2021. · Final Evaluation Report inclusive of data collection and analysis from the retrospective, baseline, OTI’s internal midline, and endline in Spanish and English due January 28, 2022. · PowerPoint presentations from any formal briefings on findings. POINTS OF CONTACT Milanga Abeysuriya (mabeysuriya@usaid.gov) will serve as point of contact for overall activity administration and coordination, and will ensure the appropriate OTI staff are included in relevant correspondence. The OTI Colombia team (dcha.oticolombiateammaillist@usaid.gov), and in particular the Colombia Country Representative, Jessica Holst (jholst@usaid.gov), will be on point for all technical evaluation 74 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV matters, and should be the main counterparts for the lead Evaluator. They will coordinate, or identify another person to coordinate, with the Evaluator on all in-country logistics and scheduling. OTI/Colombia’s M&E POC Amalia Prado will lead the internal midline assessment and provide support to Milanga and the OTI Colombia team throughout the evaluation. The TOCOR will remain the point of contact for all technical direction requiring formal concurrence. 75 | COLOMBIA TRANSFORMA TRIPARTITE MODEL: EVALUATION REPORT USAID.GOV ANNEX G: EVALUATION PHASE 2 DATA COLLECTION TOOLS The tools for Evaluation Phase 1 are included in the Baseline Evaluation and Retrospective Analysis Report (August, 2021). The tools for Evaluation Phase 2 are included under separate cover.