Performance Evaluation of USAID/Mexico’s Crime and Violence Prevention Activity: Juntos para la Prevención de la Violencia August 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Center on Conflict and Development, Texas A&M University. Photo credit: JPV CONTENTS ACRONYMS ...............................................................................................................................................................................3 EXECUTIVE SUMMARY..........................................................................................................................................................5 I. EVALUATION PURPOSE ...........................................................................................................................................10 II. PROGRAM BACKGROUND .......................................................................................................................................11 III. EVALUATION METHODS AND LIMITATIONS ....................................................................................................13 III.1 STAGE 1 METHODOLOGY................................................................................................................................................................... 13 III.2 STAGE 2 METHODOLOGY................................................................................................................................................................... 16 STAGE 1: WERE LPSS STRENGTHENED AND EVIDENCE-BASED INFORMED INTERVENTIONS IMPLEMENTED?..................................................................................................................................................................................... 19 IV.1 JPV THEORY OF CHANGE: THEORY AND PRACTICE........................................................................................................................ 20 IV.1.1 JPV expenses...................................................................................................................................................................................25 IV.2 JPV CONTRIBUTION TO INSTITUTIONAL CAPACITY-BUILDING .................................................................................................... 27 IV.3 JPV CONTRIBUTION TO THE GENERATION OF MULTISECTOR PREVENTION AGENDAS............................................................. 31 IV.3.1 Multisector groups and government participation ................................................................................................................32 IV.3.2 Anchor and strategic actors.........................................................................................................................................................35 IV.3.4 Private sector...................................................................................................................................................................................37 IV.3.5 Civil society.......................................................................................................................................................................................38 IV.3.6 Consultants, think tanks, and academic institutions.............................................................................................................39 IV.3.7 JPV organizational structure........................................................................................................................................................40 IV.3.8 USAID...............................................................................................................................................................................................41 IV.4 JPV CONTRIBUTION TO THE IMPLEMENTATION OF EVIDENCE-BASED PREVENTION INTERVENTIONS ................................. 42 IV.4.1 Key avenues for reform: what the evidence says..................................................................................................................43 IV.4.2 JPV Donations program................................................................................................................................................................44 IV.4.3 Programmatic quality of JPV-supported interventions...........................................................................................................45 Key Principle 1: Concentration................................................................................................................................................................45 Key Principle 2: Coordination ..................................................................................................................................................................54 Key Principle 3: Proactivity.......................................................................................................................................................................54 IV.5 JPV’S CONTRIBUTION TO THE SCALE-UP OF EFFECTIVE MODELS ............................................................................................... 56 IV.5.1 Pathways to scale and elements of effectiveness..................................................................................................................57 IV.5.2 Scaling-up challenges..............................................................................................................................................................60 STAGE 2—DID THE PROGRAM CONTRIBUTE TO A REDUCTIONOF AT-RISK YOUTHLIKELIHOOD PARTICIPATING INCRIME AND VIOLENCE? ............................................................................................................................. 68 IV.6 COVERAGE AND TARGETING ............................................................................................................................................................. 69 IV.7 JPV-SUPPORTED INTERVENTIONS RESULTS IN CRIME AND VIOLENCE REDUCTION IN TARGETED LOCALITIES ................... 72 V. RECOMMENDATIONS .....................................................................................................................................................75 REFERENCES ..........................................................................................................................................................................79 ANNEX I. PROGRAM BACKGROUND........................................................................................................................................................... 81 ANNEX II. DATA COLLECTION INSTRUMENTS.......................................................................................................................................... 82 ANNEX III. RESULTS ACHIEVED BY MEASSURMENT INSTRUMENT ....................................................................................................104 ANNEX IV. COMPARATIVE MUNICIPALITIES MATRIX...........................................................................................................................108 ANNEX V. DONATIONS PROGRAM...........................................................................................................................................................117 ANNEX VI. JPV ORGANIZED EVENTS AND STUDY TOURS ....................................................................................................................121 ANNEX VII. JPV’S RESOURCES DEVOTED TO TRANSFER AND SCALE-UP AND RESULTS..................................................................128 ANNEX VIII. TARGETING ANALYSIS .......................................................................................................................................................134 ANNEX IX. PRELIMINARY STATISTICAL ANALYSIS ...............................................................................................................................138 3 ACRONYMS 5Rs resources, roles, relationships, rules, results BAM Becoming a Man CAS Center of Social Services (Spanish acronym) CBT cognitive behavioral therapy CCSJ Citizen Council for Security and Justice (Spanish acronym) CEDAT Centro de Desarrollo y Atención Terapéutica CIJ Center for Youth Integration (Spanish acronym) CINLAAC Civic Council of the Institutions of Nuevo Leon (Spanish acronym) CJHM Civic Justice Homologated Model CLA collaborating, learning, and adapting COLEF College of the Northern Border (Spanish acronym) COPARMEX Confederación Patronal de la República Mexicana (Commission of Large Enterprises of the Mexican Republic) CONAPO National Population Council (Spanish acronym) ConDev Center on Conflict and Development COPREV Council for Crime and Violence Social Prevention CPTED crime prevention through environmental design CSO civil society organization DIF National System for the Family Integral Development (Spanish acronym) DOI Development Objective I EBP Best-Practices Scale ENSU National Survey of Urban Security (Spanish acronym) FECHAC Fundación del Empresariado Chihuahuense FICOSEC Fideicomiso para la Competitividad y la Seguridad Ciudadana FORTASEG Strengthening Security Program (Spanish acronym) GBV gender-based violence GOPI-P Government Performance Index—Prevention IDIQ indefinite delivery–indefinite quantity INEA National Institute for Adult Education (Spanish acronym) INEGI National Institute for Statistics and Geography (Spanish acronym) IRB Institutional Review Board IRLSP Local Prevention System Performance Index (Spanish acronym) J-PAL Abdul Latif Jameel Poverty Action Lab JPV Juntos para la Prevención de la Violencia (Together to Prevent Violence) LPS local prevention system M&E monitoring and evaluation NGO nongovernmental organization OPI-P Organization Performance Index—Prevention PoP problem-oriented policing RFP request for proposal ROLE Resources, Options, and Freedom of Election (Spanish acronym) SAVRY Structured Assessment of Violence Risk in Youth SIMOPIN Offender Monitoring System (Spanish acronym) SIPINNA National System for the Protection of Girls, Boys, and Adolescents (Spanish acronym) SESNSP Secretary General of National Public Security (Spanish acronym) STYL Sustainable Transformation of Youth in Liberia TCO transnational criminal organization 4 USAID United States Agency for International Development UVAQ Universidad Vasco de Quiroga YTT Youth-Targeting Tool 5 EXECUTIVE SUMMARY The purpose of this performance evaluation is to gauge the effectiveness of the United States Agency for International Development (USAID) program Juntos para la Prevención de la Violencia (JPV), or “Together to Prevent Violence,” in promoting reductions in levels of crime and violence and gather recommendations for improved future programming. In particular, this evaluation assesses the effectiveness of JPV in achieving the intermediate results of strengthening local prevention systems (LPSs) and in reducing the likelihood of youth participating in crime and violence. It also explores JPV’s contribution to the development objective of sustainably decreasing crime and violence in the targeted communities. To answer the primary questions of the evaluation, we pursue a two-stage approach. In Stage 1, we analyze the results achieved in strengthening local systems. We identify implemented prevention interventions, as well as the local actors and beneficiaries who were supported by JPV. In Stage 2, we study whether the interventions contributed to: a) a reduction in the likelihood of youth engaging in crime and violence, and/or b) an overall reduction in crime and violence in targeted municipalities. The results presented for Stage 2 are preliminary, because the focus group interviews and youth survey have been delayed due to COVID-19 and will be completed upon a safer environment adhering to USAID’s “No Harm Done Principle.” The categorized preliminary findings to respective questions are: 1. To what extent was the design of the JPV program relevant and sufficient for achieving the stated results and contributing to reduced crime and violence in the targeted communities? a. To what extent did JPV’s updated theory of change (i.e., targeting high-risk youth and local systems approach) hold true? b. What JPV program elements were the most effective and least effective at promoting reduction in crime and violence in targeted communities? JPV’s theory of change was successfully reflected in the implemented strategy, which focused on working with local actors to fulfill three main functions of local systems: 1) create knowledge, 2) implement solutions, and 3) improve practices. This local system-based framework evolved during the life of the program. The implementation commenced from the first function and gradually moving to the third. Because the practical implementation of JPV was launched early 2017, it did not have enough time to generate stronger improved practice outcomes. To summarize, the program achieved stronger results in developing a shared understanding of the local violence phenomenon and in developing general organizational capacities at the local level, as well as moderate results in identifying promising practices and in implementing evidence-based models. Due to lack of evaluation of implemented solutions that are necessary to develop appropriate lessons and impact policy, improved practices were not as apparent. JPV implemented a transversal institutional-strengthening strategy and four substantive transversal strategies: a) solutions for at-risk youth, b) civic justice, c) gender strategy, and d) environmental design prevention strategy. The first two strategies achieved stronger results for 6 the third function – improved practices. The following questions explore in more detail which components worked better and why. 2. What evidence is there that JPV programming has strengthened LPSs in a sustainable manner? a. To what extent are the local actors capable of maintaining the improved crime prevention policies, to the extent that there are improved policies), without USAID support? b. Have key avenues for reform and a reduction in crime and violence in the targeted communities been addressed through JPV programming and implementation? Why or why not? JPV achieved positive results with the implementation of the transversal institutional￾strengthening strategy. The targeting of activities was too broad and involved a lot of local implementing partners. This policy had an opportunity cost of gaining stronger results with possibly fewer organizations implementing more focused programs. Even though the program had notable success in increasing awareness and positioning the crime-and-violence-prevention agenda within the local and national systems; the results achieved in terms of generating sustainable, multisector, coordinated agendas were mixed. JPV succeeded at enabling multisector collaboration when prevention strategies aligned with interest groups, and government participation was obtained when related short-term milestones were specified and followed. JPV opted to create “anchor actors” and “strategic actors,” with the objective of ensuring sustainability by empowering these actors to become articulators of collaborative efforts and to help other members develop their capacities. We find that this strategy was accompanied by several challenges that limited its success, particularly because the program failed to create the conditions for these actors to become the leaders of local agendas in most cases. Moreover, our analysis shows that the conceptualization of the anchor actor, while attractive, lacked appropriate coordination and application. This role did not exist or was not necessary in all systems. Finally, we find that while JPV’s organizational structure with regional coordinators was very successful in increasing local actors’ participation, the hierarchical structure of USAID may have hindered local ownership. In sum, we find that the prevention agenda cannot always be clustered within a single group and that specialization and working with diverse leadership are desirable. In terms of the implementation of evidence-based policies, we assess whether JPV supported interventions that followed the three principles of effective violence prevention found in the literature: concentration, coordination, and proactivity. We find that JPV technical assistance activities added to the concentration principle by supporting the development of civil society organization (CSO) programs with enhanced design, delivery, and coverage, although pressing challenges remain in terms of addressing youth risks and needs to prevent recidivism. JPV also contributed, but to a lesser extent, to the coordination principle by enhancing communication between CSOs and other relevant stakeholders, although major challenges prevailed with coordination with public institutions. The proactivity principle remains an area of opportunity for future USAID efforts. 7 JPV evolved from a broader strategy looking at identifying promising interventions to supporting reintegration strategies to, finally, focusing almost exclusively on strengthening and transferring cognitive behavioral therapy (CBT) models and the Civic Justice Homologated Model (CJHM). Through its evolution, JPV made good progress in identifying promising practices, but achieved less in strengthening sustainable models. Therefore, the program suffers from generating evidence of effectiveness for replication in most cases. JPV was successful at leveraging different pathways to scale to expand two CBT models, Resources, Options, and Freedom of Election (ROLE being its acronym in Spanish), Mente Libre, and the CJHM. We identify that the focus from the beginning on scale, the provision of a meticulous strengthening process through access to overarching resources, and the generation of relationships of trust and co-investments with partners are key to achieving scale. Although JPV was very successful in supporting the adoption of these models, rushed expansion, absence of rigorous evaluations, weak technical capabilities in implementing organizations, and lack of consideration for differences in local circumstances posed challenges for model fidelity and effectiveness. 3. What evidence is there that JPV has reduced the likelihood of at-risk youth in targeted communities participating in crime and violence? a. What evidence is there that at-risk youth reached with JPV programming will engage in alternative, positive pathways and steer away from involvement in crime and violence, including with transnational criminal organizations (TCOs)? b. Is JPV’s coverage or reach of at-risk youth in targeted communities (i.e., number of at-risk youth reached) sufficient to initiate a detectable reduction in youth involvement in crime and violence? Why or why not? c. To what extent have gender-based roles, including but not exclusive to the mothers of at-risk youth, affected the ability of the interventions to achieve their stated results (effectiveness)? Our field research and evidence review suggest that crime in Mexico is a highly contextual, socio￾culturally driven behavioral phenomenon. Therefore, a successful intervention also needs to take into account the broader context of violence to which program participants continue to be exposed during and after the program. However, our amassed secondary information shows that JPV-supported interventions usually only targeted 1 out of the 10 most violent neighborhoods within the municipalities of interest, and often targeting only a limited number of youths in these neighborhoods. Our preliminary results suggest that, while the intervention may have had a positive effect on individual participants in those municipalities, a wider coverage, both in terms of services offered and youth reached, would be necessary in order to obtain community-level effects in crime and violence reduction. In terms of services offered, we find that the designs of existing interventions do not sufficiently tackle youth risk factors and that gender roles are barely addressed. The programmatic offer currently covers impulsive behavior, anger management, and employability skills, and to a lesser extent substance consumption. However, data from the Youth-Targeting Tool (YTT) confirm that a wider range of interventions is needed to address youth risk factors, particularly to promote protective factors. Interventions that are able to promote protective factors and 8 interrupt the community and family dynamics that produce antisocial behaviors remain urgent, as does expanding the offer of substance abuse treatment. These results are preliminary. At a later stage, we will perform focus groups with the beneficiaries of direct interventions to understand which components might have worked or not and why. We will also collect data of at-risk youth in the targeted municipalities through a representative sample of communities with high homicide rates to measure individual-level exposure and attitudes toward violence. 4. What evidence is there that the USAID/Mexico crime-and-violence-prevention programming through the JPV activity has contributed to achieving the goal of reducing crime and violence within the targeted JPV communities? a. Where and what are examples of the JPV activities contributing to reduced levels of crime and violence in the targeted municipalities? b. To what extent have citizen perceptions of security increased in targeted municipalities? We perform a descriptive statistical analysis of existing secondary data on homicide and other crime rates, as well as on citizen perceptions of security, to investigate JPV’s role in reducing crime and improving perceptions. However, we remain very cautious and refrain from drawing any causal connections from this analysis. The increased levels of violence in Mexico are in large part attributable to organized crime. Consequently, insecurity perceptions are also driven by violence related to drug cartels and to other highly visible acts of violence that affect perceptions. Hence, a decrease in homicide rates and reducing insecurity perceptions, may require different strategies than the one envisioned for JPV and that entail longer-term institutional reform and more traditional law enforcement action. Our initial statistical analysis shows that homicide rates present positive trends for all municipalities of interest since 2015. Insecurity perceptions have risen in all municipalities, except for Guadalupe and Escobedo (although data for these municipalities are only available since June 2018). We also explore other available indicators related to antisocial behaviors that may be better suited to explore the program’s contribution to reducing the likelihood of youth engaging in crime and violence. Unfortunately, the lack of clear counterfactual information, along with limited socio-economic data, hindered our ability to isolate the effect of the program on youth population. From our field explorations we generate a plausible hypothesis that JPV could have impacted individual youth perceptions, attitudes, and exposure to violence. In order to address the challenges with the existing data, we have designed a unique and tailored survey of youth that will measure youth exposure, attitudes, and behaviors toward violence. Our stratified sampling procedure will include youth in the targeted municipalities as well as youth from similar municipalities which remained outside the realm of JPV, to form a comparison group. Based on JPV’s learnings, we provide guidelines for more impactful, proven, and sustainable approaches for future crime-and-violence-prevention programs. First, crime-and-violence￾prevention systems should be narrowly defined according to expected outcomes rather than long-term term development goals. This is particularly relevant for issues as complex as crime and violence and in settings where the drivers of the phenomenon are too diverse. Local 9 prevention agendas must evolve around specific prevention strategies, and working with diverse leadership is desirable. The focus should be on generating networks of specialized knowledge. As different systems may be contributing to the same development goal, future programming should also facilitate communication channels between specialized systems. Additionally, concentrated efforts may also decrease the number of conduit organizations increasing bureaucratic efficiency as well reducing cost. Second, efforts should focus on catalyzing a wider range of models for at-risk youth by giving clear tools to local actors to catalyze new models and enable the conditions for specific and promising innovations to be able to thrive. Future programming should invest in developing and adapting intervention models for which there is a strong evidence base, such as multidimensional therapy, addiction treatment, and psychiatric approaches. Programs need be fostering innovations based on the concentration, coordination, and proactivity principles. Third, it is imperative that future initiatives carefully consider the need to adapt programs. Even within a country, local contexts differ vastly. An “adapt-pilot-scale” process is strongly encouraged. It is important to verify if the conditions are conducive for the implementation to be carried out as desired. If the local conditions allow for adoption of the essential components of the theory of change, then the nonessential components should be adapted to the local context. If the essential components cannot be held under the new circumstance, then other policy alternatives should be explored. Organizations should also be given ample time to polish implementation models to achieve successful transfers, and rigorous evaluations should be carried out. Investing in a few strong pilots might bring a higher payoff than scaling up suboptimal models. Finally, engaging with government agencies to create specialized capacity for the design and implementation of specific services aimed at reducing recidivism is of the highest importance. Future programming should support public institutions shift away from a public safety and punitive approach to a public health perspective. This approach should be based on the understanding of youth protective and risk factors. Law enforcement and judicial prosecutors need to work collaboratively with service providers, such as: health and mental health services, education and employment support, and family counsel, to facilitate the long-term recovery of offenders. Working with these institutions to further this approach is a necessary condition for successful and sustainable prevention policies to thrive. 10 I. EVALUATION PURPOSE The purpose of this performance evaluation is to gauge the effectiveness of the USAID JPV program in promoting reductions in levels of crime and violence (accountability) and gather recommendations for improved future programming (learning). The evaluation 1. gauges the extent to which the USAID/Mexico crime-and-violence-prevention programming has contributed to USAID/Mexico’s Development Objective 1 (DO 1): crime and violence in targeted communities sustainably decreased 2. assesses the effectiveness of the USAID/Mexico crime-and-violence-prevention portfolio’s JPV activity in achieving DO 1 intermediate results: 1.1) at-risk youth likelihood to participate in crime and violence reduced and 1.2) LPSs strengthened 3. provides recommendations on more impactful, proven, and sustained approaches for future crime-and-violence-prevention programs The key evaluation questions addressed in this performance evaluation are highlighted in the Methodology sections below. The primary intended audience for the evaluation findings and recommendations is USAID in Mexico and the Mexican government, although the evaluation team considers that these findings are also relevant to share with stakeholders who collaborated with USAID for the implementation of this program and that the lessons learned and recommendations can be widely shared with the public to help inform future crime-and-violence￾prevention policies. This evaluation was commissioned to the Center on Conflict and Development (ConDev) at Texas A&M University. ConDev seeks to improve the effectiveness of development programs and policies for conflict-affected and fragile countries through multidisciplinary research and education. Our research is focused largely on mitigating conflict related to food security, natural resources, youth populations, and local institutions. 11 II. PROGRAM BACKGROUND Mexico has experienced significant increases in crime and violence in recent years, adversely affecting the country’s security and economic growth, with an especially detrimental effect on youth. Much of the crime and violence is driven by criminal organizations battling to fill the vacuum left by the fracturing of large cartels. Violence and rapid urbanization, particularly in Mexico’s northern border cities, have contributed to increased crime rates and recruitment of youth into criminal organizations, as well as threatened the security and well-being of citizens. USAID, through the JPV activity, aims to contribute to the reduction of crime and violence in Mexico by strengthening the capacity of local crime-and-violence-prevention systems to design, implement, and evaluate prevention practices and policies.1 The USAID JPV activity, implemented by Chemonics International, Inc., represents the largest activity within the DO 1 portfolio and aims to strengthen the capacity and sustainability of local violence prevention systems for sustained results by improving the crime-and-violence￾prevention policies of the Mexican government at federal, state, and local levels. This activity was implemented from October 1, 2016 to September 30, 2020 for a total cost of $24,466,087. JPV aims to contribute to the reduction of crime and violence in Mexico through the achievement of three intermediate outcomes, as reflected in the program’s results framework in Annex 1: 1) contribute to the improvement of the Mexican government’s crime-and-violence-prevention policy; 2) strengthen the main actors that make up diverse local systems of prevention; and 3) promote the adaptation of evidence-based crime-and-violence-prevention practices and policies within local systems. This results framework was defined in 2017 after USAID/Mexico requested a change in the program’s programmatic approach to reflect the local systems framework. 2 This framework is rooted in the understanding that achieving any development outcome depends on the contributions of multiple interconnected actors and that sustaining development outcomes depends on the sustainability of the local system, understood as its capacity to adapt to changing circumstances. Under this framework, USAID’s role should be one of facilitator instead of implementer. In response, JPV sought to build local ownership by providing technical assistance to enable local stakeholders to drive their prevention agenda and to fulfill the functions of a local system. To this end, JPV prioritized strengthening local authorities’ commitment to tackle crime and violence, increasing the programming capacity of CSOs, and continuing engagement of the private sector in violence prevention. The program is premised under the following theory of change: “By strengthening local systems, local actors will lead evidence-based prevention interventions. These interventions will target the communities and youth most at-risk. Through a combination of interventions to address individual, family, and community risk factors, as well as interrupt the cycle of violence, there will be a sustainable decrease in crime and violence.” JPV encourages local partners to focus on secondary3 and tertiary4 activities toward reductions in violence levels. 1 This is the context as presented by USAID. 2 See “Local Systems: A Framework for Supporting Sustained Development.” USAID, April, 2014. 3 Secondary prevention focuses on at-risk youth. 4 Tertiary prevention focuses on youth who are jailed or on probation. 12 The activity was implemented in nine municipalities in five states: 1) Tijuana in Baja California, 2) Chihuahua and Ciudad Juárez in Chihuahua, 3) Escobedo, Guadalupe, and Monterrey in Nuevo León, 4) Zapopan and Tonalá in Jalisco, and 5) Morelia in Michoacán. 13 III. EVALUATION METHODS AND LIMITATIONS To answer the key evaluation questions of this performance evaluation, we pursued a two-stage approach following the program’s theory of change. A theory of change explains how the activities undertaken by an intervention contribute to a chain of results that lead to the intended or observed impacts. Following this logical order, in Stage 1, we analyze the results achieved in strengthening local systems. We identify implemented prevention interventions, as well as the local actors and beneficiaries who were supported by JPV. In Stage 2, we study whether the interventions, as implemented by the local systems, may have contributed to a reduction in the likelihood of youth engaging in crime and violence and to an overall reduction in crime and violence in targeted municipalities. The two-stage approach is presented in Figure 1, and below we present the methods used to answer each of the key evaluation questions. The results presented for Stage 2 are preliminary, because two components of the data collection, focus group interviews and youth survey, have been delayed due to COVID-19 and will be completed upon a safer environment adhering to USAID’s “No Harm Done Principle.” We discuss this in more detail below. Figure 1. Two-stage Methodology III.1 STAGE 1 METHODOLOGY In this stage, we focus on analyzing the program design and implementation and on understanding the results achieved on intermediate result 1.2 (LPSs strengthened). We answer the following key evaluation questions: 14 1. To what extent was the design of the JPV program relevant and sufficient for achieving the stated results and contributing to reduced crime and violence in the targeted communities? a. To what extent did JPV’s updated theory of change (i.e., targeting high-risk youth and local systems approach) hold true? b. What JPV program elements were the most effective and least effective at promoting reduction in crime and violence in targeted communities? 2. What evidence is there that JPV programming has strengthened LPSs in a sustainable manner? a. To what extent are the local actors capable of maintaining the improved crime prevention policies, to the extent that there are improved policies, without USAID support? b. Have key avenues for reform and a reduction in crime and violence in the targeted communities been addressed through JPV programming and implementation? Why or why not? As stated by USAID,5 many final performance evaluations fall into outcome evaluations. This type of evaluation tends to consider results at all levels of an activity’s or project’s logical framework, that is, whether outputs were produced, to what degree the purpose was achieved, and whether any change in the status of the goal could be detected. This evaluation stage is concerned with understanding what outputs were produced, to what degree the purpose of strengthening LPSs was achieved, how programs were implemented, and whom they benefited—to then investigate whether these activities were able to promote change on the end goals of the program. To answer these questions, we performed an in-depth desk review of all relevant and available documentation provided by JPV and USAID, including planning and performance reports, assessments, and performance-monitoring data. With this documentation, we mapped JPV’s “pathway to change” to determine to what extent the theory of change as implemented held true, as well as to provide an analysis of the activities as implemented versus the program’s objectives. It is important to mention that JPV tracked 22 performance indicators related to the results framework, as well as additional nonobligatory indicators related to the end goals. We do not study all of these indicators because that information is available in JPV’s performance reports. Rather, throughout the analysis, we bring to light specific results as they become relevant to our analysis. We also gathered primary data of the local-level actors who were supported by JPV through an online survey and semistructured interviews. Both of these instruments were designed based on the 5R framework, which highlights five key dimensions of a system: resources, roles, relationships, rules, and results, to investigate how JPV interacted with local systems and to measure perceptions on key results. 6 Data collection instruments can be consulted in Annex II. 5 “Performance Evaluation Designs.” USAID, 2019. 6 The 5R framework provides a scheme to analyze how the key dimensions of a system interact to produce desired outcomes, as follows. Human actors, both individuals and organizations, interact within a system assuming certain roles and within a network of various sets of relationships. The interactions depend on the resources actors have to produce results, and the process of transforming resources into results is governed by a set of rules. The 5R Framework in the Program Cycle. USAID, October 2016. 15 Data from the only survey were collected from April 8 to May 4, 2020. The survey was sent to a list of 147 stakeholders selected in collaboration with JPV, with a response rate of 65%. 7 Respondent statistics are presented in Tables 1 and 2. Table 1. Online Survey Respondents by Region Region # of respondents % of total Tijuana 27 16 Chihuahua 6 6 Ciudad Juárez 25 24 Monterrey 22 21 Guadalajara 12 12 Morelia 12 12 Total 104 Table 2. Online Survey Respondents by Sector Sector # of respondents % of total Local government 23 2 Federal/state government 15 16 Private sector 7 7 Civil society 51 49 Academia 6 6 Other 2 2 Total 104 The semistructured interviews were performed online from April 15 to May 30, 2020 through Zoom.8 The team invited a total of 57 key actors for interviews, including local anchor and strategic organizations, nonprofit organizations, governments, consultants, and the JPV team, with a response rate of 82%. In addition, nine interviews were carried out with field coordinators and six group interviews were conducted with program operators of selected implementing organizations. Table 3 and 4 present the distribution of performed interviews by region and sector. Data from all interviews were codified using Dedoose, a web-application for mixed￾methods research that allows codification of qualitative information. In total, the team codified 1,561 excerpts from the interviews. Table 3. Semistructured Interviewees by Region Region # of interviewees Tijuana 6 Chihuahua 8 Ciudad Juárez 10 Monterrey 9 Guadalajara 13 Morelia 4 JPV Team 8 National* 4 Total 62 Table 4. Semistructured Interviewees by Sector Sector # of interviewees % of total Local government 10 16 Federal/state government 1 2 Private sector 1 2 Civil society Directors 17 27 Field coordinators 9 15 Program operators 6 9 Academia 2 3 JPV team 8 13 Other 8 13 Total 62 7 We have a total of 96 responses from the original list. We have eight additional responses. Some of these responses correspond to different government areas. Most likely, these are public institutions that registered the name differently or in less detail than what we had in the directory. Two additional responses belong to organizations for which two people instead of one responded to the survey. We decided to leave these responses because they have no significant effect on any of the performed analyses. 8 The original plan was to conduct these interviews face to face and to spend 2 to 3 days per region for this purpose, as well as to conduct focus groups. Unfortunately, given the COVID-19 pandemic, the Condev evaluation team not able to travel to the regions, but we were successful in transitioning to an online strategy to perform semistructured interviews. 16 To answer whether the program was able to strengthen local systems, we developed a general comparative analysis of the municipalities where JPV worked. Based on the 5R framework, in this analysis, the main features of how JPV worked in each municipality were highlighted to draw conclusions on potential features that might foster or hinder progress. To answer whether the program was able to influence key avenues for reform, we took a closer look at specific interventions supported by JPV to determine whether investments followed the existing evidence of what is effective, based on the USAID Abdul Latif Jameel Poverty Action Lab (J-PAL)–commissioned report, What Works to Prevent Violence Among Youth? For this purpose, we developed case studies based on the experience of six organizations purposely selected in collaboration with JPV and USAID based on their successes and program characteristics. For the case studies, we analyzed documentation, performed interviews with program operators, and will collect data from direct beneficiaries through focus groups. The sample provides information on programs that integrate either focalized interventions, community-based interventions (territorial), or both. It also includes both organizations that work with the juvenile justice system and with the civic courts. Five of these organizations received a grant, while one had a subcontract (ProSociedad). Of those that received a grant, one was directly from USAID, and the rest were from JPV. Finally, to analyze how JPV leveraged its resources to transfer and scale up promising prevention interventions, we purposely selected three intervention models (ROLE, Mente Libre, and the CJHM in Morelia and Escobedo) as case studies. These models were selected based on their success and potential for adaptation and expansion. Table 5 summarizes the sample characteristics. Table 5. Case Study Selection and Characteristic Summary Selection criteria JPV grant/subcontract USAID grant Juvenile justice system Civic justice Model transfer Focalized Centro de Atención al Talento (CEDAT) ProSociedad Consejo Ciudadano CEDAT Consejo Ciudadano CEDAT ProSociedad CEDAT ProSociedad Territorial CASA La Tenda di Cristo Supera CASA La Tenda di Cristo Supera Supera Institutional reform CJHM Morelia CJHM Escobedo III.2 STAGE 2 METHODOLOGY In this stage, we focus on assessing the effectiveness of JPV in achieving intermediate result 1.1 (at-risk youth likelihood to participate in crime and violence reduced) and to gauge the extent to which the activity contributed to DO 1 (crime and violence in targeted communities sustainably decreased). We answer the following evaluation questions. 17 3. What evidence is there that JPV has reduced the likelihood of at-risk youth in targeted communities participating in crime and violence? a. What evidence is there that at-risk youth reached with JPV programming will engage in alternative, positive pathways and steer away from involvement in crime and violence, including with TCOs? b. Is JPV’s coverage or reach of at-risk youth in targeted communities (i.e., number of at-risk youth reached) sufficient to initiate a detectable reduction in youth involvement in crime and violence? Why or why not? c. To what extent have gender-based roles, including but not exclusive to the mothers of at-risk youth, affected the ability of the interventions to achieve their stated results (effectiveness)? 4. What evidence is there that the USAID/Mexico crime-and-violence-prevention programming through the JPV activity has contributed to achieving the goal of reducing crime and violence within the targeted JPV communities? a. Where and what are examples of the JPV activity contributing to reduced levels of crime and violence in the targeted municipalities? b. To what extent have citizen perceptions of security increased in targeted municipalities? The objective of this evaluation stage is to address outcomes and cause-and-effect questions to understand the contribution of USAID assistance to the outcomes of interest. Specifically, we are interested in understanding whether prevention interventions, as implemented by the local systems, may have contributed to a reduction in the likelihood of youth engaging in crime and violence and to an overall reduction in crime and violence in targeted municipalities. To answer the questions of interest of this performance evaluation, we pursued a mixed-methods approach based on standard case study techniques, including both qualitative and quantitative tools and survey experiments to draw conclusions on what might have worked or not. The core of the evaluation relies on the assessment of observational data. First, we analyze whether JPV’s coverage or reach of at-risk youth in targeted communities was sufficient to initiate a detectable reduction in youth involvement in crime and violence. To this end, we gathered, to the extent possible, data from beneficiary databases from supported interventions. Second, we will perform focus groups with direct beneficiaries of the case studies (see Section III.1) to understand which intervention components might have worked or not and why. We will make sure to incorporate youth who, based on the various characteristics of their environment, are more likely and less likely to respond well to the interventions selected. The possibility to reach these youth will depend on the collaboration of local organizations in supporting contact with them. In addition, deep consideration will be given to USAID’s “No Harm Done Principle.” Third, we will collect data of at-risk youth in communities with high homicide rates through an original survey to measure individual-level exposure to and perceptions of crime and violence. Our stratified sampling procedure will include youth in the targeted municipalities as well as youth from similar municipalities which remained outside the realm of JPV. A vignette experiment is embedded in the survey to measure the effect of hypothetical exposure to the intervention on subjects’ attitudes toward criminality. Vignette studies use short descriptions of 18 hypothetical scenarios (vignettes) that are usually presented to respondents within surveys in order to elicit how their judgments about such scenarios affect outcomes of interest, often revealing their perceptions, values, or social norms. This survey will be informative of the potential mechanisms through which the intervention may have affected the outcomes of interest. Unfortunately, given the COVID-19 pandemic, the team has had to delay the field work. We will perform the focus groups and collect the youth survey upon a safer environment. Finally, the we will perform descriptive statistical analysis of existing secondary data on crime and homicide rates, as well on citizen perceptions of security, to understand the context under which this program was implemented and the observable trends in treated municipalities and the rest of the country. One important limitation to highlight is that existing government surveys do not provide panel data representative of the local level. Therefore, this analysis is restricted to the level of specificity allowed by the existing data. 19 IV. FINDINGS AND CONCLUSIONS In this section, we present the findings of our analysis, organized as follows. First, we answer the questions of whether JPV was able to strengthen LPSs and whether evidence-based informed interventions were implemented. To this end, we present an analysis of 1) JPV’s theory of change, 2) JPV’s contribution to institutional capacity-building, 3) JPV’s contribution to the generation of multisector prevention agendas, 4) JPV’s contribution to the implementation of evidence-based prevention interventions, and 5) JPV’s contribution to the scale-up of effective models. Next, we answer whether JPV contributed to a reduction in the likelihood of at-risk youth participating in crime and violence activities. To this end, we present an analysis of 6) coverage and targeting and 7) JPV-supported interventions results. As explained in the Methodology sections, this analysis is preliminary given the delays in field data collection due to the COVID-19 pandemic. In the analysis, we highlight which of the questions of interest are addressed by each section, and we provide summaries of the findings in boxes. STAGE 1: WERE LPSs STRENGTHENED AND EVIDENCE-BASED INFORMED INTERVENTIONS IMPLEMENTED? 2. To what extent was the design of the JPV program relevant and sufficient for achieving the stated results and contributing to reduced crime and violence in the targeted communities? a. To what extent did JPV’s updated theory of change (i.e., targeting high-risk youth and local systems approach) hold true? b. What JPV program elements were the most effective and least effective at promoting reduction in crime and violence in targeted communities? Box 1: Question 1 Summary To answer this question, we analyze JPV’s theory of change, its design, and how it was implemented. We also review the program’s expenses to assess whether investments were directed to the activities as envisioned by the theory of change. Conclusion: JPV’s theory of change was successfully reflected in the strategy developed by Chemonics International, Inc. and in the activities it envisioned to implement to achieve its goal of reducing crime and violence, based on working with local actors to fulfill three main functions of local systems: 1) create knowledge, 2) implement solutions, and 3) improve practices. The implementation commenced from the first function and gradually moving to the third. Because the practical implementation of JPV was launched early 2017, it did not have enough time to generate stronger improved practice outcomes. The program achieved stronger results in developing a shared understanding of the local violence phenomenon and in developing general organizational capacities at the local level, as well as moderate results in identifying promising practices and in implementing evidence-based models. Due to lack of evaluation of implemented solutions that are necessary to develop appropriate lessons and impact 20 policy, improved practices were not as apparent. In the rest of the document, we review in more detail which components of each strategy were more and less effective. IV.1 JPV THEORY OF CHANGE: THEORY AND PRACTICE JPV aims to contribute to the reduction of crime and violence in Mexico through the achievement of three intermediate outcomes, as reflected in the program’s results framework in Annex 1: 1) contribute to the improvement of the crime-and-violence-prevention policy of the Mexican government; 2) strengthen the main actors that make up diverse local systems of prevention; and 3) promote the adaptation of evidence-based crime-and-violence-prevention practices and policies within local systems. As mentioned above, this results framework was defined in 2017 after USAID/Mexico requested a change in the program’s programmatic approach to reflect the local systems framework. In response, JPV defined an LPS as a set of interconnected actors belonging to the public sector, academia, civil society, or private sector who work together to reduce crime and violence incidence by identifying the causes of violence and implementing specific solutions to address them, while also ensuring knowledge management of locally generated learnings. JPV further defined that this system must generate individual incentives to encourage member participation and that incentives must be defined for other actors to join the system over time. To implement the local systems framework and reflect the theory of change, JPV developed a local systems strategy based on the three main axes that crime-and-violence-prevention systems must fulfill: 1) create knowledge by generating strategic information about crime and violence in the region, problems affecting the territory, types of violence, and types of affected populations under a prevention-and-risk-factor scheme, as well as identifying potential solutions; 2) implement solutions by strengthening institutional capacities to design, implement, and monitor solutions that address the identified problems in the territory; and 3) improve practices by evaluating the implemented solutions with the objectives of applying lessons learned to the development of other solutions and impacting public policy. JPV sought to build local ownership by providing technical assistance to enable local stakeholders to drive their prevention agendas and to fulfill the functions needed on each axis. JPV focused its efforts on strengthening six LPSs. Some of these regions, Morelia, Escobedo, Monterrey, and Guadalupe, were previously selected to become part of the “city network” strategy pursued by JPV before the change in strategy. Seeking to expand its reach, JPV then added Tijuana, Ciudad Juárez, Chihuahua, Zapopan, Tonalá, and Guadalajara. Furthermore, given the new definition of local systems based on the interconnectivity of actors rather than on regional boundaries alone, JPV redefined Zapopan, Tonalá, and Guadalajara as the metropolitan area of Guadalajara and Guadalupe, Escobedo, and Monterrey as the metropolitan area of Monterrey. Throughout the document, we will refer to these metropolitan areas simply as Guadalajara and Monterrey. Figure 2 shows the functions that comprise each axis of the local system strategy and the activities that JPV implemented to fulfill them, and it also displays the relationship of these activities with the program’s results framework. Given that JPV supports local actors in employing proven approaches and targeting the most at-risk youth for a sustained reduction in crime and 21 violence, to better reflect the scope of the program, we added a fourth intermediate result to the results framework: 4) reduce the likelihood of youth engaging in violent activities. As the Activities row shows, JPV’s technical work was focused on providing and facilitating trainings, tools, and solutions to enhance institutional capacities and evidence use and generation for the implementation of crime-and-violence-prevention policies. Trainings were targeted at the government and civil society sectors, and tools were designed to engage with local actors to measure their capabilities and design specific strengthening plans. Most importantly, the program sought to transfer these tools to local and strategic actors to ensure sustainability. JPV generated partnerships and contracted services to implement solutions and to generate knowledge on specific solutions. It provided technical assistance to local institutions to implement these solutions and helped them systematize best practices. Finally, it pursued policy influence strategies to shape the public debate and originate policy changes within systems. Figure 2. JPV Local Systems Strategy and Implemented Activities in Relation to the Results Framework Although not described as such by the program,9 we identify that JPV implemented a transversal institutional-strengthening strategy and four substantive transversal strategies: a) solutions for at￾risk youth, b) civic justice, c) gender strategy, and d) environmental design prevention strategy (see Box 2). The last two are strategies that the program had initiated before the change to the local systems framework focused on at-risk youth. Environmental prevention became a low priority for the program, and we find that the gender strategy was managed in parallel to the local systems and at-risk youth strategies. Most of the analysis presented in this document pertains to 9 The transversal strategies and strategic focus of the program changes across years. The evaluation team had to develop a framework in order to systematize the wide array of activities implemented by the program. We identified these as the main strategies implemented by the program throughout the years and organize all pieces of information accordingly. All the analysis is based on the three axes of the local functions and on these transversal strategies. Development objective: Contribute to the reduction of violence and crime by strengthening adaptation and implementation capacities of local systems of prevention. Intermediate Results IR1: Support the Government of Mexico (GOM) in improving national violence and crime prevention policy and with its collaboration with local actors. IR2: Improve the performance of actors within local violence and crime prevention systems. IR3: Contribute to local systems generating and implementing evidence-based violence and crime prevention policy. IR4: Reduce the likelihood of youth engaging in violent activities System function Activities Generate knowledge 1) Build strategic alliances; 2) Generate local information about crime and violence; 3) Generate communication products; 4) Identify and disseminate good practices Implement solutions 1) Create and agree on a common agenda; 2) Develop capacity to implement solutions; 3) Raise funds to implement solutions; 4) Implement solutions; 5) Involve relevant actors to implement solutions Improve the practice 1) Evaluate solutions; 2) Establish transparency mechanisms; 3) Create a local training system; 4) influence public policy - Trainings in C&V diagnosis and citizen participation - Evidence reviews and initiatives - Events and conferences - Systematization of good practices - Trainings in evaluation - YTT adaptations and users training - YTT Platform - Events and conferences - Measurement tools: OPI-P, EBP, GOPI-P, YTT – & technical assistance - Trainings in program design, Theory of Change, evidence use, monitoring, prevention strategy and program development - Transversal stratgies (Civic Justice, Gender violence, Youth at-risk, CPTED) - Implementation guides and manuals - Events and conferences - Study tours 22 the institutional-strengthening, at-risk youth, and civic justice strategies, but we highlight results achieved by the other two when relevant. Box 2: JPV Transversal Strategies Institutional-strengthening strategy: Recognizing that “institutional strengthening and evidence generation for violence prevention interventions training is a quintessential factor that increases actors’ knowledge, leads to the successful implementation of prevention activities, and ensures the sustainability of the impact of these activities,”10 JPV implemented a mixture of learning strategies (both online and face-to-face) targeted at a wide array of actors from all sectors to improve their capabilities, including courses on strengthening violence prevention, strengthening institutional capacity, evidence generation, and the development of measurement instruments to be transferred to local actors. Solutions for at-risk youth: Working with at-youth risk was the program’s priority. In order to develop local systems’ capacities to implement evidence-based prevention interventions, JPV developed a wide array of activities targeted at generating and systematizing evidence; identifying, strengthening, and systematizing effective interventions; sharing knowledge; and scaling up effective policies. With a particular focus on reintegration strategies, JPV implemented a donations program aimed at providing financial support to outstanding interventions while seeking to systematize their models and promote knowledge transfer. Civic justice: The CJHM was promoted by JPV based on the innovative approach of public officials in Morelia and Escobedo to deescalate violence and conflict and offer an alternative for everyday conflict resolution. This is an institutional model of community courts with restorative justice components, to which JPV also incorporated the “problem-oriented policing” (PoP) approach to strengthen the role of the police within the framework of civic justice. Through a series of technical assistance projects and in collaboration with the National Commission of Public Security, JPV focused on developing a systematized model that could then be transferred to other regions and developed trainings and materials to facilitate its adoption. Gender strategy: This strategy was focused on family violence and sexual violence. The work on family violence was initially developed in the State of Chihuahua and was focused on the design and implementation of a care model for child and adolescent victims of family violence with a national and municipal focus. It then got transferred to Pachuca-Hidalgo, Guadalajara, and Monterrey. The work on sexual violence sought to build safe communities for girls, adolescents, and young women also in the State of Chihuahua. This strategy included the provision of specialized training though the Diploma on Comprehensive Care for Sexual Abuse during Childhood and Adolescence, developed by JPV in alliance with National Pedagogical University of Chihuahua. Crime prevention through environmental design (CPTED): This strategy, implemented in Chihuahua, Tijuana, Tonalá, and Guadalupe, aims to prevent crime and increase security perceptions in specific neighborhoods. Through this model, Government authorities, organized citizens, CSOs, academia, and the private sector work together to improve the environment through citizen participation activities such as the recovery of public spaces, cleaning of streets and areas of risks, social reinforcement activities in common areas, painting works, and reforestation, among others. JPV developed a specific strategy to promote synergies between the government and CSOs and between the CPTED model and specific youth risk factors. 10 JPV Annual Work Plan 2017-2018. 23 JPV’s theory of change was successfully reflected in the strategy developed by Chemonics International, Inc. and in the activities it envisioned to implement to achieve its goal of reducing crime and violence. Figure 2 shows that all the intermediate results and their components are covered by the local systems strategy and its activities. The implementation of the local system strategy commenced from the first function and gradually moving to the third. Because the practical implementation of JPV was launched early 2017, it did not have enough time to generate stronger improved practice outcomes. This is confirmed by the LPS Performance Index (IRLSP being its acronym in Spanish) developed by JPV to measure the evolution of the local systems' three main functions. JPV applied this measurement instrument twice in 2019: one in January and February (reported in April), and a second one in September and October (reported in November). Table 6 presents the results (all scores are out of 100 unless otherwise noted). The average results provide an overview of the areas where the program made more progress. The first axis, create knowledge, has the highest score, with averages of 71.33 in April and 72.35 in November. The second-highest scores are for the second axis, implement solutions, with averages of 62.13 and 70.4 (out of 125).11 Finally, the third axis, improve practices, shows the lowest scores, with averages of 36 and 43.09. As a general conclusion, this index shows that the local systems performed better in fulfilling the activities related to the first function, such as developing a shared understanding of local violence and building shared agendas, and it shows moderate performance implementing evidence-based interventions. Although the index shows an important improvement in the function of improve practices from April to November 2019, it depicts that the local systems still have ample areas of opportunity in related activities such as evaluating solutions and impacting public policy. Table 6. IRLSP (April and November 2019) Region Axis 1. Create knowledge (max. 100)12 Axis 2. Implement solutions (max. 125)13 Axis 3. Improve practices (max. 100)14 Total index (max 1.00) Apr Nov Apr Nov Apr Nov Apr Nov Chihuahua 74.05 75.36 79.16 77.03 44.58 54.17 0.61 0.64 Ciudad Juárez 69.06 68.81 56.12 73.19 35 41.76 0.49 0.57 Morelia 67.95 72.5 60.54 68.92 30.95 41.67 0.49 0.56 Tijuana 69.02 76.61 60.79 73.05 36.72 45.36 0.51 0.6 Guadalajara 61.09 69.30 44.52 68.53 32.55 41.33 0.43 0.55 Monterrey 86.83 71.52 71.64 59.5 36.22 34.25 0.60 0.51 Average 71.33 72.35 62.13 70.04 36.00 43.09 0.52 0.57 11 This function also evaluates the adoption of a gender perspective in implemented solutions. The scores were low across municipalities. 12 The LPS generates and disseminates information about crime and violence, as well as the strategic problems that affect a territory, the main affected populations, and the most effective practices to prevent violence. 13 The LPS has the capacities to design, finance, and implement evidence-based solutions in response to the problem of violence. Furthermore, it encourages the creation of spaces and mechanisms for the articulation of actors from different sectors. This axis also considers the adoption of a gender perspective. 14 The LPS encourages the evaluation of implemented solutions. In addition, it generates training and strengthening mechanisms, as well as spaces for advocacy in public policy related to the prevention of crime and violence. 24 Stakeholders' perceptions also support this finding. We asked the online survey respondents to rate different statements about JPV’s contribution to each of the functions of the local systems using a scale of 1 to 5, with 1 being totally agree and 5 being totally disagree. ● Regarding the create knowledge function, we asked stakeholders to rate JPV’s contribution to the following. 1. Identify, design, and implement programmatic high-potential prevention policies. 2. Local actors in the violence prevention system assign more resources to effective prevention strategies. 3. A diverse set of actors is more sensible about the importance of crime and violence prevention. ● Regarding the implement solutions function, we asked stakeholders to rate JPV’s contribution to the following. 1. Provide an elaborate diagnosis with disaggregated sociodemographic characteristics. 2. Have tools to identify and follow up with beneficiaries. 3. Improve capacity to monitor interventions. ● Regarding the improve practices function, we asked stakeholders to rate JPV’s contribution to the following: 1. Organization has developing capacities to carry out monitoring and evaluation (M&E) itself. 2. Facilitate cooperation with evaluation expert organizations. 3. The organization was able to influence policy during the last 4 years. For the first and second functions, the results are quite positive, with averages of 1.82 and 1.86, respectively. For the third function, the perception is less positive. For the first item, the average score is 2.02 and for the second is 2.25. Nevertheless, for these items, the anchor and strategic actors have significantly less positive impressions, with scores of 3.13 and 3.6 (leaning more toward disagreement). We deem that the difference in perceptions might be caused by the fact that anchor and strategic actors, overall, have more technical capacities than the rest of the sample, and, therefore, their vision is less optimistic. Also, there might be social desirability bias in these responses because actors self-report on their perceived evaluation capacities. Even though the improve practices function shows low scores in both the IRLSP and in our survey, and in particular on its evaluation component,15 from the comments we received from both the interviews and online survey, it is evident that JPV successfully managed to get organizations to internalize the need to have models informed by evidence and to evaluate their work. Also related to the third function, it is remarkable that 62% of stakeholders consider that their organization influenced policy (understood as the capacity to influence the decisions, actions, and policies adopted by the government). 15 The component of evaluation of implemented solutions had the lowest score in the IRLSP of all functions in three of the six regions, with scores between 7.03 and 12.5 of 25. 25 IV.1.1 JPV EXPENSES To better understand the implementation strategy pursued by JPV, it is worth looking at its expenses. By providing financial support and facilitating and sponsoring specialized trainings and consultancy services, JPV sought to promote the rise and expansion of effective interventions that could be escalated or replicated in different parts of Mexico. JPV implemented different instruments to inject financial resources into the systems: a donations program (grants) and varied subcontract instruments (subcontracts, purchase orders, and indefinite delivery–indefinite quantity [IDIQ]). Table 7 presents the distribution of expended resources, confirming the gradual implementation from the first to the third functions of the local systems discussed above. As can be observed, JPV assigned almost 60% of its resources to the first axis of create knowledge, 40% to the second axis of implement solutions, and less than 1% to improve practices. This figure only shows resources assigned directly to evaluation technical assistance or projects related to the third function, but other contributions were made in this regard. In terms of transversal strategies, the biggest proportion, 39%, was spent on supporting interventions for at-risk youth. The implementation of the CJHM received 15% of the investment. Of these resources, about half were assigned to matters of training and institutional reform for government implementation, while the other half was assigned to create therapeutic solutions to channel youth through this system. The gender strategy received 10% of the expenditures, and CPTED received about 1%. Of the total, 35% was spent on the general institutional-strengthening strategy. Table 7. JPV Expenditures16 (All Amounts are in MXN) Knowledge generation (technical assistance and training) Implementation of solutions Improve practices (evaluation) Total % of total expenditures At-risk youth 10,247,763 24,100,941 295,800 34,644,505 39% Civic justice 6,444,638 6,807,69017 - 13,252,328 15% Gender strategy 4,580,019 3,953,601 - 8,533,621 10% CPTED 350,000 610,250 - 960,350 1% Institutional￾strengthening strategy 30,657,04018 - - - 35% TOTAL 52,279,462 35,472,583 295,800 88,046,846 % of total expenditures 59.38% 40.29% 0.34 % Below, we analyze whether the implemented activities were able to achieve the expected result of strengthening LPSs. For this purpose, we analyze in detail the quality and scope of the implemented activities and whether they were sufficient to observe improvements in each of the functions of the local systems. 16 These numbers exclude expenditure toward IDIQ that are still open, a total of 32,848,498 MXN, given that we do not have information on what exactly has or will be spent. Most of these resources are for capacity-building and knowledge generation. 17 These are interventions targeted at at-risk youth that are derived from the civic justice system. There is some overlap with interventions that were initially developed for at-risk youth in other settings. We only classify in this cell when the intervention or adaptation was specifically designed for the Civic Justice Model. 18 This number includes tool development activities, trainings, and overall capacity-building for local systems. 26 2. What evidence is there that JPV programming has strengthened LPSs in a sustainable manner? a. To what extent are the local actors capable of maintaining the improved crime prevention policies, to the extent that there are improved policies, without USAID support? b. Have key avenues for reform and a reduction in crime and violence in the targeted communities been addressed through JPV programming and implementation? Why or why not? Box 3: Question 2 Summary For this question, we review the results achieved by JPV in strengthening local systems. First, in Sections IV.2 and IV.3, we analyze whether JPV was able to bolster local actors’ capabilities and promote the creation of multisector collaborations. Then, in Section IV.4, we analyze whether JPV contributed to the implementation of effective evidence-based policies by taking a closer look at the specific interventions implemented by local actors and their quality. Finally, in Section IV.5, we analyze JPV’s contributions to the scale-up of effective interventions and the challenges involved. Conclusion: JPV achieved positive results with the implementation of the transversal institutional￾strengthening strategy, but the targeting of activities was too broad and involved a lot of local implementing partners. This policy had an opportunity cost of gaining stronger results with possibly fewer organizations implementing more focused programs. Even though the program had notable success in increasing awareness and positioning the crime-and-violence-prevention agenda within the local and national systems, the results achieved in terms of generating sustainable, multisector, coordinated agendas were mixed. JPV succeeded at enabling multisector collaboration when prevention strategies aligned with interest groups, and government participation was obtained when related short￾term milestones were specified and followed. In terms of the implementation of evidence-based policies, we assess whether JPV supported interventions that followed the three principles of effective violence prevention found in the literature: concentration, coordination, and proactivity. We find that JPV technical assistance activities added to the concentration principle by supporting the development of civil society organization (CSO) programs with enhanced design, delivery, and coverage, although pressing challenges remain in terms of addressing youth risks and needs to prevent recidivism. JPV also contributed, but to a lesser extent, to the coordination principle by enhancing communication between CSOs and other relevant stakeholders, although major challenges prevailed with coordination with public institutions. The proactivity principle remains an area of opportunity for future USAID efforts. In terms of scaling up effective models, JPV was successful at leveraging different pathways to scale to expand two CBT models, ROLE and Mente Libre, and the CJHM. We identify that the focus from the beginning on scale, the provision of a meticulous strengthening process through access to overarching resources, and the generation of relationships of trust and co-investments with partners are key to achieving scale. Although JPV was very successful in supporting the adoption of these models, rushed expansion, absence of rigorous evaluations, weak technical capabilities in implementing organizations, and lack of consideration for differences in local circumstances posed challenges for model fidelity and effectiveness. 27 IV.2 JPV CONTRIBUTION TO INSTITUTIONAL CAPACITY-BUILDING The institutional-strengthening strategy was the initial tool used by JPV to approach actors and one of the main resources brought into the local systems by the activity. Given this and the significant amount of resources invested in this line of action, we start the local systems analysis by examining the results achieved by the mixture of capability-building efforts. The main achievement was increasing the awareness and positioning the prevention agenda within systems and at the national level. This was accomplished through the development of capacities to include a prevention perspective into local interventions and by organizing conferences, events, and study tours, which facilitated knowledge-sharing within and between systems. However, we find that the targeting strategy of these capacity-generating activities was too broad, which came with an opportunity cost of gaining stronger results with possibly fewer organizations implementing more focused programs. We provide a detailed analysis of these findings below. Through the implementation of a series of trainings and development and transferring of measurement instruments, JPV was very effective at helping a wide array of organizations to adopt prevention programming by introducing concepts and methodologies such as theory of change, evidence use for program design, risk factor diagnosis, and strategies for incorporating preventive approaches. Both the online survey and the semistructured interviews show stakeholder perceptions of these capacity-building tools to be quite positive. Table 8 shows that these tools successfully reached the majority of actors within local systems. It is remarkable that of all respondents, 84% know the trainings and 70% know the measurement instruments.19 As the numbers show, the usage of the tools is high. The most used tools are the trainings at 66% and the measurement instruments at 57%. Furthermore, of all respondents who indicated knowing each tool, 86, 58, 52 and 41% consider the trainings, measurement instruments, study tours, and events to be useful, respectively. Table 8. Knowledge and Use of JPV-Developed Tools Measurement instruments developed by JPV (YTT, GOPI-P, OPI￾P, EBP) Trainings Study tours Events/ seminars Stakeholders who know the tool 70% 84% 20% 53% Stakeholders who use the tool 57% 66% 8% 31% Usage of those who know the tool 81% 78 % 40% 58% Stakeholders who believe the tool is useful to design and implement better practices 58% 86% 52% 41% 19 The fact that only 20% of actors know the study tours is expected because this activity was only targeted toward specific actors. 28 JVP developed four main measurement instruments, the Organization Performance Index— Prevention (OPI-P) and the Best-Practices Scale (EBP being its acronym in Spanish) to work with CSOs, the Government Performance Index—Prevention (GOPI-P) to work with governments, and the YTT for general application (see Box 4 for a description of these tools). We find that the development of these tools is one of the greatest JPV contributions and that the provision of technical assistance, along with their application, impacted organizations in a positive way. Furthermore, JPV pursued a strategy to transfer these tools to local actors, which we regard as outstanding because it allowed the program to reach more organizations while ensuring sustainability. Anchor and strategic actors agree that these instruments are technically sound and easy to use, which makes them accessible to a wide range of stakeholders. Below, we review the results achieved by the GOPI-P, OPI-P, and EBP. The YTT is analyzed in Section IV.4. Work with governments for capacity-building started early in the program (2016–2017), which allowed for a medium-term perspective to implement capacity-building strategies. Annex III presents the GOPI-P results across the life of the program. Overall, the municipalities supported by JPV increased their capacity by 50%, 20 which is remarkable. During the interviews, we also identified that the application of the GOPI-P instrument helps increasing awareness within diverse government institutions about the relevance of crime-and-violence-prevention work and about the need to coordinate prevention strategies. As one stakeholder expressed: “…now the different institutions within the municipality know each other and know who does what. Previously, they were like islands. This would have been enough as an objective itself." From the interviews, it also became evident that the application of the tool, accompanied by continuous and reliable technical assistance, helped boost the government's capacities. Beyond the anchor actors in Chihuahua, Ciudad Juárez, and Guadalajara, who have fully adopted the implementation of GOPI-P, other local actors have also adopted the tool to reach more municipalities. Mesa Metrópoli, a strategic actor in Monterrey, has adopted the tool to implement in 13 additional municipalities of the metropolitan area, and the Jalisco State Center for Social Prevention has adopted the tool and will implement it in 80 municipalities. Our analysis shows that the work with the civil society sector was also very successful at increasing prevention awareness and generating capacities, although the targeting strategy for the provision of technical assistance appears to have been unclear for some tools. In collaboration with local anchor and strategic actors, JPV applied a total of 70 OPI-Ps to CSOs and provided technical assistance to a subset of them. As presented in Annex III, of 24 organizations that received technical assistance through application of the OPI-P, only 9 improved their score. Of 24 organizations that applied the EBP tool and received technical assistance, 20 showed improvement. The EBP, unlike the OPI-P that covers organizational and technical capacities, is focused on the quality of evidence for specific prevention interventions to identify effective and scalable models. 20 The municipalities that have had the greatest improvements are Monterrey (155 %), Ciudad Juárez (88%), and Chihuahua (86%). Conversely, Guadalupe and Tonalá showed a decrease in their capacity (–9 and –2%, respectively). JPV explains that this is caused by government changes. 29 Our analysis finds that increasing organizational capacities is an intensive endeavor, especially given the low existing levels as confirmed by the OPI-P results. 21 Therefore, working with organizations in narrower matters and with more focused interventions, as with the EBP, may be more effective. Moreover, the results from the EBP show that funding for a specific intervention, accompanied by close relationships and technical support, can really boost organizations’ capacities. All of the organizations that received funding and technical assistance through application of this tool improved their scores. In this regard, it is noticeable that only three of the organizations that applied the OPI-P received funding through the donations program, and only one organization has both the OPI-P and the EBP. This broad targeting strategy is also reflected in the implementation of trainings. Of the online survey respondents, 65% believe the content to be robust and 74% believe it to be relevant. However, we also identified a shared perception that, in order to achieve better results, trainings must be followed by more intensive assistance; otherwise, learnings are hard to internalize.22 Materializing training learnings into measurable changes within organizations is a medium- to long￾term process that requires more intensive work. It appears that JPV pursued a diversified strategy to work with organizations at various capacity levels and tried to reach several organizations within the LPS. Although the work to increase capacities is always valuable in itself, considering the value for money of these investments is important. A broad targeting strategy raises questions about potential opportunity cost to having had more focalized efforts to achieve stronger results in the substantive transversal strategies. It is important to reiterate that the evaluation does not find that the resources invested in capacity-building were misspent. The provision of organizational capacity-building activities is particularly important in the Mexican context, where often resources are lacking and baseline capacity levels are quite low. Furthermore, several organizations consider that the use of tools provided by JPV gives them legitimacy, not only because they expand their internal capacity to develop evidence that supports their work and strategies to communicate it, but also because these tools are implemented by external and expert entities, which gives them more credibility. As one interviewee stated: “There is no longer an excuse to not ‘register’ models. Now it is possible to present our work to the government and businessmen with clear results thanks to the systematization we have gone through. We are no longer only organizations with good intentions, we now have serious projects.” There is always a tradeoff between reaching more organizations versus strengthening a few while achieving stronger results. We believe that making this tradeoff transparent is important, particularly given the amount of resources that were assigned to institutional capacity-building. Overall, given the potential of the tools, we deem that more focalization could facilitate stronger results for the implementation of specific interventions. We discuss this in more detail in Section IV.4. 21 In the last measurement of the OPI-P, only one organization was classified as high-level capacity, 11 have a moderate capacity, and 12 exhibit a low or basic capacity level. We include the baseline level of organizations that only have one measurement. 22 These numbers also include government-representative and private-sector perceptions about the trainings. 30 Finally, JPV’s openness and willingness to discuss findings, have a dialogue, and adjust the technical assistance strategies according to organizational needs is noteworthy across tools. There is a general perception that the comments and feedback provided by JPV is valuable. More than 80% of the stakeholders who answered the online survey expressed this as such. Furthermore, 73% of stakeholders agreed that the modifications done to their programs or policies in response to JPV’s recommendations have helped them increase their effectiveness. Box 4: GOPI-P, OPI-P, EBP, and YTT The GOPI-P is aimed at measuring and evaluating the state of violence prevention capacity within municipal governments. Through the application of this tool, an institutional diagnosis is developed, identifying strengths and areas of opportunity. The results are delivered to municipal governments and include timely and feasible recommendations to increase capacities. The tool comprises five categories: 1) cooperative government-society relationship, 2) evidence-based operational and administrative prevention tools, 3) services and specialized public servants, (4) municipal regulatory framework, and (5) information and communication systems. Based on the results, the GOPI-P categorizes government capacity in four categories: willing to participate (0–49), basic (50–74), acceptable (75–89), and superior (90–100). The GOPI-P was used by JPV to approach local governments since the beginning of the program and was applied in all municipalities where JPV worked. The OPI-P is aimed at measuring the institutional and technical capacities of CSOs that work on crime and violence prevention. Information gathered from this tool serves as a baseline for measuring changes in capabilities in the medium and long terms. Additionally, it supports the definition of strengthening plans that facilitate the organization's transition toward a results-based approach based on the generation of evidence. Through its variables, the tool promotes a theory of change that establishes adoption of internal changes and institutional policies that favor the organization’s performance in the field, positively impacting the places or people it tends to benefit. The tool evaluates 22 variables grouped under the following categories: governance, financial and administrative management, human resources, donations management, strategic planning, and technical capacity in violence prevention. It classifies organizations into four capacity groups: low, basic, moderate, and high. The EBP is aimed at identifying the use and generation of quality evidence in violence prevention programs with the ultimate goal of promoting evidence-informed policies. It analyzes the quality of evidence that backs up crime-and-violence-prevention interventions to identify programs that can be replicated or scaled up. The tool evaluates four categories: 1) theoretical background—analyzing whether the program has a design instrument and indicators, as well as whether it is based on existing evidence; 2) impact—analyzing whether the program has an evaluation, its quality, and results; 3) implementation quality—analyzing implementation performance and fidelity; and 4) alignment with prevention interventions that have been identified as effective in different contexts. It classifies interventions into four categories: not informed by evidence, informed by evidence, promising practice, best practice. The YTT is aimed at assessing the risk profile of youth participating in CSO programs with three purposes: 1) ensuring that CSOs are focalized in high-risk youth, 2) having a baseline to determine individualized plans to intervene with youth, and 3) monitoring youth progress during participation in the program. The tool draws from the Structured Assessment of Violence Risk in Youth (SAVRY), along with recommendations from specialized Mexican institutions and organizations. Making use of open￾ended questions and a scoring scale, the YTT assesses the following: 1) overall risk of youth, 2) overall 31 exposure to violence of youth, 3) propensity to perpetrate violence of youth, and 4) (when applied in group sessions) overall situation of a youth collective. The YTT implementation strategy relied on customizing it to CSO and public-institution programs that are directly working with youth and on training operators who, according to the tool manual, should be experienced in community processes with youth, providing psychological support to at-risk youth, working with youth in conflict with the law, and administering evaluations to determine youth needs. Beyond generating basic organizational capacities through the application of the measurement tools, JPV organized conferences and seminars with the aim of positioning the crime-and-violence￾prevention agenda in the public discussion, as well as creating awareness of evidence-based interventions and disseminating technical knowledge. The study tours, which were targeted at specific key actors, were organized to share experiences between regions and to facilitate knowledge transfer. Although only 41% of online survey respondents believe that the events were useful, from the interviews, we learned that the events and forums organized by JPV were successful in their objectives. First, they had an impact in generating pressure on governments to change the way they conceptualize crime-and-violence-prevention work and demand strategy changes. Second, they helped position different organizations’ work without atomizing the dialogues of individual demands. Third, they helped local organizations meet peers who implement internationally recognized models, learn from their experiences, and face challenges. Our analysis finds that these events are a tangible way of bringing organizations closer to experts. A good example of this work was the event organized around CBT, which gave organizations the opportunity to hear from implementing experts in Chicago and Central America, as well as to learn from leading academics. As one stakeholder put it: “Those events were incredibly helpful for us. We could explore different aspects and understand the CBT implementation perspective from leading organizations, such as the World Bank or universities. ” As expected, there is more work to be done to have authorities commit to changes and the adoption of regulation modifications, but JPV’s approach to inclusive events in Mexico City and in each municipality helped advance the conversation. In Section IV.4 there is a more detailed description of the technical content of these events. IV.3 JPV CONTRIBUTION TO THE GENERATION OF MULTISECTOR PREVENTION AGENDAS To implement the local systems framework, JPV worked to build multisector groups with shared crime and violence agendas within the regions and focused its efforts on identifying and working with diverse actors to enhance their prevention work. The program prioritized strengthening local authorities’ commitment to tackle crime and violence, increasing CSOs’ programming capacity, and continuing engagement of the private sector in violence prevention. Also, JPV opted to create anchor actors and strategic actors, with the objective of ensuring sustainability by empowering these actors to become articulators of collaborative efforts and to help other members develop their capacities. In this section, we analyze JPV’s contribution to building institutionalized multisector groups and how the program was able to influence key actors, their roles, and their relationships and to promote sustainability and local ownership. 32 We find that JPV enabled the coordination of sustainable multisector groups only when focused on the implementation of a specific prevention strategy. Relatedly, government participation was fostered when short-term milestones were provided. The strategy of having anchor actors was accompanied by several challenges that made its success limited, mainly given that the program failed in empowering local actors to lead the agendas, with relevant exceptions. In this regard, we find that the conceptualization of the anchor actor, while attractive, lacked appropriate coordination and application. Our analysis shows that this role did not exist or was not necessary in all systems and that working with diverse leadership is preferable. We also find that JPV had a very interesting approach to involving the private sector in the prevention agendas, and although the co-investment of the sector was not considerable, JPV did manage to enable this sector to act as a key facilitator of local agendas. In terms of the civil society sector, JPV achieved remarkable results. It empowered the sector and placed it as a change￾generating actor by recognizing the innovative and programmatic value brought by CSOs to the systems, enhancing it, and creating spaces for organizations to showcase their work to the public and private sectors. Finally, we find that while JPV’s organizational structure with a regional team was highly effective in creating local buy-in and satisfaction by providing actors with local counterparts to discuss projects and ideas, the hierarchical structure of USAID, which reflected into JPV’s, negatively impacted local ownership. We discuss these findings in more detail below. IV.3.1 MULTISECTOR GROUPS AND GOVERNMENT PARTICIPATION As mentioned above, one of JPV’s greatest accomplishments was placing the crime-and-violence￾prevention agenda as a priority in the national and local discussions and articulating diverse actors to work together on the topic. However, as far as contributing to the development of institutionalized multisector groups with clear shared prevention agendas and working plans, the results are mixed. Even though JPV considers Chihuahua and Tijuana to be success cases, we find that the latter presents sustainability challenges. While efforts were initiated in Guadalajara and Ciudad Juárez, these didn’t start until 2019, so it is premature to talk about institutionalization. In Monterrey, several relevant actors were invited to the multisector group, but coordination remains a challenge. Finally, while in Morelia JPV did not pursue the strategy to develop a local system as in the other regions, an articulated multisector group has evolved around implementation of the CJHM. Annex IV presents a more nuanced analysis of the evolution of each system in a comparative matrix. Overall, we conclude that the most successful cases of multisector collaboration are those that have one of the following three attributes. 1) Groups that focused, at least in origin, on specific strategies rather than on developing broad crime-and-violence-prevention agendas. Our comparative analysis shows that articulation between actors happens faster and in a more natural way when articulated around a specific strategy. Morelia and Escobedo, with the implementation of the CJHM, are great examples, but so is the case of Chihuahua with the CPTED implementation. Moreover, our analysis shows that groups focused on specific strategies rather than on developing broad crime￾and-violence-prevention agendas with ample menus, such as Tijuana and Monterrey, were able to better attract interest from the public and private sectors. As two anchor actors mentioned: 33 “The temptation in [crime and violence] prevention is to try to solve it all at once, and you end up solving little. It is better to give precision shoots.” “…it is necessary for the objectives to have a greater degree of focalization with a clear accompaniment so certain processes are easier to address.” 2) Groups that had short-term milestones attractive for government participation. Mayoral terms last only 3 years. It is important to consider enough time for politicians to be able to capitalize on successful interventions to get their buy-in to implement promising strategies. The prevention policy alternatives that can be implemented at the local level are not always clear, and public officials do not fully commit to their importance until they have the opportunity to get involved and understand their effects. Having champions within the government is a condition that enables government buy-in, but it is not enough. The cases of Morelia, Escobedo, and Chihuahua show how JPV was able to leverage government interest by providing specific milestones on which politicians could capitalize. While changes in government are always a challenge, these two attributes together, specific strategies and short-term milestones, contribute to the probability of surviving administration changes. By showcasing achievements, the political costs of abandoning promising strategies are high. The case of Morelia reveals important lessons in this regard. A clever communication strategy was used to convince the incoming government of the benefits of the CJHM, and a swift training plan was offered to keep the model in operation. The message was simple and clear, using anecdotes and available indicators, and there was a clear plan ahead. 3) Groups that aligned agendas with specific interest groups. Our analysis shows that in bigger metropolitan areas, such as Monterrey and Guadalajara, or in areas with more complex crime and violence scenarios, such as Ciudad Juárez, organizing the crime-and-violence-prevention agenda under a unique group is not necessarily desirable or feasible. In more complex areas, there are several interest groups with diverse agendas, and, therefore, forcing them to collaborate under one umbrella may be impractical. As one actor put it: “At the beginning the effort in Jalisco did not work because it started as an up-down collaboration trying to set an agenda that cannot be agreed. There should be a local vocation for things to work better. There should be incentives for people who are making decisions. That is more practical. Once that model is consolidated and you find the means, then it is possible to think about a public policy.” The results of these early efforts were groups without clear responsibilities and with coordination problems. For instance, JPV found it very challenging to develop an organized crime-and-violence￾prevention agenda in the Monterrey metropolitan area, and it was only able to articulate actors for the implementation of the CJHM through the last year of the program. The multisector work in Guadalajara only took off when organized around the implementation of the CJHM and reintegration strategies. In the case of Ciudad Juárez, different multisector groups have evolved around gender and social reintegration strategies. JPV conceptualized the systems work as a single group, but the experience shows that in complex agendas such as crime and violence prevention, a system can be composed of different groups that implement distinct strategies. For example, in 34 a metropolitan area, civic justice will be more successful with buy-in from state authorities, while territorial strategies for at-risk youth may need stronger leadership from local authorities or CSOs. We deem that, in some cases, JPV faced an opportunity cost while trying to organize too many actors with different incentives under one agenda. The takeaway from these attributes is that the prevention agenda cannot always be clustered within a single group and that specialization is desirable while facilitating communication channels. The contribution of a program such as JPV is better suited to technically strengthen specific working groups and facilitate communication among them while working with different leadership entities and pursuing differentiated strategies. In sum, the above attributes mean that the construction of agendas must evolve around specific strategies with a clear technical offer, defined target populations and indicators, and clear responsibilities and goals. One local system may have more than one agenda, and its construction must depart from clear areas of specialization, such as at-risk youth, gender strategy, civic justice, or police work. In fact, this is desirable. While there may be an overlap of actors among these groups, particularly from the government, evidence shows that it is more practical to coordinate different specialized working groups, where the government may participate in more than one, than to have a wide array of actors sitting at the same table trying to coordinate varied agendas. In terms of generating government buy-in, an important contribution of USAID is helping governments understand the value of working with at-risk youth and reintegration strategies. One of the biggest challenges faced by USAID at the local level is the lack of understanding and resistance of public officials toward a vision of prevention and reintegration, as these topics are still handled from a public safety rather than a public health perspective. 23 This agenda requires specialized and intensive services for specific groups, and aligning incentives with the government is particularly challenging because results are expected in the medium to long term and the number of beneficiaries is relatively low. The risk of not having specific interventions that can serve as an example for the government is that politician intentions remain in broad agreements that do not translate into operational changes. As several CSOs expressed during our interviews, the tangible and daily commitment of authorities, reflected in the will to invest in and facilitate specific interventions, should be the focus of this type of program. USAID developed a microeconomic model to estimate criminality costs. In Mexico, the economic and social costs of the criminal life of a young person exposed to different risk factors would total 27 million pesos. Considering a series of interventions with a hypothetical value of 2 million pesos, this report estimates the cost-benefit of investing in prevention and finds that for each peso invested in prevention, the long-term return can be 11 times greater. 24 The biggest opportunity is for governments to invest in this type of program. USAID can influence government priorities and promote co-investment when engaged with the municipality. Municipalities look for USAID prestige, and this should be leveraged as a vehicle to generate new ways to work with communities. 23 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 24 USAID, “Executive Summary: Estimation of the costs of crime and violence for the private sector in Mexico and the importance of its participation in violence prevention as a strategy for its long-term business model”. First Published June 2019. 35 IV.3.2 ANCHOR AND STRATEGIC ACTORS Following the evidence suggesting that strong and adaptive systems have actors playing roles as stewards, facilitators, brokers, knowledge hubs, networks, and advocates (5R framework), JPV decided to implement the roles of anchor actor and strategic actor as a key component of its local systems strategy. The objective was to ensure sustainability of their efforts, with these actors taking on the facilitator role in collaborative crime-and-violence-prevention efforts. 25 ● Anchor actor is the actor who articulates the members of a system around a common goal or agenda, taking into account the roles, relationships, resources, and rules that operate within a system. ● Strategic actor strengthens capacities and functions of members of the system through the use of JPV-developed tools. The data gathered during this evaluation show that the strategy of having anchor actors was accompanied by several challenges that limited its success. While the experience varies by region, the overall perception is skeptical. We identify four factors that impacted this result. First, contracts with anchor actors were not signed until late 2018, which left them with a very tight timeline, but with very ambitious goals. Second, we find that JPV did not take full advantage of the capabilities and knowledge had by local actors as prescribed by the local systems perspective. In general, anchor actors were not given the opportunity to lead the development of the local agendas. The mapping of relevant actors, decisions on whom to convene with, which institutions to offer technical assistance to, and how to build local agendas (thematic focuses of the working groups) were heavily influenced by JPV—and JPV’s perspective seemed to be above that of the local anchor actors. In terms of the strategic actors, we find similar perceptions. JPV, not the strategic actors, decided which organizations were to receive technical support and for what purpose. In fact, these stakeholders shared that this selection often was at odds with local realities. From the interviews, it appears that JPV made decisions on which organizations to support based on the goals it wanted them to achieve, but not in terms of their baseline capacity or area of focus. The consequential pursuit of certain goals was, therefore, unrealistic in some cases and involved a high opportunity cost. Third, the rules that governed how the anchor and strategic actors took on their roles failed to empower local actors to lead the articulation of agendas. JPV offered subcontracts, which, by definition, are conceived by the sponsor and are relatively inflexible as to scope of work, budget, and other changes. Rather than providing financial support to achieve a public purpose, subcontracts are binding agreements between a buyer and a seller to provide goods and services in return for monetary compensation.26 Furthermore, these contracts have restrictions for the use of providers’ brands, which creates disincentives for sustainability and diminishes the leadership role. The role of subcontractor diminished the decision power of anchor and strategic actors to define strategy and invest in activities that they felt were the most promising and, therefore, more sustainable. Points two and three are supported by data from the online survey. When asked about the distribution of decision power for program implementation on a scale from 0 to 100, with 0 25 JPV’s Annual Report 2017-2018. 26 See Office of Research, “Federal Funding (Grants vs. Cooperative Agreements vs. Contracts)”, UC Santa Cruz. 36 meaning more power concentrated in JPV and 100 meaning in the local organization, anchor and strategic actors reported an average of 37. Interestingly, the average for all other actors is 59. Furthermore, the online survey reveals that, indeed, local actors still perceive JPV/USAID as the articulation leader of the local prevention agenda. Of the total respondents, 73% believe that clear leadership exists to convene actors working on crime and violence prevention. Of these, 39% believe this leadership to come from civil society, 17% believe it to come from the local government, and an important 35% still believe it to come from the international cooperation sector, understood as JPV/USAID. The perception of JPV as leader is particularly high for Ciudad Juárez, Chihuahua, and Tijuana, with 47, 40, and 39%, respectively. Fourth and finally, the selection of anchor actors may have forced organizations to take on several roles that may not be part of their nature. Some anchor actors, such as in Guadalajara, Monterrey, and Tijuana, were selected because of their role in the local systems, but not necessarily because of their technical expertise in prevention work, which may have complicated the evolution of the agenda. Moreover, from the interviews, we also find that other local actors questioned the designation of this role in some regions. In Guadalajara and Monterrey, the anchor actors have been more successful in institutional capacity-building (through the implementation of GOPI-P, OPI-P, and provision of technical assistance)27 than in the articulation of clear prevention strategies. The only case where the leadership appears natural was in Chihuahua and Ciudad Juárez. Fideicomiso para la Competitividad y la Seguridad Ciudadana (FICOSEC) is a business organization with years of experience in security work and with funds to invest from a local trust. We believe this to be the only case where the anchor actor was able to fulfill its purpose. Whereas the case of Tijuana is generally considered a success, our analysis suggests caveats. While this actor thrived in bringing together a wide array of actors to work together on the prevention agenda, it was less successful in providing substantive technical leadership in prevention work. Moreover, given that this local system was not able to generate much co￾investment from the private or public sector, the sustainability of this role still seems to be highly subject to JPV. In short, the conceptualization of the anchor actor, while attractive, lacked appropriate coordination and application. Our analysis shows that this role did not exist or was not necessary in all systems. In more complex systems, such as metropolitan areas, there are several actors that play the role of articulators and leaders for different strategies. The existence or ability of one single actor who is able to articulate all the others seems to be at odds with the natural functioning of the local systems and may have the unintended consequence of creating power dynamics that contradict its objectives. As the 5R framework explains, the identification of the “to-be” system needs to be done in collaboration with local actors, and although there is little evidence, it hypothesizes that strengthening existing roles is probably more feasible than creating new roles. 28 We believe that certain cases would have been more successful if JPV had focused its efforts on strengthening diverse leaderships and leveraging its comparative advantages. In fact, as mentioned above, the evolution of the local agendas shows that they emanate from natural leadership and from organizations pursuing specific interventions. 27 In Guadalajara, the anchor actor was responsible for implementing the GOPI-P and OPI-P and providing technical assistance. In Monterrey, the anchor actor was only responsible for the OPI-P and provision of technical assistance, whereas a different strategic actor was in charge of the GOPI-P and respective technical assistance. 28 See “The 5R Framework in the Program Cycle”, USAID, 2016 (p-13). 37 IV.3.4 PRIVATE SECTOR In Mexico, the philanthropy and social responsibility agendas are incipient and exhibit low levels of professionalization, making the involvement of this sector particularly challenging. Nevertheless, even though the co-investment of the sector in the violence and prevention agenda was not considerable, JPV did manage to enable this sector to act as a key facilitator of local agendas. In particular, buy-in and support from different business organizations and companies were key to achieving the socialization of initiatives promoted by civil society and to contributing to permanence of initiatives through government transitions. In this sense, JPV was able to build a relationship with the private sector that enhanced its role within the system as an influential actor and increased its participation in the prevention agenda. The online survey shows that 54% of actors believe the private sector to have increased its participation in this agenda. The most successful case of private sector involvement and leveraging co-investment was in Chihuahua and Ciudad Juárez with FICOSEC, which, as mentioned above, was a natural choice to also become the anchor actor. JPV succeeded in generating alignment with this actor and leveraged its resources, both financial and in terms of leadership, for advancement of the prevention agenda. In fact, the state of Chihuahua concentrates 68% of the co-investments able to be leveraged by JPV from the sector, with a total of 5,777,907 MXN. From the total 8,540,318 MXN leveraged by the program, the next-highest amount (20%) was leveraged in Mexico City. This means that only 13% of the total resources leveraged from this sector were from the rest of the regions. Nevertheless, JPV was successful in capitalizing on the private sector’s stature and enhancing its participation as an influential actor. For example, in Morelia, the Confederación Patronal de la República Mexicana (Commission of Large Enterprises of the Mexican Republic) (COPARMEX) took part in the civic justice system by offering some of its corporate social responsibility programs as alternative measures to justice. Then, this group also became an advocate, which facilitated the continuation of civic justice after the government transition. In Monterrey, Supera became an ally to advocate for at-risk youth interventions. In Tijuana, the sector became an advocate for increasing organizations’ visibility and strengthening community cohesion around violence prevention by generating convening opportunities. In one forum29 organized by JPV on the role of the private sector, one actor expressed: “Decisions about prevention policies should not be only political, but should be sustained in evidence. The private sector can act as a critical mass to increase citizen participation in these processes.” One additional learning, and similar to the public sector engagement, generating interest from the private sector is more successful when involved in specific strategies. Such are the cases of Supera involved in an at-risk youth strategy in Monterrey, BBVA and Grupo Ruba involved in CPTED and gender strategy, respectively, in Tijuana, and COPARMEX involved in civic justice in Morelia. Finally, JPV supplied an important contribution in showcasing a model of local trust in Chihuahua as an avenue to direct resources to the prevention agenda in other regions. These efforts were particularly influential in Morelia, where a similar trust was created and discussions began 29 USAID Mexico, Forum “The role of the private sector as part of the security and violence prevention strategy at the local level”, June 24, 2020. 38 happening in other regions such as Guadalajara and Tijuana, although these efforts have not been materialized. In the same forum, one actor from the private sector in Morelia mentioned: “We had the opportunity to know the models in Chihuahua, and thanks to this we had light on how to finance prevention efforts. USAID accompaniment is key to generate confidence to launch these types of trust models.” IV.3.5 CIVIL SOCIETY In this portion of the analysis, we focus on understating how JPV empowered the role of the civil society sector overall, and in Section IV.4, we study in more detail JPV’s contribution in increasing its programmatic capacities. Through the local actor mappings, events, and multisector group discussions, JPV managed to give entrance to discussion from CSOs that otherwise find it hard to be heard. Moreover, JPV brought legitimacy to its work and placed them as change￾generating actors. JPV succeeded in recognizing the innovative and programmatic value that CSOs bring to the systems and enhanced their empowerment by facilitating the catalyzation of innovative interventions with technical and financial support and by creating spaces for organizations to showcase their work to the public and private sectors. The online survey confirms this finding using a scale of 1 to 5, with 1 being low and 5 being high, on which stakeholders rated how positive they believe JPV’s influence to be on different sectors in helping them improve their prevention practice. The highest rank was for civil society with 4.32, with an average of 3.62 for the rest of the sectors. For all regions the highest rank among sectors in terms of JPV’s influence was for civil society, except in Morelia, where it was ranked third after academia and local government. This result is both expected and illustrative of the challenges faced by this region. While Morelia is a success story in terms of government buy-in, the civil society sector is quite debilitated. Box 5 presents a comparison between the regions of Morelia and Ciudad Juárez, which were the two extremes in terms of government and civil society empowerment, with Morelia being highly successful in working with the government but less so with the civil society sector and Ciudad Juárez being the other way around. The case of Morelia shows that a successful local system is only such to the extent that it also involves strong CSOs, and this is particularly relevant for implementation of the CJHM. Box 5: Case Study—Ciudad Juárez and Morelia Comparative analysis of Ciudad Juárez and Morelia reveals important insights into the relevance of applying the local systems framework to specific strategies, particularly regarding the importance of multisector collaboration. In these two regions, JPV did not have a clear plan to build multisector groups from the beginning,30 but the agendas evolved naturally around specific strategies. These cases present advantages and challenges that are two sides of the same coin. In Ciudad Juárez, JPV was very successful at increasing CSO capacity to implement reintegration strategies, but lagged in creating more public services for at-risk youth. Conversely, Morelia is a success story in terms of government collaboration for implementation of the CJHM, but lagged in generating capacities for CSOs to collaborate with the government to support youth. 30 From the interviews, we learned that Morelia was not considered a priority within the program until civic justice gained traction, and in Ciudad Juárez, there was not even a regional coordinator until 2018. 39 These cases show that focusing on a particular strategy is desirable to increase the delivery capacity of local actors, but also offer insights on the risks of not envisioning a clear coordination strategy between sectors to fulfill the strategy goals. In Ciudad Juárez, JPV focused its efforts on strengthening local CSOs working on reintegration strategies. This region received the biggest proportion of the donations program, which empowered CSOs to organize and form a Social Reintegration Committee. In collaboration with JPV and the anchor actor, this committee is working to influence public policy.31 Given the great work with CSOs, JPV leveraged 12,507,043 MXN in co-investments from the civil society sector in this region, representing 53% of the total leveraged from the sector across regions. Nevertheless, the program did not engage with the government to strengthen public services for at￾risk youth until late 2018, achieving poor results in this regard. This region received the lowest amount from public sector co-investment.32 The case of Morelia is a success story in terms of catalyzing institutional change through the implementation of the CJHM. It comes as no surprise that the co-investment from the government for this initiative was the second highest.33 However, there was very little work with CSOs, threatening the effectiveness of referral processes of the CJHM and casting doubt on its sustainability. 34 The region received one-quarter of what Ciudad Juárez received from the donations program, and no resources where leveraged from this sector. JPV did implement strategies to address these challenges once the need to create synergies between the different sectors for the provision of specific services targeted at at-risk youth became clear, although these strategies were implemented toward the last one-third of the life of the program, leaving little time to achieve stronger results and ensure sustainability. IV.3.6 CONSULTANTS, THINK TANKS, AND ACADEMIC INSTITUTIONS As part of implementing the local systems framework, JPV sought to gain substantive knowledge on specific topics from consultancies, think tanks, and academic institutions. The interviews revealed mixed experiences. Our analysis shows that smaller and locally based consultancy firms and research institutions that collaborated with JPV to develop specific tools had very positive experiences. In fact, beyond being grateful for the opportunities provided, they expressed that through JPV, they had a unique opportunity to influence policy. Nevertheless, the experience with external actors, bigger organizations, Mexico City–based organizations, and international organizations is less positive. Our analysis shows that, in these cases, JPV failed to take full advantage of their knowledge and expertise. Stakeholders’ perception is that these relationships were deemed as buyer-seller to cover one specific service requirement rather than being partner relationships building toward the construction of incremental agendas with iterative interactions. In particular, we find that organizations with established reputations and their own agendas found it difficult to collaborate with JPV. Two circumstances led to this result. First, there seemed to be an urgency to meet certain short-term goals, which led JPV to fail to consider expert advice 31 Replicating an initiative that was born in Chihuahua and with the support of the local court, various government agencies and civil society actors participate in this interinstitutional working group and contribute to the social reintegration of adolescents in conflict with the law. 32 These minimal resources can be viewed in the comparative Municipalities Matrix in Annex IV. Moreover, these resources come from the Strengthening Security Program (FORTASEG being its acronym in Spanish), which is a federal subsidy for municipalities for security-related policies. 33 The highest co-investment is in Zapopan for the implementation of CPTED. 34 We will study the challenges involved in the implementation of the CJHM in Section IV.5. 40 or to let go of promising initiatives. Second, the co-existence of brands and intellectual property of the products developed by relevant actors is an additional challenge that emerged from the subcontracts used by JPV and that creates disincentives for continuity and appropriation. In this regard, we also find that, given USAID regulations, consultants cannot share which projects they are working on, hindering transparency and public discussions. As a consequence, the possibilities to leverage the knowledge of different firms is diminished. As one interviewee put it: “USAID presents the results of reports without any type of peer review or opportunities for feedback, so there are no technical opportunities to improve ideas.” In order to truly advance cooperation, advance ownership, and build lasting relationships, activities such as JPV must relax these restrictions and allow actors to retain intellectual property of generated products. Otherwise, we find that the products will always be at odds with sustainability. IV.3.7 JPV ORGANIZATIONAL STRUCTURE One of the greatest successes of JPV was the establishment of a regional team. From our interviews, we learned that most of the successful stories and satisfaction with the program derive from the work of this team. Stakeholders express the relevance of having this local counterpart to discuss ideas, concerns, and progress. Moreover, they highlight the openness of the staff to find solutions to any circumstance and their commitment to make their voices heard with the JPV central office and USAID. We received a wide variety of comments highlighting the technical preparedness and local knowledge of the regional staff, and the majority of stakeholders praised them as one of the most relevant pieces to bring the local systems alive. Examples include: “In our region, it was clear that the regional coordinator had a profound understanding of the issue and community to promote the agenda.” “The communication with JPV was good, mainly through the regional coordinator who included us in all discussions.” “We expressed some concerns about the need to further adapt GOPI-P to our circumstances. The regional coordinator was always open to work in good harmony. Furthermore, although it was not always possible for him to be on top of everyone’s progress, he was always open to discuss feedback and recommendations with us.” One actor claimed that through this model: “JPV found the key to make international cooperation sustainable.” One area of opportunity we find is the need to better align the technical work with the local systems team. These agendas seemed to have evolved in parallel at the beginning of the strategy implementation, although they merged naturally as the program advanced. We learned that regional staff did not always take part in technical decisions, but JPV understood not to detract from a potential lack of very specific technical expertise when central teams lack awareness of 41 the local context. In fact, we deem the success of several strategies to be attributable to the outstanding work of regional coordinators and to their previous professional relationships, which all of them leveraged to advance JPV’s agenda. For future projects, we deem that more empowerment of regional staff would supply potential for an even-more-local systems strategy, and we consider it imperative to give continuity to their work. IV.3.8 USAID As a USAID contractor, JPV decisions are restricted by USAID objectives, requests, and opinions. We find that one constraint for the application of the local systems framework is USAID’s hierarchical structure and dynamic with its contractors. The same logic of buyer-seller relationships that we described above exists within USAID. Contracts with activity implementers are regulated under acquisition procedures, through which the agency contracts out services to achieve certain desired outcomes. Under this arrangement, USAID typically exercises a higher level of control over the partner in obtaining results.35 While naturally some degree of supervision is needed, we find a tension between USAID’s management schemes and the enhancement of locally led development. First, USAID has direct control over high-level relationships at the local level. While involvement of high-level officials is desirable to further key partnerships, coordination and articulation are also needed with central and local actors. We found instances where JPV staff were restricted from having direct contact with or managing certain relationships, as well as instances where partners were requested to avoid direct contact with USAID because of their subcontractor role with JPV. Second, USAID has other crime-and-violence-prevention efforts in the same regions and, in some cases, with the same actors who are managed outside of JPV’s local systems agenda. This is confusing for local stakeholders and prevents the consolidation of local systems. We find this case to be particularly relevant in Monterrey, where USAID directly manages high-level relationships with the private sector and leads parallel agendas such as Focused Deterrence or Red SumaRSE. Given the lack of clear communication mechanisms, we believe this to have harmed the development of the agenda in this region. The existence of parallel relationships and agendas creates a forked agenda development, creating opportunity costs for the local system to leverage outputs and, in turn, to ensure intervention sustainability. The 5R framework reflects the principles of the collaborating, learning, and adapting (CLA) framework set forth by USAID to improve its development effectiveness. 36 One key aspect of this framework is the adapting piece, and evidence suggests that managing adaptively is more likely to improve outcomes when decision-making autonomy is placed as close to frontline staff and partners as possible.37 However, we find that given USAID’s hierarchical structure, the program’s decision making processes were highly bureaucratic and centralized. We find that oftentimes decisions had to be made in three different layers (USAID, JPV in Mexico City, and JPV regional staff and local actors), and that frequently, decisions were made in Mexico City. Finally, we find that weak or nonexistent accountability loops from JPV and USAID to the actors within the system impact collaboration and sense of ownership. During the interviews, a 35 See “Grant and Contract Process”, USAID, 2020. 36 See “Introduction to Collaborating, Learning and Adapting (CLA) in the Program Cycle”, USAID Learning Lab, 2019. 37 See “Evidence Base for Collaborating, Learning and Adapting. A summary of the literature review”, USAID, March 2020. 42 significant number of actors asked the evaluation team about the possibility to see the results from this evaluation and expressed general discomfort with being continuously asked to share their perspectives and experiences, but barely hearing back about the results from this type of exercise. Creating mechanisms for the international cooperation to be accountable to the local actors and share feedback is crucial when thinking about iterative collaborations and sustainability. When thinking about systems practice,38 it is imperative to recognize that USAID and its contractors become an actor within the system. By providing resources and defining the rules that accompany them, USAID creates a series of modifications in the relationships, interactions, and incentives that happen at the local level. USAID is not an external actor, but rather an active player in the system. Attributes of local systems are at odds with USAID’s hierarchical culture, and this culture reflects and has consequences outside of USAID. As such, USAID must address its own institutional barriers that diminish local ownership. While this is not particular to the Mexico mission, 39 we believe there to be important areas of opportunity to revise and reflect on these aspects to define a more inclusive and horizontal culture that better enables the development of local systems. IV.4 JPV CONTRIBUTION TO THE IMPLEMENTATION OF EVIDENCE-BASED PREVENTION INTERVENTIONS In this section, we zoom into the specific interventions implemented by local actors to assess whether the program was able to influence key avenues for reform. After a short overview of JPV’s donations program, through case study analysis, this section examines the quality of JPV efforts to enhance CSO program effectiveness based on the three principles of effective crime￾and-violence-prevention strategies as found in the literature: concentration, coordination, and proactivity. The objective is to analyze the extent to which causal mechanisms could be theorized between the prevention interventions and the expected intermediate outcome of reducing the likelihood of youth to engage in crime and violence activities. That is, we analyze whether implemented interventions exhibit designs and an implementation quality that respond to what the evidence identifies as effective. We find most of JPV’s technical assistance activities to have added to the concentration principle by supporting CSOs program development with enhanced design, delivery, and coverage. To accomplish this, JPV improved the capacity of CSOs to use evidence, assess youth risks, specialize services through CBT protocols, and increase program coverage. However, pressing challenges remain in terms of addressing youth risks and needs to prevent recidivism. For this, more complex interventions with multidimensional strategies are necessary, and technical assistance to strengthen models with this approach is strongly recommended. In this view, work with CSOs with territorial interventions is a promising area. Additionally, an outstanding agenda is capacity￾building among government agencies to generate a specialized offer of youth services. 38 See “The 5R Framework in the Program Cycle”, USAID, 2016. 39 USAID’s evidence review of CLA (2020) highlights certain attributes of learning organizations, such as flexibility in resources, risk-taking time, and flat organizational structure, that may be at odds with USAID’s existing culture for various reasons (p. 11). We believe the organizational structure to be particularly relevant for local systems development. 43 To a lesser extent, JPV contributed to the coordination principle. Main achievements here relate to enhanced communication between CSOs and other relevant stakeholders. This happened indirectly through knowledge-generation activities, which strengthened CSOs networks and facilitated communication with high-level government actors that may not be aware of CSOs programs. JPV also helped CSOs navigate political changes and staff rotation in public agencies to ensure continuity in their collaborations with the government. Nonetheless, major challenges prevailed regarding CSOs coordination with the juvenile justice system and the civic courts, mostly related to youth referrals. Finally, the third principle remains as an ample area of opportunity for future USAID programs. While JPV did have lines of action to reduce recidivism, an important area of opportunity remains for programs that address underlying factors that cause violence. We discuss these findings in more detail below. IV.4.1 KEY AVENUES FOR REFORM: WHAT THE EVIDENCE SAYS When thinking about avenues for reform, the literature identifies overall key principles and elements of effectiveness that are crucial features of effective strategies and programs. Two of these principles are the concentration and coordination principles, which together establish that institutions should specialize in specific services targeted at the highest-risk places, individuals, and behaviors and should coordinate the provision of specific individual components with the understanding of how different interventions may interact and complement each other. The literature proposes for institutions to focus on a narrow set of risks and protective factors in just one domain, but to coordinate a multisector response to address this complex multidimensional phenomenon. 40 A third principle is that of proactivity, which suggests that interventions should seek to prevent crime and violence before they take place by understanding and addressing the underlying factors causing them. Finally, this principle also suggests that, if crime and violence cannot be prevented, the focus should be on reducing recidivism by focusing on rehabilitation programs. 41 But designing a prevention policy and combining interventions are not minor tasks. The literature shows that certain interventions can produce strong and positive effects, but unfortunately, most of the interventions have mixed or small effect sizes. 42 Given the inexistence of “silver bullets” for crime prevention, strategies must contemplate the appropriate combination of modestly effective programs that, when combined, can produce robust effects. That is, prevention strategies most incorporate elements of existing effective programs while also promoting innovation based on key elements that underlie the effectiveness of successful interventions, and these interventions should pursue the concentration, coordination, and proactivity principles. In terms of effective interventions targeted at at-risk youth, the literature shows that CBT￾informed interventions, multidimensional therapy, drug court, and drug addiction treatment, are effective interventions to deter at-risk youth from criminal and violent behaviors.43 These 40 See Abt, 2017 and “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 41 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 42 Ibid 43 The literature also finds that focused deterrence and hot-spot policing are effective interventions. Focused deterrence programs have been effective in taking a narrow focus on deterring high-risk and chronic offenders from committing specific crimes (mainly homicide) by engaging the police and other service providers in providing targeted sanctions and rewards (J-PAL, p.10). We do 44 interventions focus on understanding the drivers of individual behavior and shift related incentives or cognitive biases that may produce it, as well as support individuals with developing new prosocial skills. In particular, multidimensional interventions aim to prevent juvenile criminal activity by monitoring offenders inside and outside of school, furthering prosocial skill development, training caregivers and parents on behavior management techniques, and offering substance abuse treatment. The USAID J-PAL–commissioned report finds that investing in these types of therapy models that can overcome weak family structures is of the highest relevance in the Mexican context. There is another group of interventions for which there is promising, but still incomplete, evidence. These interventions include vocational and training employment, restorative justice, alternatives to incarceration, and conditional cash transfers. The main caveats with vocational and employment programs are related to the fact that, while they are able to provide alternatives to crime, studies find that increased income alone may not always reduce criminal behavior. We know that there is a need to develop “softer” skills such as self-control and decision-making, but as the USAID J-PAL–commissioned report mentions, interventions also need approaches targeted at the underlying risk factors that affect, for example, gang involvement. In this sense, programs that are able to further nongang or noncriminal social identities or aspirations are desirable. Finally, there are a number of interventions with inconclusive evidence. These programs include community-based programs and school-based programs, among others. We encourage the reader to read the J-PAL–developed report, What Works to Prevent Violence Among Youth?, to find more nuances on the evidence findings. IV.4.2 JPV DONATIONS PROGRAM To implement the program’s prioritized strategy of catalyzing solutions for at-risk youth, JPV launched a donations program to identify outstanding interventions and, together with technical assistance, support their implementation, systematization, and evaluation. Also, the program organized a set of events to enhance knowledge-sharing and replication of successful interventions. After analyzing the evolution of the donations program and organized events, we conclude that the program evolved from a broader strategy looking at identifying promising interventions to supporting reintegration strategies to, finally, focusing almost exclusively on strengthening and transferring the CBT and civic justice models. Annex V presents the evolution of the donations program, showing the technical agenda as announced in the requests for proposal (RFPs), as well as the awarded grants. The first call for proposals (2016–2017) was much broader and invited programs that had a variety of evidence￾based models.44 As this call was published before the change in strategy, it also financed programs outside of the six regions selected for the implementation of the local systems framework. As the program advanced during 2017 and 2018, the donations program took a much more focused strategy on the social reintegration of at-risk youth, while at the end (2019–2020), it pivoted toward financing projects to support at-risk youth as derived from the CJHM. Annex VI not include this intervention in this analysis because focused deterrence is beyond the scope of JPV, and USAID manages this intervention separately from this program. 44 Including CBT, parental skills development, alcohol and drug abuse, restorative justice, social cohesion, new masculinities, social reinsertion, focused deterrence, gender-based violence (GBV), and environmental prevention 45 presents the events organized by JPV. 45 Early events were focused more on disseminating evidence on what works to prevent crime and violence and identifying promising practices (such as CBT). Then, the strategy concentrated on two main objectives: 1) sharing best practices for replication through study tours, mainly CBT and civic justice, and 2) positioning the CJHM and related interventions, such as community policing, on the national discussion and creating buy-in for their implementation.46 In line with the argument made above about the gradual implementation of the local system strategy, we find that JPV made good progress in identifying promising practices, but less progress in strengthening models to the point of generating evidence of effectiveness for replication in most cases. The donations program resulted in most of the interventions having systematized models that could later be leveraged by other organizations, which is an important contribution of the program. In fact, most of these learnings, together with other relevant publications from the program such as diagnosis and evidence reviews, are published in the Learning Virtual Community developed in collaboration with Tec de Monterrey. Nevertheless, as can be observed in the last column of the table in Annex V, only six of the grants have some type of evaluation, and most of these are results evaluations. Only CEDAT, Supera, Prosalud, and Desafío have slightly more rigorous evaluations, although most of them are still pre-post evaluations. From this table, it is also noticeable that only a few organizations received grant renewals, limiting the capacity of those that did not to further evaluate and scale up their models. In sum, we find that the evolution of JPV’s financial and technical offer, while having a clear strategy to scale up certain interventions, stopped short of generating evidence about the effectiveness of individual interventions and in furthering more programmatic collaborations for the implementation of at￾risk youth strategies that would reflect the concentration and coordination principles. Below, we take a closer look at the programmatic quality of the interventions implemented by local actors as a result of JPV support. IV.4.3 PROGRAMMATIC QUALITY OF JPV-SUPPORTED INTERVENTIONS Through analysis of the case studies, this section examines the quality of JPV efforts to enhance CSOs program effectiveness based on the three principles of effective violence prevention strategies: concentration, coordination, and proactivity. KEY PRINCIPLE 1: CONCENTRATION From the three key principles of effective violence prevention programs, JPV’s technical assistance to CSOs mostly contributed to the concentration principle. This occurred through CSOs program development making use of four mechanisms: 1) strengthened use of evidence, 2) enhanced assessment of risk factors, 3) specialization, and 4) program continuity and expansion. As shown in Table 9, all CSOs considered in this analysis experienced improvements associated with these technical assistance activities. 45 In the text, we only discuss the events organized in the transversal strategies of solutions for at-risk youth and civic justice, but in the Annex, we include all of the events for the reader to see the evolution of the program. 46 JPV also invested in positioning the gender agenda in Chihuahua, but to a lesser extent. 46 Table 9. JPV Technical Assistance per CSO Activity/strategy Territorial Focalized JPV support/ CSO CASA La Tenda di Cristo Supera ProSociedad CEDAT Consejo Ciudadano Use of evidence None → basic47 None → basic48 Preexisting use of evidence49 + focalization tool None → moderate50 Preexisting use of evidence51 Preexisting use of evidence52 Methodological strengthening No support provided No support provided Gender-based violence prevention (GBV) component and Youth Transforming Youth model development ROLE model development Model transfer and scaling (incorporation of additional services) No support provided Internalization of CBT and mindfulness No transfer Delivered among other activities -Delivered among other activities -Adapted protocol to include GBV and cultural activities Focalization and fidelity vary among implementing partners Core intervention (preexisting) Delivered among other activities Use of YTT and monitoring tools -YTT is used for JPV reporting The intervention also uses additional tools to report to other funders -YTT is used for JPV reporting, youth profile generation, and socio￾demographic and legal interviews -YTT is used for JPV reporting The intervention also uses an additional self￾designed tool for monitoring -YTT is used for JPV reporting and youth profile generation and progress tracking The intervention also uses additional tools for monitoring -JYTT is used for JPV reporting and youth profile generation The intervention also uses additional tools for monitoring tools -JPV reporting -Youth profile generation The intervention also use additional tools for monitoring 47 The program solidified its theory of change, intervention systematization, and development of strategies to enable replication, but weak M&E systems persist, as well as the absence of rigorous piloting. 48 The program solidified its theory of change, intervention systematization, and development of strategies to enable replication, but weak M&E systems persist, as well as the absence of rigorous piloting. 49 The program restructured the existing theoretical framework and added additions to existing procedures and tools. 50 The program developed manuals and procedures, M&E system to track youth progress, and rigorously piloted interventions, but challenges with the program’s theory of change, data and evaluation quality remain. 51 The program restructured the existing theoretical framework and added additions to existing procedures and tools. 52 The program restructured the existing theoretical framework and added additions to existing procedures and tools. 47 Strengthened use of evidence This mechanism points to theory-of-change development, systematization of models and interventions, and strengthening of M&E systems. Based on CSO comments, these activities might have improved assistance and service delivery to youth. By developing the programs’ theory of change, CSOs enhanced their capacity to determine necessary interventions to address identified problems. As explained by two interviewees: “CSOs, sometimes, sadly, work based on hunches. The theory of change helps understanding the causal logic of things. That is a major takeaway.” “…if we hadn’t understood the value of the theory of change, we wouldn’t know the causal relationship between our activities and where they are leading us…for example, a higher artistic knowledge leads to a minor drug use.” Similarly, systematization of models and interventions strengthened the capacity of CSOs to learn from previous experiences and provided more resources to select strategies to work with new generations of youth. Even though it is not possible to develop manuals for every single intervention strategy and certain CSO models—particularly those following a territorial strategy—require some extent of flexibility to customize their work to specific identified needs, systematization did help them to have “more conceptual, methodological, and pedagogical clarity,” as pointed out by the director of another organization, and to replicate what has proven to be effective in the past. Although all CSOs did have M&E indicators before the JPV/USAID donation, only a few had actual M&E systems measured with reliable instruments that served to inform internal decision-making (see Table 9). M&E systems were enhanced or even entirely developed with the donations, which helped CSOs have a clearer sense of intervention effectiveness and make more informed decisions on how to intervene with individual cases. As described by an interviewee from a CSO with no preexisting use of evidence on how M&E tools were developed: “It was a trial and error. It is a deep exercise, a bit chaotic, not so organized…we built some [instruments] because USAID asked for a diagnostic…the other [tool] was also an exercise, because even if we have a vast experience, we had nothing organized nor systematized, we had to sit down and do the exercise to recover all that we knew to fill the registers, to do the diagnostics, the interpretative frameworks…now we go for larger groups, and that has pushed us to build the tools, profiles, models…before, it was improvised, now we have documents that support our work in the community…” Moreover, regular case monitoring with quantitative indicators, not only based on operator perceptions, serves to identify necessary changes to individualized intervention plans because it informs on additional interventions or services that each youth may need to advance in their social reintegration process. These indicators are used, as well as a parameter to determine when someone is ready to leave the program, as the same interviewee explained: 48 “These are long processes, the coldest part, the quantitative or measurable part, is when he has satisfied all considered variables in the program. Each program has a measurement tool. When he complies with all evaluation tools, [for example] when he has a practical or artistic skill, when he is emphatic with others’ problems…” Overall, strengthening the use of evidence among CSOs is a clear achievement of JPV’s technical assistance. However, the extent of the internationalization of some of these efforts and their sustainability beyond the donation is not clear, particularly with regard to M&E systems and the use of JPV’s assessment tools, as explained below. Enhanced assessment of risk factors A major part of JPV’s technical assistance to CSOs was directed to increase their capacity to assess youth risk factors, for which JPV developed the YTT (see Box 4) and encouraged its use and customization. The program required CSOs to assess youth risk factors making use of the YTT, which, beyond improving youth risk assessment and targeting, also supported CSOs reports to JPV on beneficiaries’ progress and results. Furthermore, it was customized to CSOs programs to increase its suitability. One interviewee explained the YTT customization process in his organization: “We advanced a lot with JPV and the YTT. I think we destroyed the YTT, we did a new tool based on it. We have the same measurement logic, the same measurement method, but we cut half of the variables and introduced others. Actually, before starting the project with JPV, we used to do the profile based on our intuition and on information from the neighborhood…there wasn’t a clear systematization process and I think the YTT helped us building it…” In the case of Supera, the YTT served as an input, along with preexisting tools, to develop an improved version of the organization’s own pre-post assessment tool with the support of JPV’s technical consultants: “The YTT determined the level of risk and the pretest determined there was an actual cognitive change in him. Because the YTT is not a pretest tool, we had to adapt it to the pre and post, and to intermediate evaluations to determine progress in them…The part of the posttest stems from the YTT, but we had already applied it in 2014 and 2015 with USAID’s support. What we did is using what we already had, along with JPV’s technical assistance, to build it — not just the YTT, which we have to apply because it’s the donation-- we were able to generate a more solid tool, that statistically can determine if there is a change and not only based on qualitative information.” In spite of its technical value, it is not clear if the YTT will be internalized beyond JPV. On one hand, CSOs with preexisting use of evidence (see Table 9) already had tools for youth assessment at the different stages of their programs (eligibility, monitoring, follow-up), which, in the cases of Consejo Ciudadano and CEDAT, are more robust and specialized. There, the YTT has a low added value. In the case of SUPERA, it is expected that the newly developed tool will continue to be applied. In contrast, CSOs that had limited or no M&E systems before the donation 49 do recognize its value, but perceive the YTT as another administrative requirement. In the specific case of CASA, the tool seems to be co-existing with other instruments utilized to report to other funders, and its main function seems to be reporting. On the other hand, all CSOs, with the exception of ProSociedad—which is not applying the YTT—identified opportunity areas in its design and pointed out difficulties in its application. In general terms, there are three main areas for improvement: 1. Length: the questionnaire is too long and is hardly applicable in one single session— according to one interviewee, a YTT session can take up to 3 hours. This is quite a relevant issue for organizations with a territorial intervention strategy that often identify their potential beneficiaries in high-risk areas (i.e., picaderos). Overall, the tool was designed to be administered in a private space, which does not correspond to the reality of CSOs with a territorial approach. 2. Bias and stigmatization: the tool is focused on the assessment of risk factors and exposure to violence. According to some interviewees, this generates a bias toward negative personal characteristics, which, unintendedly, stigmatizes youth during the interview. Furthermore, to have a complete assessment to develop an individualized intervention plan, it is necessary to address protective factors as well, which is what some CSOs did during the YTT customization process. 3. Sensitivity: the questionnaire addresses highly sensitive topics for a first-contact interview, when it is too soon to have the necessary rapport and trust between youth and the operators, so the youth tend to lie. For instance, it is hardly probable that in a first interview someone recognizes or will admit to having a drug abuse problem. This implies that the initial profiles of youth are often inaccurate and have to be adjusted once the operators know them better. Additionally, it is worth noting that the YTT’s highest added value occurred where there were no assessment instruments before JPV’s donation. A clear example of this is civic courts, which did not have any tool to determine youth profiles and make decisions on referrals before adopting the YTT. However, training on how to apply the tool has to be reinforced because public officials lack specialized training to conduct assessments, as we will review in more detail in Section IV.5. Specialization This third mechanism contributed to the concentration principle with efforts to enhance CSOs specialization to work with at-risk youth. However, as discussed in more detail in this section, there is significant room for improvement to develop specialized services among analyzed CSOs. To enhance CSOs specialization, JPV followed two different strategies at different stages in time, which are described in the above analysis of the donations program. In the first stage, the program conducted knowledge-generation activities: dissemination and study events on what works to prevent crime to identify best practices. In this process, CSOs learned from other CSOs, researchers, and relevant stakeholders by sharing their models, intervention strategies, and experiences. This might have increased CSOs awareness of extant tools and methodologies in 50 the field, but questions remain on how these experiences were internalized by participants to improve their own programs. In a second stage, JPV focused its efforts on facilitating CBT protocols to CSOs. First, JPV focused its efforts on developing protocols: CEDAT’s Mente Libre and ProSociedad’s ROLE. In the case of ProSociedad, JPV funded the development of the CBT protocol for the ROLE model. CEDAT’s Mente Libre protocol already existed before the JPV donation, but JPV supported its strengthening and scale-up, with CEDAT’s role being to train other CSOs. Then, the program focused its efforts on scaling up these protocols. As shown in Table 9, from our case studies, JPV was successful in transferring the CBT tool to Consejo Ciudadano, Supera, and La Tenda di Cristo, which reported using it as part of their interventions. These protocols were also transferred to be applied as part of the CJHM. All of these transfers are analyzed in more detail below. In addition to this, JPV supported professional development of CSOs staff and public officers through online training during the life of the program, resulting in certification in violence prevention and a course on violence prevention against adolescents. The certification, named “Violence Prevention and Strengthening of Citizen Security,” lasted 140 hours and was delivered in partnership with the Fundación Carlos Slim and the former Vice-Ministry of Prevention and Citizen Participation. The course, named “Prevention of Violence against Adolescents and their Participation in Building Social Fabric,” was developed in conjunction with the National Human Rights Commission and is now part of the commission’s permanent online-training offer. Some CSOs interviewees, mainly directors and coordinators, mentioned having participated in these initiatives, although it is hard to determine how these trainings improved program specialization and delivery. Overall, JPV’s efforts to build capacity among CSOs to address the highest-risk factors affecting most youth were limited. Specialized intervention in capacity-building was centered on facilitating the CBT intervention model. Although CBT has proven to be effective to produce temporary behavior change among young offenders, there are other community, family, and personal risk factors that need to be tackled to prevent recidivism. This is confirmed by the results of the YTT. As shown in Tables 12 and 13 in Section IV.6, where we analyze the YTT in more detail, the three highest-risk factors for men are community dynamics, drug abuse, and school context. For women, they are community dynamics, lack of a parental figure, and exposure to domestic violence. The urgency to address other risk factors is also confirmed by interviewee comments, who pointed out recurrent factors that interfere with youth adherence to programs and that hamper their sustained social reintegration. These factors relate to youth needs that are unmet by analyzed programs. The most salient ones are as follows. ● Drug abuse: Most CSOs do not have the technical capacity to work with youth who have drug abuse problems, in spite of the fact that this is one of the highest-risk factors identified by the YTT—particularly in the case of men. Furthermore, extant studies and evidence suggest that the benefits of interventions such as CBT might be null or limited for populations with substance abuse problems and that special protocols should be used 51 instead. 53 Based on CSOs comments, two principal approaches seem to be in place to deal with this issue. The first one is not admitting them, but instead referring them to institutions specialized in drug abuse treatment. This approach is mainly taken by CSOs with a focalized strategy. The other approach is harm reduction, which operates in CSOs with a territorial strategy. ● Family dynamics: Analyzed CSOs do not have major interventions with families beyond family therapy and orientation sessions in some cases. All CSOs operators highlighted the need to strengthen interventions in this area, as family is often the origin of youth antisocial behavior. However, JPV did not support work with families because these are indirect beneficiaries. This is an important gap in JPV and in CSOs programs because some of the most frequent risk factors relate to absence of a parental figure, domestic violence, and parents’ criminal activity, as shown by the YTT results. This was pointed out by several interviewees, for instance: “We do work with them in several emotional aspects, but when they get home, there are violence issues that can affect them. In the end, the work does not have that much impact…they’re only 2 or 3 hours per week with us, but most of the time, they’re in their family context, thus there can be setbacks related to violence.” ● Lack of employment opportunities: Unemployment is a major threat to the social reintegration of youth after program conclusion, as economic necessities might push them back to criminal activity. In addition to this, at-risk youth tend to live in depressed areas where employment opportunities are limited and provide poor pay. As explained by one interviewee: “…where youth live, it’s in the most distant areas from downtown, and job opportunities are 30 to 40 km from their communities. Thus, distance is even longer. We had already made some connections with youth, but they can’t go because with public transport, it would take them between 2 and 2.5 and a half hours to get there.” CSOs have weak links with labor market intermediation services and insufficient connections with the private sector to link youth with job opportunities. Even Consejo Ciudadano, which has a strong relationship with FICOSEC, and Supera, which works with members from the private sector, face challenges in getting youth employed. It seems that for most youth, self-employment might be a better alternative, given barriers mentioned above and other factors, such as age (many are still minors), emotional instability, and stigmatization. La Tenda di Cristo and Supera mentioned providing some support to youth entrepreneurial initiatives, although the use of specialized methodologies and expert advice in this field was not identified during the interviews. ● Psychiatric follow-up: The need for psychiatric services was highlighted by only a few interviewees, particularly with regard to dual-disorder diagnostics and treatment, in spite 53 See Vaugh et al., 2003; Easton et al., 2007; and Friedman et al., 2008. 52 of the high sensitivity of this intervention area. Although mental health issues are common among at-risk youth, none of the considered CSOs offers psychiatric services, and questions remain in terms of their capacity to diagnose mental disorders that would need more specialized treatment. Addressing these high-risk factors and youth needs requires multidimensional methodological approaches. In this view, a promising area of work is enhancing the specialization and intervention capacity of CSOs with territorial strategies, as their programs are more comprehensive and adaptive to the individual personal needs of youth, as illustrated in Table 10. Furthermore, these programs integrate strategies at the community level that can facilitate follow-up after program conclusion, which is a key aspect to prevent recidivism. Collected information from the interviews suggests that CSOs with this approach face more challenges and need more methodological strengthening to work in the different areas that they are addressing, along with enhanced process systematization and institutionalization. However, a major constraint possibly faced by JPV in bringing more support to programs with a territorial perspective is that intervention processes are longer than what was allowed by JPV’s regulations. Table 10. Comparison of Territorial and Focalized Model Strategies Territorial Focalized • Multidimensional approaches, but need methodological strengthening • Customized interventions to youth characteristics • Customized program duration • Follow-up strategies after program conclusion, but informal and weak • Lack of evaluation and systematization (CASA and La Tenda di Cristo) • Therapeutic protocols with supplementary activities • Preestablished interventions with menu of services to be offered • Preestablished program duration (judicial measures and protocols) • No follow-up after program conclusion (Citizen Council for Security and Justice [CCSJ being its acronym in Spanish]: three home visits) • Manuals, evaluation, and systematization are integrated to operation Lastly, an outstanding agenda related to the concentration principle—and one that was frequently pointed out during the interviews—is the lack of specialized government youth services. Youth are often sent to the standard public services offered, which often do not respond to their needs or realities. An example of this is public education programs, which do not consider constraints faced by youth who have been in a detention facility or who have been out of school for a significant amount of time. Organizations usually work with distance education systems, such as 53 those offered by the National Institute for Adult Education (INEA being its acronym in Spanish) or Prepa Abierta, but these do not ensure learning among these youth, and supplementary strategies have to be put in place with tutoring and remediation, as some CSOs have been doing. This lack of public service specialization is reflected as well in public officer approaches—and even reluctance—to work with these youth. This was clearly explained by one interviewee: “Institutions do not approach us to work…the kid is at the extreme of an institution that is not ready to work with youth…its rationale is not wanting to work with youth…there’s no institution that is willing to change and stop seeing youth as youth from the 80s or 70s, and to start seeing them instead as youth facing today’s problems.” Program continuity and expansion In general, donations helped sustain organizations’ work, as certain organizations did not have financial certainty to maintain their service provisions. Furthermore, donations also supported program expansion by increasing program services, which was possible thanks to JPV’s budgetary flexibility. This was clearly explained by one interviewee: “We finally have a large team to implement the program. Also, the flexibility to have new items in the budget. There are funders that don’t support youth therapy, but USAID gave us the opportunity to have that, the same with medical services to prevent withdrawal symptoms, scholarships for training…to provide them with lunch.” In most cases, USAID funding not only ensured the operational continuity of CSOs during the grant duration, but also enabled coverage expansion, either by being capable of reaching a higher number of their target population or by opening up to new populations. An example of this is organizations that were just working with youth in conflict with the law and that are now working with noncustodial sanctions and with civic courts. However, questions remain about the sustainability of CSOs’ work beyond JPV. This was pointed out by one director who highlighted the risks of only having one funder: “I can’t imagine us without [USAID’s] technical assistance. I’ve got used to having this great ally…Fortunately, FICOSEC just approved one of our projects.” CSOs also highlighted the value of USAID directing funds to work with these populations, which has had an effect in other funders that are now integrating these youth in their calls for proposals (i.e., Fundación del Empresariado Chihuahuense [FECHAC]). Furthermore, CSOs like La Tenda di Cristo and CASA mentioned the value of being capable of communicating and making use of evidence such as how they work and the effectiveness of their programs, which facilitate funding search processes. 54 KEY PRINCIPLE 2: COORDINATION As explained before, the coordination principle points to mobilizing multisector responses to address violence. In this view, two of JPV’s strategies had some contribution to this purpose. The first one was knowledge-generation activities, which created connections between CSOs and government actors. They managed to enhance awareness and inform stakeholders on relevant topics on youth and best practices for intervention, but, more importantly, they helped to sensitize relevant local systems stakeholders on CSOs work and to improve CSOs coordination with public agencies, as explained by one judge who participated in these workshops: “Two months ago, in Mexico City…the topic was boys, girls, and adolescents, I was the only one from the judicial branch there, the rest were CSOs participating in JPV…The training was on how to work with the judicial power, regarding CSOs and adolescents…these trainings are a substantial exchange of experiences on what civil society does in collaboration with authorities.” JPV also supported CSOs in navigating political changes that accompany public officer rotation and often interrupt ongoing programs with government agencies. This was explained by the director of one CSO: “JPV has helped us with the dialogue on some occasions, and in others, FICOSEC. That is one of the most important things that USAID should do: sit the stakeholders down and help grease the collaboration process…JPV’s way is politically correct because they have long-term relationships, thus we can have more private meetings [with the stakeholders] where they can better understand the context or the programs’ guidelines, that the civil society can do it better because it has nothing to lose.” Nonetheless, we identified major coordination challenges between CSOs and the justice system. These coordination problems relate to justice officers’ limited knowledge of services offered by CSOs. This often results in young people being sent to CSOs that are incapable of assisting them and that have to redirect them, which interferes with their adherence to treatment programs and the social reintegration process. This issue appears to be particularly relevant for the CJHM, where youth are often sent to organizations that cannot even admit them based on their inclusion criteria, as we will review in more detail below. Other difficulties in the relationship with CSOs have to do with poor coordination among justice system officers, which can put youth at risk. Protocols need to ensure that public officers share information on the background of youth who are directed to the same organization and belong to confronted groups or were involved in the same crime. KEY PRINCIPLE 3: PROACTIVITY Proactivity refers to interventions directed to prevent violence before it takes place, which entails addressing underlying factors causing violence. Although JPV supported a few programs with secondary prevention strategies, its focus was to work with youth in conflict with the law, so most of its contribution to the proactivity principle was strengthening CSOs programs to prevent youth recidivism. As explained above, JPV’s technical assistance to CSOs did enhance their 55 capacity to intervene with youth and to better address some of the risk factors affecting them. It is worth noting that, in the specific case of CSOs analyzed in this section, JPV did not support any community-based intervention or strategy intended to mitigate factors that can lead youth to get involved with violence. The extent to which CSOs programs are capable of preventing recidivism is hardly measurable and comparable. On one hand, there are limitations in terms of data collection and information systems to track youth trajectories after program conclusion. Information systems of different states’ public agencies, such as public ministries, juvenile justice systems, criminal justice systems, and civic courts, are not sufficiently integrated. On the other hand, CSOs intervention models are different and target youth with varied characteristics. For instance, while certain CSOs do not admit youth with drug abuse problems, others do. Another example is the context of youth. While CSOs with a territorial approach are based in the highest-risk communities, CSOs with focalized models work with youth from different communities, which are not always the highest￾risk. This is important because community-level risk factors can push youth to recidivism. We present a targeted analysis in more detail in Section IV.6. CSOs mentioned conducting follow-up visits or calls up to 6 months after program conclusion— mostly as part of JPV’s requirements, although some had preexisting follow-up mechanisms. After that window, only CSOs with territorial strategies (see Table 10) seem to have informal contact and information about youth situations, as they perform regular activities in the communities where youth live. It is worth noting that interviewee perceptions on what constitutes effective social reintegration is varied. To some, their programs are effective in achieving social reintegration because youth do not come back. This was a more common view among interviewees from CSOs with focalized approaches. To others, social reintegration is a complex process with ups and downs that does not entirely depend on youth. This was explained by two interviewees from different CSOs: “Recidivism in criminal groups often relates to their economic situation. We have many kids that didn’t manage to transition to the labor market and ended up returning to criminal activity. What is more common is that someone in the family gets sick [and so they start selling drugs again]. The families themselves push them to return to criminal activity [with economic pressure]. There are cases of mothers who poison their sons to avenge their brothers [when they are murdered].” “The context, connections with organized crime, community, social, and family contexts, gangs’ presence. One in every three can relate to a psychological cause that was not addressed during childhood and to drug use. Working with friends is also necessary to enhance reintegration.” Considering this and as pointed out above, strengthening the programs of CSOs with territorial approaches is a promising area of work—not just because these are more comprehensive and adaptive programs addressing more risk factors and youth needs, but because they have sustained interventions in communities to enhance protective factors at that level. In the words of one interviewee: 56 “There are other factors, such as the community. We identify potential allies, support networks that can be mothers, certain police officers, school teachers, and even their own crew can constitute a protective factor.” This is relevant from the perspective of the proactivity principle. Community-level protective factors can benefit all youth living there—not only those who have been through the juvenile justice system or the civic courts—and can help prevent others’ future involvement with violence. IV.5 JPV’S CONTRIBUTION TO THE SCALE-UP OF EFFECTIVE MODELS This section analyzes JPV’s efforts to encourage the piloting, systematization, and scale-up of promising models to prevent crime and violence. We find that JPV was successful at leveraging different pathways to scale and that the focus from the beginning on scale, the provision of a meticulous strengthening process through access to overarching resources, and the generation of relationships of trust and co-investments with partners are key to achieving scale. Although JPV was remarkably successful in supporting the adoption and scale-up of the ROLE, Mente Libre, and civic justice models, rushed expansion, absence of rigorous evaluations, weak technical capabilities in implementing organizations, and lack of consideration for differences in local circumstances posed challenges for the fidelity and effectiveness of models. JPV’s objective was to identify and enable the conditions for local systems to scale up effective policies. To this end, JPV allocated resources and provided technical assistance for the implementation, adaptation, and evaluation of interventions in the four substantive transversal strategies of the program and facilitated interventions’ systematization and knowledge transfer within and between regions to achieve scale-ups. JPV’s assistance consisted mainly of recruitment of specialized consultants, organization of trainings and events with key international and national actors, facilitation of knowledge transfer through study tours, publication of manuals and implementing guides, funding of process and impact evaluations, and development of targeting tools. By providing access to overarching resources and focusing its technical assistance on policy influence and transfer, JPV was able to leverage its resources and interactions with the system to “close the cycle” and achieve the scale-up and transfer of promising models. The scale-up results achieved by each strategy are as follows. • CPTED: Chihuahua’s municipal government implemented and evaluated a CPTED intervention, “My Neighborhood is My House,” which then was expanded to four additional municipalities. The CPTED methodology was also adopted in Guadalupe, Mexicali, Tijuana, and Zapopan. • Gender strategy: The care model was transferred to institutions in Ciudad Juárez, Monterrey, and Guadalajara. • Civic justice: JPV was able to influence the Mexican government to launch a national police and civic justice model in 2019, which incorporates in its entirety the Civic Justice Homologated Model and a PoP model. This model is being scaled up to more than 30 municipalities in the country, with additional demand for replication in others. • At-risk youth: Two CBT-based interventions, ProSociedad’s ROLE and CEDAT’s Mente Libre, are being implemented by several organizations in Monterrey, Ciudad Juárez, Chihuahua, Morelia, and Guadalajara and as part of the treatment services of the CJHM. 57 Annex VII shows the resources and activities dedicated by JPV for each of these strategies and presents the results achieved in each function of the local system that led to scale-ups. In this section, we study the cases of CBT and civic justice because they achieved a higher degree of expansion, and we deem them to be the most successful. As reviewed before, these two strategies were a priority for the program and received significant investment. By studying these cases, we aim to shed light on the key resources and avenues that can contribute to successful expansion of the interventions, as well as to highlight remaining challenges. IV.5.1 PATHWAYS TO SCALE AND ELEMENTS OF EFFECTIVENESS In general, evidence-based policy design consists of a constant process of adaptation and innovation, testing, and scaling-up. There are different ways in which implementing organizations, governments, and international development agencies can leverage evidence to inform and shape policy at scale. The key differences in the pathways to foster the transfer and expansion of promising interventions are the existing evidence bases from which these processes depart. JPV followed different pathways to catalyze, systematize, and scale the selected interventions. • Adapting, testing, and scaling up an intervention model informed by intervention models of renowned evidence-informed programs. The ROLE model was developed by ProSociedad per JPV’s request. Given that there is a strong base of rigorous evidence supporting the effectiveness of specific CBT programs, such as Becoming a Man (BAM) 54 and Sustainable Transformation of Youth in Liberia (STYL), 55 JPV awarded a subcontract to ProSociedad with the objective of tailoring a CBT program for the Mexican context with components that could be transferred to other institutions and that could also be adopted in a school context. • Strengthening, testing, and replicating an innovative model informed by research insights of rigorous evaluations. The Mente Libre model was developed by CEDAT, and JPV facilitated its strengthening and expansion. Through a partnership with local academics, CEDAT had developed a model that accounted for insights from different disciplines such as psychology and neuroscience. JPV built upon these efforts and awarded two donations and a follow-up grant to CEDAT. The objective was to evaluate and systematize the model with a public policy approach in order to transfer it to the local government and other organizations. • Innovating, systematizing, and replicating a promising institutional reform informed by elements of effectiveness. The Civic Justice Homologated Model is an institutional innovation based on community courts, restorative justice, and PoP components that JPV developed in collaboration with the governments of Morelia and Escobedo and the federal government. Based on an idea that emerged at the local level and for which there was no evidence base or program to adapt from, JPV provided technical assistance to design a model, connected implementers with specialists in court innovation implementing community court models, generated administrative capacities for 54 See Heller et al., 2017. 55 See Blattman et al., 2017. 58 its implementation, and systematized the model so that it could be transferred to more municipalities and influence policy at the national level. We identified the following three common characteristics that made these scale-up models succeed. 1. JPV’s technical assistance focused from the beginning on developing interventions with potential for scale. Defining this clear objective allowed JPV to focus its efforts and resources on facilitating organizations’ access to platforms to showcase their work and on developing products that would give further exposure to the models. Additionally, because these products systematized the use and generation of evidence and provided tools for their implementation in other contexts, they were instrumental in gaining traction to drive their adoption and replication. 2. JPV implemented meticulous strengthening processes providing access to overarching human capital and financial resources. Access to overarching resources that contemplated the implementation, evaluation, and transfer of the programs and specialized technical assistance from consultants strengthened the models and generated buy-in from relevant actors in the different local systems. The access to overarching resources is reflected in the total amount of investment that JPV allocated to the strengthening, implementation, evaluation, and systematization of these models (approximately 20% of the total amount of the program’s expenses) and in the diversity of the services provided. Table 11 shows the contracts awarded to different organizations to achieve the scale-up of the selected intervention models in detail. Additionally, organizations such as CEDAT were among the few organizations that received multiple donations and even a follow-up grant to give continuity to the results obtained through the previous award. 3. JPV was effective at building credibility and generating trust in the potential for expansion of the models, securing co-investments from implementing organizations and interest from relevant players. The credibility built by JPV and its responsiveness to local actors’ initiatives led CEDAT and the local governments of Morelia and Escobedo to designate sizable resources to strengthening and systematization and to invest significant efforts in facilitating model transfer (see Table 11). JPV was able to create trust relationships with partners. As one interviewee put it: “The comparative advantage of USAID is that they say they will support evidence￾based programs to [scale up models], and to support evidence generation and provide funding for evaluations. This is something innovative that we had not seen in Mexico for 10 years. We will sit with you so you can evaluate your project, and if you do that right, we will give you the platform and we can refinance you…These secretly conditional grants are very good. I condition you to grow. Grow and I will give you more money.” 59 Table 11. JPV Expenditures in the Scale-Up of Selected Intervention Models Model Organizations involved in the scale-up of the model Total expenditure by type of contract and activity ROLE ProSociedad (consulting organization and developer of the model and transfer methodology) Laboratorio de Estudios Económicos y Sociales (consulting organization and evaluator of the model) Total invested amount: 2,625,388.40 MXN • Subcontract to ProSociedad to design and pilot the intervention model and develop a transference methodology to the school context: $1,729,176.80 • Purchase order to Laboratorio de Estudios Económicos y Sociales for the external evaluation of the program in public schools: $295,800 • Purchase order to ProSociedad to support the transfer of the model to organizations providing services to infractors in the civic justice process in Morelia: $600,411.60 Mente Libre CEDAT (local nongovernmental organization [NGO] and developer and implementer of the model) Total invested amount: 3,125,048 MXN • Grant to strengthen the model and assess the possibility of transferring the model to Jalisco’s detention centers: $999,598 • Follow-up grant to generate evidence of the impact of the model and transfer the model to Jalisco’s detention centers: $524,505 • Grant to design an integral program including Mente Libre as a component to serve youth in conflict with the law on noncustodial sanctions and/or civic justice processes in the metropolitan area of Guadalajara: $1,690,945 * Additionally, CEDAT co-invested 1,208,971.40 MXN Civic Justice Homologa￾ted Model C-230 Consultores (consulting organization, developer of the homologated version of the model, and advisor in the design of civic justice law) Observatorio Ciudadano de Seguridad, Justicia y Legalidad (consulting organization and developer of training guides and sessions guided to public officials) Inteligencia Pública en el Análisis Político (consulting organization and advisors and assistants to local governments in the implementation of the model) ProSociedad (consulting organization and assistants in the adoption and transference of the intervention model) Universidad Vasco de Quiroga (UVAQ) (local university and implementer of treatment model) Center for Youth Integration (CIJ being its acronym in Spanish) (local NGO and implementer of treatment model) Supera (local NGO and implementers of treatment model) Total invested amount: 12,827,153.65 MXN • Subcontract to C-230 Consultores to develop the Civic Justice Homologated Model together with the Mexican government and the municipal governments of Escobedo and Monterrey: $2,790,531 • Subcontract to Observatorio Ciudadano de Seguridad, Justicia y Legalidad to develop a training of public servants in PoP and mediation in Morelia, Ciudad Juárez, Chihuahua, and Guadalajara: $3,826,969.90 • Purchase order to Inteligencia Pública en el Análisis Político to support the implementation of the model in Chihuahua and consolidate the model in Morelia and Escobedo: $2,017,257 • Purchase order to ProSociedad to support the transfer of the model to organizations providing services to infractors in the civic justice process in Morelia: $600,411.60 • Grant to UVAQ in Morelia to adapt ProSociedad’s intervention model to assist infractors derived from the civic courts: $1,116,877.08 • Grant to CIJ in Morelia to implement an addiction treatment model to assist infractors derived from the civic courts: $701,436.45 • Grant to Supera in Monterrey to adapt ProSociedad’s intervention model to assist infractors derived from the civic court: $1,773,670.62 * Additionally, the local governments of Morelia and Escobedo co￾invested 14,859,826 MXN and 25,007,475 MXN, respectively, for the implementation of the CJHM. 60 Furthermore, the credibility also brought attention and interest from relevant partners, such as the private sector in Morelia. Securing co-investment and interest from key players favors intervention sustainability, and it also shields intervention continuity from government changes, as was also the case in Morelia. IV.5.2 SCALING-UP CHALLENGES Although JPV was remarkably successful in supporting the adoption and scale-up of the ROLE, Mente Libre, and civic justice models, rushed expansion, absence of rigorous evaluations, weak technical capabilities in implementing organizations, and lack of consideration for differences in local circumstances posed challenges for the fidelity and effectiveness of interventions. Rushed expansion JPV’s efforts succeeded in building traction and local demand for the expansion of civic justice and CBT models. However, the transfer and expansion processes intensified during the last 2 years of the JPV activity. Because this left a relatively short time to achieve results, we find that the expansion processes were rushed and did not allow sufficient time for polishing implementation processes or enabling the consolidation of key intervention components. As a result, some models are being consolidated as expansion grows. As two interviewees expressed: “There is a pressure to scale really quickly. If the first years the [program’s] focus would have been on polishing the good things that happened in each municipality before scaling, I think that the impact would have been higher. Many of these initiatives require maturation time, there are many mistakes that are avoidable if you do not scale up so quickly.” “Suddenly there is a pressure [to scale] from the international agenda. They want many numbers and many impacts. If they designed multiannual plans where they had the possibility to work very well on one methodology in a few places, and then scale it up, I think they would have a bigger impact.” In the case of ProSociedad, we find that the school context implementation happened with many unanswered questions about the theory of change. As one interviewee put it: “The main design problems were in the causality relationships, there were many causal relationships and there were errors in how the lines of causality were drawn…There was no clear logic to decide which dynamics to work on during the sessions…Theory was used to justify the program, but the program did not have a strong theoretical framework and there was confusion between the different program components…It is a very sellable program…but a lot of work is needed on the model’s specifications.” In the case of civic justice, we find that key model components, such as the screening instrument and youth referrals to treatment alternatives, were not sufficiently piloted in Morelia and 61 Escobedo before expansion. Box 6 presents a detailed analysis of the challenges involved in scaling up the CJHM. As mentioned by one interviewee: “The civic justice system is under construction and many procedures are needed. When there are no procedures to respond to specific circumstances, we end up responding by occurrence…There are protocols, guides, manuals and procedures that still need to be developed…There is a lack of coordination between the institutions to serve young people, many times youth do not comply with their sanctions.” Scale-up strategies should prioritize giving organizations sufficient time to refine and consolidate all the components of model implementation before scaling them up. This step is essential because transferring suboptimal models constitutes a risk to the credibility in the medium term and to intervention effectiveness. Box 6. Key Components of the Civic Justice Homologated Model and Implementation Challenges As the CJHM is rapidly expanding nationally, the challenges of implementing it at scale need to be addressed in more detail. One of the key premises of this model is that it helps to deescalate future conflicts and to reduce recidivism by offering a means to identify and treat higher-risk offenders. As depicted in Figure 3, the CJHM’s preventive approach depends on the effectiveness of the judge’s risk assessment, the referral process to adequate treatment services, and the adherence of the offender to the treatment services. The case of Morelia provides important lessons about the implementation of this model and the challenges involved in securing the correct functioning of its processes. First, the civic justice system processes tens of thousands of offenders, with the majority being administrative faults such as parking in a restricted area and public noise disturbance, among others. Therefore, the capacity of public officials in the civic courts to identify high-risk youth is crucial and requires awareness about youth risk factors. Furthermore, to determine the required treatments, finer assessments tools are needed. The referral process is complex and requires not only strong diagnosis skills, but also knowledge of the potential treatment options. As one interviewee put it: “The probability that an at-risk youth will arrive through the civic justice system is high. Since treatment alternatives are expensive, we should focus on those at higher risk. In this sense, risk assessment is crucial to prioritize the provision of treatment.” The case of Morelia shows that these processes are highly complex. Lack of protocols, lack of staff training, and limited local capacity to identify the risk factors of offenders passing through the system have directly affected the treatment process. For example, the Center of Social Services (CAS being its Spanish acronym) encountered the challenges of receiving ineligible youth and not having enough group participants to initiate therapy groups given the long time taken by the referral process. Municipalities where the CJHM has been transferred are facing these same challenges. For example, an interviewed public official mentioned that because of the lack of capacity to assess all offenders (more than 30,000 offenders for 1 social worker), judges learn informally along the way how to prioritize treatment services. As he put it: “The main program is the lack of personnel. We just have one social worker. In 2019, over 30,000 people were presented to the civic court. Judges have learned to prioritize the 62 56 In particular by giving grants to UVAQ in partnership with ProSociedad to transfer the ROLE model to CAS and CIJ. treatment for those who have the smaller probability of incidence. The experience gives you light to distinguish which cases are savable.” Second, the CJHM faces important challenges in ensuring program adherence and in having adequate processes to monitor compliance with alternative solutions. Following the example of CAS, this experience showed irregular turnout. The lack of social workers’ specialized training results in referrals that do not meet offenders’ needs and in the need to rechannel offenders, which significantly hinders offenders’ willingness to participate. To address these challenges and through a very intensive technical assistance process, JPV supported the creation of the Directorate for the Execution of Administrative Sanctions, a specialized unit in charge of referral and monitoring processes, as well as worked to adapt the YTT for its application within the civic justice system. Third, the possibility to refer offenders to treatment services assumes that there is a supply of those programs. The case of Morelia showed that this is not always the case. JPV recognized this challenge and during 2019 and 2020 launched a strategy to create capacity in local CSOs to offer therapeutic￾component interventions to youth derived from the civic justice system.56 Furthermore, the program recognized the need to expand the responsibilities of the Directorate for the Execution of Administrative Sanctions to congregate other actors of the local system. While these efforts show preliminary success, the offer of therapeutic components for youth derived from the CJHM is still highly reliable on JPV support, casting doubt on the model’s sustainability. This issue is also found in other regions, where personnel from the civic justice system expressed a lack of resources and options to send offenders to treatment programs. Moreover, questions remain about whether the available treatments respond to youth needs. Finally, weak information systems can hinder proper monitoring and tracking and, thus, could affect the reduction of recidivism. While Morelia has achieved great progress in developing a registry of offenders, it is unclear if the data quality allows for keeping track effectively and to what extent service provision organizations report relevant information promptly. These challenges are further exacerbated in metropolitan areas such as Escobedo, where there is a need for coordination between different municipalities and state-level information systems. 63 Figure 3. Civic Justice Homologated Model Source: Based on JPV progress presentation, February 2020. Absence of rigorous evaluation Accelerated model expansion and project timeline hindered proper evaluation of the models, either by not allowing for process and impact evaluations to be carried out or by failing to create spaces to address issues raised by evaluation processes before continuing expansions. In the case of CEDAT’s Mente Libre, it is remarkable that, through a grant, JPV facilitated an impact evaluation of the model. Although the sample size is small, this significant investment allowed the organization to get more information on how the program was functioning and served to increase the evidence base for its expansion. As one interviewee put it: “The simple fact of having had the opportunity to do an evaluation with electroencephalograms, which is very expensive, and to generate very straightforward indicators of the impact of the program, such as that the youth who participated reduced their impulsive behaviors by 30% compared to a control group, gave us solid evidence to continue working the project in other contexts and to present it in forums.” Police ( Problem oriented) Civic Judge Risk assessment Monitoring of high-risk offenders Addiction treatment CBT Employment opportunities Canalization of high-risk offenders Administrative fault Community Community conflict Multisystemic therapy Thousands of offenders 64 In the case of ProSociedad’s ROLE, a process evaluation was performed for its implementation in schools. This evaluation was disrupted by a delay in the government’s release of the funding for the project’s implementation, which affected the quality of the data collected and reduced the evaluation timeline. Nevertheless, the evaluation delivered important findings, such as the ones highlighted above about the missing pieces in the program’s theory of change. Also, this evaluation finds several fidelity issues in which the model was not delivered as needed by the theory to achieve changes, mainly because of lack of implementer capacity, lack of supervision protocol, and lack of buy-in from facilitators (teachers in this case). We did not find clear instances in which the model addressed these issues. Furthermore, the model is being transferred to different contexts (not school settings), which most likely present additional fidelity concerns. No other evaluation is being run. In the case of the CJHM, evaluations are nonexistent. Rigorous evaluation is a key element for the transfer, adaptation, and replication processes, and it is imperative that the following USAID activities in Mexico prioritize evaluating the interventions before continuing their expansion. Process evaluations are needed to ensure program fidelity, and impact evaluations are needed to confirm whether the program’s intended mechanisms for change actually work. • For the CBT-based models, ROLE and Mente Libre, although there is a strong evidence base of effectiveness, further rigorous evaluation is needed to assess which approaches are more appropriate and effective in the local contexts. As noted in the USAID J-PAL– commissioned report, there are important pending research questions about whether these models with complementary interventions, such as cash support or more integral treatment approaches, could be more effective. Also, there is a significant opportunity to determine the most cost-effective length of treatment and how to adapt the programs for a higher-risk population. • For the CJHM, the lack of evidence is worrisome. The evidence that has supported the model’s alleged impact is purely anecdotal or based on perception. Yet, it has expanded quickly across the country. The future expansion of the model to different municipalities offers a timely opportunity to rigorously evaluate the CJHM. First, process evaluations are imperative in all municipalities to make sure that the key model components are properly implemented. Second, impact evaluations of the different components are desirable to understand whether and how the mechanisms are working. To this end, the following are needed: a. Define what we want to learn, what are the relevant questions, what outcomes are expected, and how can they be measured. For example, the outcomes of interest could range from changes in citizen perception of safety by reducing impunity for minor offenses to changes in recidivism of offenders who go through the system and receive treatment alternatives to changes in citizen trust in the police due to community outreach efforts. 65 b. Then, an impact evaluation design with a control group should be determined considering all the political, administrative, and ethical constraints. For example, an experimental phase-in design would guarantee that all units of analysis would eventually receive the “treatment.”57 Although rigorous evaluations are not easy to perform because they are time-consuming processes and require specialized technical knowledge, it is imperative that models are rigorously evaluated at a small scale before expanding them. Otherwise, there is a risk of transferring models that have not been tested and that may not have the desired impact. Weak technical capacities in organizations that adopt the models When replicating and expanding promising interventions, developing implementation capacity in organizations that adopt the models is crucial to ensure fidelity. Transfer and implementation of any policy need to account for the specialized training of operators. In the case of CBT, CEDAT provides intensive training and certification to program operators. For CEDAT, a priority in the transfer process is the specialized training of facilitators. The transfer model includes 100 hours face to face and practical and theoretical training, and approximately 30% of the trainees earn certification. Unlike CEDAT, the training for program operators who are adopting the ROLE model is unclear. While the manuals consider training, the parameters are quite broad, casting doubt on the extent to which facilitators are sufficiently trained. We consider that the key difference is because of CEDAT’s extensive experience working directly with youth and with training operators, whereas ProSociedad’s ROLE is a second-floor organization. Therefore, particularly in cases where there is less experience in implementing, process evaluations are of the highest priority. CEDAT’s good training practice is consistent with the academic CBT literature, which indicates that training of treatment providers and their ability to connect with youth are key to guarantee effectiveness. 58 For interventions such as the CJHM, the lack of adequate training of the staff in charge of channeling offenders can affect the model's functioning and undermine its effectiveness. We find that the model is being transferred without ensuring proper staff training in the key intervention components, as highlighted in Box 6. A factor that could have hindered capacity-building at the operator level is that JPV favored a top-down approach during the transfer processes. As mentioned by one interviewee: “JPV worked very top-down. First, we had to talk to the governor, the prosecutor, the commissioner, and the big players. That is the most traditional way of doing public policy, but we also find a lot of value in going from the bottom up…The governor can say yes but the social worker can say no, sometimes the line actors are left aside but that is not always the most effective when it comes to practice.” 57 For more information on how to design impact evaluations, see https://www.povertyactionlab.org/sites/default/files/research￾resources/Introduction-to-Evaluations_0.pdf 58 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 66 Weak adaptation to local contexts A crucial element in the adaptation and replication of models to other contexts is to account for how the new local circumstances affect the underlying model theories. Because no two contexts are identical, scaling programs in different places is not as simple as transferring an "out-of-the￾box" solution. In this sense, a program should only be transferred when the new context allows for adoption of the essential causal mechanisms of the intervention’s theory of change. For the selected interventions, we find that oftentimes the scale-ups were carried out in widely different contexts without careful adaptation. As a result, the selected interventions may not be ideal to address local risk factors and dynamics of crime and violence. While JPV succeeded in facilitating knowledge transfer and leveraging the effort invested in systematizing the ROLE and Mente Libre models by transferring them to treat offenders referred by the civic courts, questions remain about whether these models were properly adapted. Even though CBT interventions are relatively easier to systematize (see Table 10) given that they are focalized and have a determined curriculum, careful consideration must be given to underlying risk factors of different target populations. For instance, youth at school might have different risk factors and needs than the target population of the civic justice system, ranging from an out-of￾school, employed 25-year-old sent to a CBT treatment to a 40-year-old given the same treatment option. Particularly for the case of ProSociedad, it is important to note that the pilot evaluation was carried out for the model’s implementation in schools, but it is unclear how this model transfers to implementation with high-risk youth identified by civic courts. Moreover, failures in the referral process may affect adherence. As mentioned by one interviewee: “There is no sensitization process by the courts. Offenders arrive to the programs with reservations regarding their participation and with the idea that the measure is a punishment. This hinders the intervention process, at least during the first sessions.” For the CJHM, in the absence of a rigorous evaluation, many doubts remain about in which contexts it is optimal to adopt the model. The piloting of the model in Morelia and Escobedo was carried out in very particular contexts, with specific political dynamics and accompanied by substantive reforms in police operations. In this sense, when thinking about expansion, it is key to take into account how the underlying theory of the model is affected by differences in crime dynamics and institutional capabilities. For example, police officers working in high-violence cities may present more hesitation to adopt mediation strategies. Implementing a PoP approach might be challenging in places where commanders do not favor cultural change or in places where the local police are unable to perform investigative work. Furthermore, the lack of operating rules and procedures in some localities generates a disconnect between the objectives pursued by the police and the judges. As explained by different interviewees: "In Morelia we found that the police stopped detaining people because they considered the civic justice to be a revolving door model and they did not want to arrest anyone if the judges were going to let offenders go." "In Juárez it has been very difficult to train police officers because they know that mediation does not help them when they have to focus on avoiding being killed." 67 "In general, daily police work does not allow the civic justice and PoP model to operate because the work of the police continues to depend on public prosecutors and indicators such as the number of arrests." “We have many problems with the Secretary of Public Security because the judges already operate the model, but the police have not been willing to implement it. The approval of a regulation would allow us to have teeth for the police to cooperate with the model. Sometimes we have to talk to superiors so that they order middle-ranking officials to follow the implementation of the Civic Justice model.” While the CJHM could be a relevant and effective alternative in different contexts across Mexico, the model should not be seen as a silver bullet for regions with great differences in crime dynamics and institutional capabilities. 68 STAGE 2—DID THE PROGRAM CONTRIBUTE TO A REDUCTION OF AT-RISK YOUTH LIKELIHOOD PARTICIPATING IN CRIME AND VIOLENCE? 3. What evidence is there that JPV has reduced the likelihood of at-risk youth in targeted communities participating in crime and violence? a. What evidence is there that at-risk youth reached with JPV programming will engage in alternative, positive pathways and steer away from involvement in crime and violence, including with TCOs? b. Is JPV’s coverage or reach of at-risk youth in targeted communities (i.e., number of at-risk youth reached) sufficient to initiate a detectable reduction in youth involvement in crime and violence? Why or why not? c. To what extent have gender-based roles, including but not exclusive to the mothers of at-risk youth, affected the ability of the interventions to achieve their stated results (effectiveness)? Box 7: Question 3 Summary To answer this question, we analyze whether JPV’s coverage or reach of at-risk youth in targeted communities was sufficient to initiate a detectable reduction in youth involvement in crime and violence. To this end, we gathered, to the extent possible, data from beneficiary databases from supported interventions. The results presented herein are preliminary. At a later stage, we will perform focus groups with the beneficiaries of direct interventions to understand which components might have worked or not and why. We will also collect data of at-risk youth in the targeted municipalities through a representative sample of communities with high homicide rates to measure individual-level exposure and attitudes toward violence. Preliminary conclusion and next steps: Our field research and evidence review suggest that crime in Mexico is a highly contextual, socio-culturally driven behavioral phenomenon. Therefore, a successful intervention also needs to take into account the broader context of violence to which program participants continue to be exposed during and after the program. However, our amassed secondary information shows that JPV-supported interventions usually only targeted 1 out of the 10 most violent neighborhoods within the municipalities of interest, and often targeting only a limited number of youth in these neighborhoods. Our preliminary results suggest that, while the intervention may have had a positive effect on individual participants in those municipalities, a wider coverage, both in terms of services offered and youth reached, would be necessary in order to obtain community-level effects in crime and violence reduction. In terms of services offered, we find that the designs of existing interventions do not sufficiently tackle youth risk factors and that gender roles are barely addressed. The programmatic offer currently covers impulsive behavior, anger management, and employability skills, and to a lesser extent substance consumption. However, data from the Youth-Targeting Tool (YTT) confirm that a wider range of interventions is needed to address youth risk factors, particularly to promote protective factors. Interventions that are able to promote protective factors and interrupt the community and family dynamics that produce antisocial behaviors remain urgent, as does expanding the offer of substance abuse treatment. 69 IV.6 COVERAGE AND TARGETING In terms of the number of at-risk youth reached by the JPV-supported interventions, we can only know the number of youth who have received at least one intervention by municipality, but we cannot speak to the effectiveness of these interventions (up to this point). Table 12 shows the number of beneficiaries by region.59 The number of beneficiaries is much higher in Ciudad Juárez and Monterrey than in the rest of the regions. But if we want to understand whether the level of coverage is enough to initiate a detectable reduction in youth involvement in crime and violence, then we need a different type of analysis. Following the evidence that crime is concentrated in certain people, places, and behaviors, we need to understand whether benefited youth come from high-risk places. We were able to perform this analysis by linking data of the beneficiaries’ neighborhoods of residence, available from intervention databases, with neighborhood-level homicide data that we requested from local authorities. 60 For this analysis, we worked with the individual databases of JPV-supported interventions. One important caveat is that the YTT does not have beneficiaries’ neighborhoods of residence, and intervention databases do not have complete information of the YTT, which prohibits crossing the information of risk factors by neighborhood or even by municipality.61 Also, the data are incomplete, and we have several youth for whom neighborhood of residence is not specified. Table 12. Beneficiaries by Gender and Region Municipality Male Female Total Chihuahua 76 32 108 Monterrey 416 138 554 Guadalajara 97 9 106 Ciudad Juárez 548 391 939 Tijuana 171 145 316 Morelia N/A N/A Other 150 214 216 Total 1,458 929 2,023 In Annex VIII, we present this targeting analysis, for which we rely on the data provided by local authorities. It is important for the reader to keep in mind that these numbers do not coincide with those reported by Mexico’s National Institute for Statistics and Geography (INEGI being its acronym in Spanish) or by the Mexico’s Secretary General of National Public Security (SESNSP being its acronym in Spanish). In Annex VIII, we present a comparative table showing the totals between these numbers, and then we show the top 10 neighborhoods in terms of homicides and the most common beneficiary neighborhoods registered in the intervention databases for Ciudad Juárez, Chihuahua, Tijuana, Escobedo, and Guadalupe. Unfortunately, we do not have data of 59 We got these numbers from intervention databases. See footnote 58. 60 We requested homicide data as follows: for Monterrey, Guadalupe, and Escobedo, we requested the data from the Nuevo Leon State Attorney General; for Ciudad Juárez and Chihuahua, we requested the data from the Chihuahua State Attorney General; and for Tijuana, we requested data from the Municipal Security Secretariat. 61 We tried to merge the YTT with the intervention beneficiary databases, but unfortunately, the individual identifiers do not coincide. It is also important to note that the total numbers of beneficiaries in both sets of data do not coincide. There are more beneficiaries in the individual databases than there are in the YTT, but still, for some interventions we do not have access to their database. 70 homicides at the neighborhood level for Morelia or any municipality of the metropolitan area of Guadalajara, and we do not present analysis for Monterrey because we have very few beneficiary observations with neighborhood of origin. The analysis shows that JPV-supported interventions work with only 1 of the top 10 neighborhoods in terms of homicides in Ciudad Juárez, Chihuahua, Tijuana, and Escobedo and work with 2 in Guadalupe. Moreover, our field research and evidence review suggest that crime in Mexico is a highly contextual, socio-culturally driven behavioral phenomenon. 62 Therefore, a successful intervention also needs to take into account the broader context of violence to which program participants continue to be exposed during and after the program. Our preliminary results suggest that, while the intervention may have had a positive effect on individual participants in those municipalities, a wider coverage, both in terms of services offered and youth reached, would be necessary in order to obtain community-level effects in crime and violence reduction. In terms of services offered, we analyzed the data available from the YTT and find that the designs of existing interventions do not sufficiently address youth risk factors, also confirming the interviews findings highlighted in the previous section. Table 13 shows the risk factors, ordered from the highest occurrence to the least for male beneficiaries, and Table 14 shows those for female beneficiaries. To build these tables, we summed the scores of each risk factor for all beneficiaries available in the database.63 The YTT scores show how acute each risk factor is for each beneficiary as follows: 0 is low, 1 is moderate, 2 is high, and 3 is critical. These tables confirm that that the top five risk factors are not necessarily covered by the programmatic offer, and moreover, they show important differences between genders. For both genders, within the top five risks, we find insecurity in the community and lack of parental figure. For females, among the top five risks factors, we find violence at home, lack of social support, and anger management, while for males we find difficulties in the school context, substance consumption, and impulsive behavior. The programmatic offer currently covers impulsive behavior, anger management, and, to a lesser extent, substance consumption, but data from the YTT confirm that a wider range of interventions is needed to address youth risk factors, particularly to promote protective factors. In this sense, interventions that are able to interrupt the community and family dynamics that produce antisocial behaviors are imperative, as is expanding the offer of substance abuse treatments. 62 See Bandura and Walters, 1977 and “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 63 We use data available for the YTT version 2, as this has significantly more beneficiaries. We did not sum version 1 and version 2 because not all variables are comparable. 71 Table 13. Male Beneficiaries’ Risk Factors (YTT V.2) Risk factor Total general Insecurity in the community 664 Difficulties due to substance consumption 580 School context 550 Lack of parental figure 498 Impulsive behavior 492 Peer crime involvement 486 Early start in violence 485 Breeding style 464 Anger management 464 Perceived social support 461 Exposure to violence at home 437 Attitude toward violence 421 History of childhood abuse 343 Physical violence 325 Conflict with the law 313 Empathy/regrets 300 History of nonviolent crime 293 Goal fulfillment 252 Criminality of parents or caregivers 228 Lack of attention / hyperactivity 228 Table 14. Female Beneficiaries’ Risk Factors (YTT V.2) Risk factor Total general Insecurity in the community 400 Lack of parental figure 336 Exposure to violence at home 304 Anger management 291 Perceived social support 277 Breeding style 270 Impulsive behavior 260 History of childhood abuse 234 School context 230 Attitude toward violence 215 Difficulties due to substance consumption 215 Early start in violence 201 Peer crime involvement 194 Goal fulfillment 148 Physical violence 140 Criminality of parents or caregivers 131 Lack of attention / hyperactivity 131 Empathy/regrets 92 History of nonviolent crime 77 Conflict with the law 66 The YTT data also show that gender roles affect intervention effectiveness, as misogynistic and established masculinity beliefs and values reinforce risk factors, such as violence normalization, particularly at home. However, gender roles were barely addressed by supported interventions. The program’s gender strategy was managed in parallel to the at-risk youth transversal strategy. To our knowledge, the only program that incorporates a more comprehensive gender perspective is Supera (with masculinity workshops and work with mothers and women neighbors), and CASA has masculinity workshops. Nevertheless, in several interviews, this topic came up as pending priority area. As two interviewees put it: “Once we organized a mixed soccer league, in which the team had to be conformed by three men and three women. We had to work a month and a half for youth to understand why.” “Another factor is associated with gender. We refer to the violence that some young women suffer from. Here we have several women that are used as shields; that is, they are the ones who carry the drugs and weapons to transport them from one place to another. Young women victims of abuse. As they grow up, they become part of the lines of organized crime.” While all of these data give us some insights, we need to understand youth perceptions, behaviors, and attitudes toward violence. At a later stage we will perform focus groups with direct beneficiaries of JPV-supported interventions to explore which components may have worked or not and why in reducing the likelihood of youth being involved in crime and violence. We will 72 also collect data of at-risk youth in the targeted municipalities through a representative sample of communities with high homicide rates to measure individual-level exposure and attitudes toward violence. This survey will be informative of the potential mechanisms through which the intervention may have affected the outcomes of interest. IV.7 JPV-SUPPORTED INTERVENTIONS RESULTS IN CRIME AND VIOLENCE REDUCTION IN TARGETED LOCALITIES 4. What evidence is there that the USAID/Mexico crime-and-violence-prevention programming through the JPV activity has contributed to achieving the goal of reducing crime and violence within the targeted JPV communities? c. Where and what are examples of the JPV activity contributing to reduced levels of crime and violence in the targeted municipalities? d. To what extent have citizen perceptions of security increased in targeted municipalities? Box 8: Question 4 Summary To answer this question, we perform a descriptive statistical analysis of existing secondary data on crime and homicide rates, as well on citizen perceptions of security. Preliminary conclusions and next steps: Our initial analysis finds that homicide rates present positive trends for all municipalities of interest since 2015. Insecurity perceptions keep rising in all municipalities, except for Guadalupe and Escobedo (although data for these municipalities are only available since June 2018). However, we remain very cautious and refrain from drawing any causal connections from this analysis. The increased levels of violence in Mexico are in large part attributable to organized crime. Consequently, insecurity perceptions are also driven by violence related to drug cartels and to other highly visible acts of violence that affect perceptions. Hence, a decrease in homicide rates and reducing insecurity perceptions, may require different strategies than the one envisioned for JPV and that entail longer-term institutional reform and more traditional law enforcement action. The National Survey of Urban Security (ENSU being its acronym in Spanish) provides other available indicators related to antisocial behaviors that may be better suited to explore program contribution to reducing the likelihood of youth engaging in crime and violence. Unfortunately, the lack of clear counterfactual information, along with limited socio-economic data, hindered our ability to isolate the effect of the program on youth population. From our field explorations we generate a plausible hypothesis that JPV could have impacted individual youth perceptions, attitudes, and exposure to violence. In order to address the challenges with the existing data, we have designed a unique and tailored survey of youth that will measure youth exposure, attitudes, and behaviors toward violence. Our stratified sampling procedure will include youth in the targeted municipalities as well as youth from similar municipalities which remained outside the realm of JPV, to form a comparison group. We perform a descriptive statistical analysis of existing secondary data on homicide and other crime rates, as well as on citizen perceptions of security, to understand the context in which this program was implemented and the observable trends in treated municipalities and the rest of the country. However, we remain very cautious and refrain from drawing any causal connections from this analysis. Increased levels of violence in Mexico are in large part attributable to organized crime. Estimations from different sources agree that over one-third of homicides are related to 73 this source of violence. 64 Consequently, insecurity perceptions are also driven by violence related to drug cartels and to other highly visible acts of violence that affect perceptions. Addressing this type of violence requires different strategies than the one pursued by JPV. A decrease in homicides rates and in general insecurity perceptions would require different strategies that entail longer-term institutional reform and more traditional law enforcement action. 65 In terms of homicides, we built a database that includes the monthly occurrence at the municipality level from 1990 to 2018. We created this database with data from INEGI, which uses vital statistics based on death certificates; the data tend to be more precise than those reported by SESNP.66 Nevertheless, in our database, we also include homicide statistics from SESNP from 2015 to 2020 to be able to also observe patterns in 2019 and 2020 (INEGI data, unfortunately, are published with over a year of delay). Our database also includes statistics on other crimes from SESNP (car robbery with and without violence, extortion, and kidnapping), and we also include other sociodemographic characteristics at the municipality level, such as age distribution, marginalization index, employment, and education from INEGI and the National Population Council (CONAPO being its acronym in Spanish). Finally, we have built an additional database, drawing on data from ENSU quarterly from January 2016 to March 2020. We have performed an initial descriptive analysis, for which graphs are available in Annex IX. We find that homicide rates present positive trends for all municipalities of interest since 2015. Insecurity perceptions keep rising in all municipalities, except for Guadalupe and Escobedo (although data for these municipalities are only available since June 2018). The ENSU provides other available indicators related to antisocial behaviors that may be better suited to explore program contribution to reducing the likelihood of youth engaging in crime and violence, for instance, perception data on alcohol and drug consumption and sale in public spaces, the existence of street gangs,67 and the percentage of people who report engaging in conflict. These data show mixed trends. For alcohol consumption in public spaces, trends are all negative (including the national average) for all municipalities of interest except Tijuana. Conversely, drug consumption and sale perception tendencies are positive for Tijuana, Morelia, Monterrey, and Ciudad Juárez, and are negative for Chihuahua, Escobedo, Guadalupe, Tonalá, and Zapopan (the national average tendency is slightly positive). The perceptions of the existence of street gangs have barely changed during the last 4 years, and the rates of people who report engaging in conflict are negative for all municipalities (the national average is also negative).68 It is important to keep in mind that that the intervention was highly heterogeneous, as implemented prevention strategies varied by municipality. Some had a greater focus on civic justice, while others focused on reintegration and territorial strategies. Therefore, a more nuanced analysis of mechanisms would be needed for each case. Furthermore, these data present three additional challenges. First, the sample is only representative of the population who are 18 years and older. Second, the data are based purely on perceptions, which may have respondent 64 See Calderón et al., 2018. 65 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 66 For a more nuanced analysis of the differences in crime reports and potential underreporting, see Diego Valle - Jones blog: https://elcri.men/en/underreporting-crime/ 67 Pandillerismo in Spanish; there is no exact translation for this term. 68 For Tonalá, Zapopan, Escobedo, and Guadalupe, data are only available since June 2018. 74 bias. Third, the data measure perceptions on the existence of drug and alcohol consumption and street gangs, but do not directly measure youth participation in these activities. Given all of this and the lack of a counterfactual, the effect the program may have had directly on youth cannot be disentangled conclusively. The preliminary statistics we present aim only to provide the reader with a descriptive overview of the trends in the municipalities of interest. From our field explorations we generate a plausible hypothesis that JPV could have impacted individual youth perceptions, attitudes, and exposure to violence. In order to address the aforementioned challenges to the extent possible, we will conduct a youth survey that will measure youth exposure, attitudes, and behaviors toward violence. This sample will be representative of youth in the municipalities of interest and will also include a set of similar municipalities to descriptively compare outcomes with a more representative view. We will build this sample with data gathered from INEGI and CONAPO. Finally, we will address potential respondent bias by embedding a vignette experiment into the survey to measure the effect of hypothetical exposure to the intervention on subjects’ attitudes toward criminality. As explained in the Methodology sections, vignette studies use short descriptions of hypothetical scenarios (vignettes) that are usually presented to respondents within surveys in order to elicit how their judgments about such scenarios affect outcomes of interest, often revealing their perceptions, values, or social norms. This survey will help identify potential mechanisms through which the intervention may have affected the outcomes of interest. 75 V. RECOMMENDATIONS The purpose of this performance evaluation is to gauge the effectiveness of USAID’s JPV program in promoting reductions in levels of crime and violence and gather recommendations for improved future programming. In particular, this evaluation provides recommendations on more impactful, proven, and sustained approaches for future crime-and-violence-prevention programs. Based on JPV’s learnings, we provide the following recommendations to inform future policies. 1. Define narrower systems for crime and violence prevention. The local systems framework provides a practical perspective to engage and empower local actors in collaborations to achieve specific and sustained development results. One key feature to make the local systems framework successful is to build such systems around definite outcomes. We recommend that the result structures respond to the program’s immediate outcome rather than the program’s long-term development goal. Particularly in issues as complex as crime and violence prevention and in settings where the drivers are too diverse, systems must address narrower results. Otherwise, they will run the risk of being too broad, making responsibilities and contributions confusing, thereby collaborations weak. Furthermore, the result should be commensurate with the program’s reach. To achieve this, the local systems can be defined around the goals of reducing youth participation in violence, construction of a culture of legality, or reducing gender violence. We deem that these narrower goals would allow for more effective interaction with the systems. Additionally, concentrated efforts may also decrease the number of conduit organizations increasing bureaucratic efficiency as well as reducing costs. A few corollary recommendations also emerge. a. Facilitate communication channels between specialized systems that contribute to the same long-term development goal. Different systems might be contributing to the same development goal. The local systems approach should consider facilitating communication channels between systems. This strategy could leverage the potential overlap of a few actors between systems. For instance, the executive branch of a municipal government might take part in all three of the systems highlighted above as examples. b. Work with diverse leadership. The strategy to engage with the system must respond to natural leadership and should leverage comparative advantages. More specialized systems also facilitate the recognition of leadership, which allows for a finer engagement. c. Develop local agendas around specific strategies with a clear technical offer, defined target populations and indicators, and clear responsibilities and goals. d. Generate networks of specialized knowledge. These networks should respond to the accumulation of knowledge about techniques, models, and tools to work with the targeted goals, for instance, for reintegration strategies or domestic violence. e. Program evaluation indicators should respond to narrower goals. The project monitoring, evaluation, and learning plan for activities that involve local systems 76 should respond to the narrower goals (the same as those around which the systems will be organized) rather than to the long-term development goal. To this end, it is worth working with the theory of change and thinking backward about the steps needed for a required change. In this mapping, the links between the program activities and expected outcomes should become clear. 2. Focus efforts on generating a wider range of models for at-risk youth by giving clear tools to local actors for catalyzing new models and scaling up. The aim should be to enable the environment for specific and promising innovations to be able to thrive, to generate knowledge, and to identify new effective solutions with the potential for scaling. Unfortunately, the evidence base on what works to reduce violence among youth is relatively narrow. Governments, CSOs, researchers, and the private sector are addressing this issue by conducting a variety of policy experimentations. The objective of aid programming should be on developing specific subject-matter related technical expertise, implementation, and M&E capacities. In particular, investment should achieve the following. a. Develop intervention models adapted to the local context on interventions for which there is a strong evidence base, in particular multidimensional therapy models, addiction treatment, and psychiatric approaches. Given the risk factors presented for youth, the rapid development of these models is imperative. b. Foster innovation based on the concentration, coordination, and proactivity principles. When the existing evidence base of effective interventions is not enough, the approach should be to leverage local innovations and to provide the necessary resources to solidify models following these key principles. 3. Adapt, pilot, and then scale. Evidence-based policy design consists of a constant process of adaptation and innovation, testing, and scaling up. As explained in the USAID J-PAL–commissioned report, “Programs that are effective at improving outcomes in one context may help in other places where the key problems and underlying reasons for the problems are similar. But since no two contexts are identical, scaling programs in a different place is not as simple as transferring an ‘out of the box’ solution. To scale appropriately and effectively, policymakers and researchers must account for local conditions and implementation capacity, and then adapt programs to that new context accordingly.”69 It is imperative that international cooperation initiatives take careful consideration of the need to adapt programs. Even within a country, local contexts differ vastly. The risk of scaling up without the adaptation piece relies on the effectiveness of the intervention and, as a consequence, may harm credibility and prevent further adoptions. In particular, when considering scaling up, it is necessary to: 69 For more information, see https://www.povertyactionlab.org/evidence-to-policy/adapting-and-scaling-program 77 a. Allow enough time to refine implementation. Organizations should be provided sufficient time to refine and consolidate all of the model components before scaling up a strategy. This will ease transfers and secure fidelity by establishing protocols to monitor whether the implementation follows the desired theory of change. b. Foster comprehensive processes of replication that consider differences in context. When adapting a program, careful consideration should be given to the local context and to whether the theory of change applies to the new circumstance. It is important to verify if the conditions are conducive for the implementation to be carried out as desired. If the local conditions allow for adoption of the essential components of the theory of change, then the nonessential components should be adapted to the local context. If the essential components cannot be held under the new circumstance, then other policy alternatives should be explored. 70 c. Invest in a few strong pilots, which might bring a higher payoff than scaling up suboptimal models. International cooperation can play a groundbreaking role by catalyzing innovations at the local level and shifting incentives to attract the attention of other actors including the government and private sector, which have the power of investing at scale. In this sense, the investments of programs such as JPV may bring higher payoffs by nurturing promising interventions, enabling the conditions for them to thrive and generating rigorous evidence about their effectiveness. These strong and evidence-based examples can then become catalysts of further innovations in the medium and long terms. The approach should be to rigorously evaluate pilots and generate data driven evidence of success at a small scale before expanding models. d. Evaluate. Rigorous evaluation is a key element for the transfer, adaptation, and replication processes, and it is imperative that the following USAID activities in Mexico prioritize evaluating the interventions before continuing their expansion. Process evaluations are needed to ensure program fidelity, and impact evaluations are needed to confirm whether the program’s intended mechanisms for change work as expected. 4. Engage with government agencies to create specialized capacity for the design and implementation of specific services aimed at reducing recidivism. Increasing the capacity of CSOs to work with at-risk youth is paramount, especially given that the civil society sector has more flexibility to try out innovative approaches. However, there is also a need to generate specialized capacity within governments to support youth to build alternative pathways to crime and violence is of the greatest priority. In particular, it is imperative to achieve the following: a. Generate capacity within the police and judicial systems. Police departments and the judicial system need to shift away from a public safety and 70 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 78 punitive approach to a public health perspective to prevent youth crime and violence. This approach should be based on an understanding of youth protective and risk factors. Law enforcement and judicial prosecutors need to work jointly with service providers to facilitate the long-term recovery of offenders. Working with these institutions to further this approach is a necessary condition for prevention policies to thrive. There is an enormous area of opportunity to work with the Mexican government at all levels to translate the National Juvenile System Law approved in 2016 into real policy alternatives for youth. There are opportunities to augment the judicial system’s human capital and technology and to develop clear protocols and channels to work with youth. The aims should be to avoid the “revolving door” phenomenon for minor offenses by creating a system that supports youth while keeping them accountable. The judiciary needs to ensure that prosecuted cases send youth to the services they need (more on this below). 71 b. Generate capacity within governments for the provision of social services to at-risk youth. Related to the above, combining post-release or alternatives to incarceration with comprehensive social services that are aimed at remedying the behavior leading to the original arrest and that promote safety, strength, and permanence in the community, schools, home, or employment while keeping youth accountable72 is the most promising approach to reduce recidivism sustainably. These services should coordinate health and mental health services, education and employment support, and family counsel. There remain ample opportunities to engage with public institutions to generate this specific offer. c. Facilitate the development of information systems. To better diagnose and target policies, governments need to significantly increase their technological capacities to gather and share data, and international cooperation is well positioned to support this goal. Data should include, among others, georeferenced incidents, settings and circumstances of incidents, deaths due to violence, and nonfatal violence-related injuries. Information systems should incorporate data from different departments including police, justice, labor, education, and health.73 Moreover, it is paramount that data systems are comparable between municipalities and that data are shared in bigger geographical areas. This is particularly relevant for metropolitan areas. d. Facilitate youth case follow-up. There is enormous room to generate information systems that facilitate youth follow-up and coordination between agencies and CSOs. While the YTT is a good start, we recommend making the platform more flexible. That is, instead of having predetermined variables, such a platform could allow organizations to define their own variables, on top of choosing from an existing pool, to define the monitoring protocol that makes more sense. By making this platform more flexible, institutions will also be able to further exploit existing information. 71 See “What Works to Prevent Violence Among Youth?”, USAID J-PAL–commissioned report, 2018. 72 See Cook et al., 2015. 73 World Health Organization, 2015. 79 REFERENCES Abt, Thomas P. 2017. “Towards a framework for preventing community violence among youth.” Psychology, Health & Medicine 22 (1): 266-285. doi:10.1080/13548506.2016.1257815 Bandura, Albert, and Richard H. Walters. Social learning theory. Vol. 1. Englewood Cliffs, NJ: Prentice-hall, 1977. Blattman, Christopher, Julian C. Jamison, and Margaret Sheridan. 2017. "Reducing Crime and Violence: Experimental Evidence from Cognitive Behavioral Therapy in Liberia." American Economic Review 107 (4): 1165–1206. Calderón, Laura, Octavio Rodríguez and David A. Shirk. 2018. “Drug Violence in Mexico: Data and Analysis Through 2017” Justice in México Project (april). https://justiceinmexico.org/wp-content/uploads/2018/04/180411_DrugViolenceinMexico-12mb.pdf Cook et al., 2015, An Experimental Evaluation of a Comprehensive Employment-Oriented Prisoner Re-entry Program, Journal of Quantitative Criminology. Easton, Caroline J., Dolores Mandel, Karen Hunkele, Charla Nich, Bruce Rounsaville, and Kathleen Carroll. 2007. “A cognitive behavioral therapy for alcohol-dependent domestic violence offenders: an integrated substance abuse-domestic violence treatment approach (SADV).” The American Journal on Addictions 16 (1): 24-31. doi:10.1080/10550490601077809 Friedman, Alfred S., Arlene Terras, and Kimberly Glassman. 2008. “Multimodel Substance Use Intervention Program for Male Delinquents.” Journal of Child & Adolescent Substance Abuse 11 (4): 43-65. doi: 10.1300/J029v11n04_03 Heller, Sara B., Anuj K. Shah, Jonathan Guryan, Jens Ludwig, Sendhil Mullainathan, and Harold A. Pollack. 2017. "Thinking, Fast and Slow? Some Field Experiments to Reduce Crime and Dropout in Chicago." The Quarterly Journal of Economics 132 (1): 1–54. Office of Research, “Federal Funding (Grants vs. Cooperative Agreements vs. Contracts)”, UC Santa Cruz. https://officeofresearch.ucsc.edu/osp/prep/fed-funding-mechanisms.html USAID, “Evidence Base for Collaborating, Learning and Adapting. A summary of the literature review”, First Published March 2020. https://usaidlearninglab.org/sites/default/files/resource/files/eb4cla_summary_literaturereview_20171020 _final.pdf USAID, “Executive Summary: Estimation of the costs of crime and violence for the private sector in Mexico and the importance of its participation in violence prevention as a strategy for its long-term business model”. First Published June 2019. USAID, “Grant and Contract Process”, Last updated January, 2020. https://www.usaid.gov/work-usaid/get-grant-or-contract/grant-and-contract-process 80 USAID J-PAL–commissioned report, “What Works to Prevent Violence Among Youth? A White Paper on Youth Violence, Crime Prevention, and the Mexican Context.” First Published September 2018. http://www.cca.org.mx/forocdmx/pdfs/White%20Paper%20JPAL_Ingles_Final.pdf USAID, “Local Systems: A Framework for Supporting Sustained Development”, First Published April 2014. https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf USAID, “Performance Evaluation Designs”, First Published September 2019. https://www.usaid.gov/project-starter/program-cycle/project-monitor-evaluation-plan/monitor￾evaluation-plan-evaluation-component/performance-evaluation-designs USAID, “The 5Rs Framework in the Program Cycle”. First Published October 2016. https://usaidlearninglab.org/sites/default/files/resource/files/5rs_techncial_note_ver_2_1_final.pdf USAID Learning Lab, “Introduction to Collaborating, Learning and Adapting (CLA) in the Program Cycle”, 2019. https://usaidlearninglab.org/course-introduction-cla-program￾cycle#:~:text=Collaborating%2C%20Learning%2C%20and%20Adapting%20(,their%20Journey%20to%20S elf%2DReliance. Vaughn, Michael S., Furjen Deng, and Lou-Jou Lee. 2003. “Evaluating a Prison-Based Drug Treatment Program in Taiwan.” Journal of Drug Issues 33, (2): 357–83. doi:10.1177/002204260303300205. World Health Organization. Preventing youth violence: an overview of the evidence. World Health Organization, 2015. 81 ANNEX I. PROGRAM BACKGROUND Table 1. Juntos para la Prevención de la Violencia Results Framework Goal: Reduce violence in target regions Development objective: Contribute to the reduction of violence and crime by strengthening adaptation and implementation capacities of local systems of prevention. Intermediate Result 1. Support the Government of Mexico (GOM) in improving national violence and crime prevention policy and with its collaboration with local actors. Intermediate Result 2. Improve the performance of actors within local violence and crime prevention systems. Intermediate Result 3. Contribute to local systems generating and implementing evidence-based violence and crime prevention policy. Component 1.1. The Government of Mexico adopts and applies continuous training and knowledge management frameworks to public officials on matters of prevention. Component 2.1. The local actors have improved their performance in matter of institutional enforcement and crime prevention Component 3.1. Actors within local prevention systems have improved their capacities to produce evidence Component 1.2. The Government of Mexico adopts and applies tools that facilitate the design and the implementation of the national violence and crime prevention policy. Component 2.2. Strategic actors within the local system collaborate in more efficient ways in CVP initiatives Component 3.2. Local actors have increased the demand and use of knowledge on matters of violence and crime prevention 82 ANNEX II. DATA COLLECTION INSTRUMENTS II.1 ONLINE SURVEY I. General Information 1. Name of organization: 2. Region: a. Tijuana, Baja California b. Chihuahua, Chihuahua c. Ciudad Juárez, Chihuahua d. Metropolitan Area of Monterrey, Nuevo León e. Metropolitan Area of Guadalajara, Jalisco f. Morelia, Michoacán 3. Sector: a. Local Government b. State Government c. Federal Government d. Private Sector e. Civil Society f. Academia 4. Current Role: a. Management or Leadership b. Administration c. Operational Staff d. Other: ____________ 5. Has your organization received financial assistance from JPV? a. Yes b. No c. N/A 6. Has your organization received technical assistance from JPV? Technical assistance here refers to professional or special services to support and strengthen the technical and organizational capacities of organizations, including guided instruction, skills training, methodologies and materials development, strategic planning, transmission of special expertise, among others. a. Yes b. No c. N/A 7. Does your organization directly implement any project, program, or public policy to prevent violence or crime? a. Yes b. No 8. Does your organization support any project, program, or public policy to prevent violence or crime by means of financial or technical assistance? a. Yes b. No Please answer the following questions based on your interactions with JPV/USAID over the course of the past four years. 83 II. Resources These questions all pertain to the resources that the JPV/USAID program has provided: 1. Which of the following resources provided by JPV/USAID to strengthen local systems and improve program implementation are you familiar with? (Please select all that apply) a. Systematization of best practices b. Publications with evidence and research initiatives c. Tools developed by JPV / USAID (YTT, GOPI -P, OPI-P, EB P) YTT = Youth Target Tool GOPI = Government Performance Index OPI = Organizational Performance Index EBP = Scale of Best Practices d. Guides and manuals for the implementation of violence prevention programs e. Personnel training (diagnosis, evidence-based intervention design, theory of change and indicator design, monitoring and evaluation, communication) f. Study Trips g. National/International events (conferences, forums, summits, etc.) h. Diffusion or promotion of best practices i. Other: __________ 2. Which of the following resources provided by JPV/USAID to strengthen local systems and improve program implementation do you consistently use at your work? a. Systematization of best practices b. Publications with evidence and research initiatives c. Tools developed by JPV / USAID (YTT, GOPI -P, OPI-P, EBP) d. Guides and manuals for the implementation of violence prevention programs e. Personnel training (diagnosis, evidence-based intervention design, theory of change and indicator design, monitoring and evaluation, communication) f. Study Trips g. National/International events (conferences, forums, summits, etc.) h. Diffusion or promotion of best practices i. Other: __________ j. None 3. On a scale of 1 to 5, with 1 being least useful and 5 being most useful, how useful would you consider the following resources in informing your daily operations? a. Systematization of best practices (1 , 2 , 3 , 4 , 5) b. Publications with evidence and research initiatives (1 , 2 , 3 , 4 , 5) c. Tools developed by JPV / USAID (YTT, GOPI -P, OPI-P, EB P) YTT = Youth Target Tool GOPI = Government Performance Index OPI = Organizational Performance Index EBP = Scale of Best Practices (1 , 2 , 3 , 4 , 5) d. Guides and manuals for the implementation of violence prevention programs (1 , 2 , 3 , 4 , 5) e. Personnel training (diagnosis, evidence-based intervention design, theory of change and indicator design, monitoring and evaluation, communication) (1 , 2 , 3 , 4 , 5) 84 f. Study Trips (1 , 2 , 3 , 4 , 5) g. National/International events (conferences, forums, summits, etc.) (1 , 2 , 3 , 4 , 5) h. Diffusion or promotion of best practices (1 , 2 , 3 , 4 , 5) 4. If you have not participated in the events or personnel trainings provided by JPV/USAID, what is the reason? Please select all that apply. a. Events and/or personnel training have not been developed in your region b. You participated before and materials had not been updated. c. You have access to other events and / or trainings through your own organization or others. d. They don’t seem useful e. They are not necessary for the role that you perform f. You were not aware of these resources and/or were not made aware of the invitation g. Other: __________ 5. Why have you not used the tools and/or publications developed by JPV/USAID? a. They are difficult to interpret and apply b. They are not easily accessible c. There are better tools and/or publications in your own organization, internally or via other organizations d. They don’t seem useful e. They are not necessary for the role that you perform f. You were not aware of these resources g. Other: __________ 6. Which of the following resources do you consider most useful for the identification, design, and implementation of best practices? (Select all that apply) a. Systematization of best practices b. Publications with evidence and research initiatives c. Tools developed by JPV / USAID (YTT, GOPI -P, OPI-P, EBP) d. Guides and manuals for the implementation of violence prevention programs e. Personnel training (diagnosis, evidence-based intervention design, theory of change and indicator design, monitoring and evaluation, communication) f. Study Trips g. National/International events (conferences, forums, summits, etc.) h. Dissemination or promotion of best practices i. Other: __________ 7. To what extent do you agree or disagree with the following statements related to the technical capacity trainings provided by the JPV/USAID program? a. For the most part, the training material was technically solid. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 85 b. The training materials were relevant for my organization. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree c. JPV/USAID provided the support and follow-up necessary to implement the capacities learned in each training. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree *Please leave any additional comments here: ___________ 8. Would you say that your organization has been able to adopt the capacities developed with the support of JPV/USAID? a. Yes b. No c. N/A 9.In what way(s) has your organization adopted or institutionalized the capacities developed? ________________________________________________________________ 10. Do you consider the resources and supplies provided by JPV/USAID to be consistent and predictable enough to plan for their efficient use in the future? Consistent here means that there is no time variation and predictable means that you can know with certainty that they will be available in the future. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 11. As a product of cooperation with JPV/USAID, has your organization identified alternative sources for additional financing from other actors￾private sector, government funding, trusts or other- be they local or outside, to ensure the sustainability of its prevention initiatives? a. Yes b. No 2. What necessary resources could be provided through international cooperation to strengthen local prevention systems that do not currently exist or exist in limited fashion? _______________________________ 86 III. Roles The following questions will pertain to the participation of relevant actors and their roles in the local prevention systems: 1. To what extent would you agree or disagree with the following statements regarding the local context? a. JPV/USAID effectively identified the relevant local actors who work in crime and violence prevention in your region. (including the private sector, government, academia, and civil society) a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree b. JPV/USAID has a good understanding of the local context and the challenges faced by relevant actors. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree c. JPV/USAID’s plan of action and recommendations are supported by a thorough understanding of local contexts. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree *Please leave any additional comments here: ___________ 2. How would you rate the performance of the following actors in preventing crime and violence in your region in terms of efficiency? By efficient we refer to their capacity to adequately perform or complete their respective role. a. State Government a. Excellent b. Good c. Neutral d. Poor e. Very poor b. Municipal Government a. Excellent b. Good c. Neutral d. Poor e. Very poor c. Private Sector 87 a. Excellent b. Good c. Neutral d. Poor e. Very poor d. Civil Society a. Excellent b. Good c. Neutral d. Poor e. Very poor e. International Cooperation a. Excellent b. Good c. Neutral d. Poor e. Very poor f. Academia a. Excellent b. Good c. Neutral d. Poor e. Very poor 3. Who do you believe have the most influence in policies for crime and violence prevention in your region? On a scale from 1 to 5, please indicate who you believe to have the most influence, with 1 being least influential and 5 being most influential. a. Civil society (1 , 2 , 3 , 4 , 5) b. International cooperation (1 , 2 , 3 , 4 , 5) c. State Government (1 , 2 , 3 , 4 , 5) d. Municipal Government (1 , 2 , 3 , 4 , 5) e. Academia (1 , 2 , 3 , 4 , 5) f. Private sector (1 , 2 , 3 , 4 , 5) 4. Would you say that JPV/USAID involves relevant actors in the activities it organizes? a. JPV/USAID involves all relevant actors b. JPV/USAID involves the majority of relevant actors c. JPV/USAID involves some relevant actors d. JPV/USAID involves very few relevant actors e. JPV/USAID does not involve any of the general actors 5. How would you describe the changes to private sector involvement in crime and violence prevention has changed since the beginning of the JPV/USAID program? a. The private sector is more involved b. The private sector is somewhat more involved 88 c. Private sector involvement is unchanged d. The private sector is somewhat less involved e. The private sector is less involved/not involved 6. How much influence does civil society have in the local crime and violence prevention system? a. Local system actors always take the opinion of civil society into account. b. Local system actors usually take the opinion of civil society into account. c. Local system actors sometimes take the opinion of civil society into account. d. Local system actors rarely take the opinion of civil society into account. e. Local actors never take the opinion of civil society into account. To what extent would you agree or disagree with the following statements regarding the local context? 7. JPV has been able to empower Civil Society organizations to occupy a more prominent role in violence prevention. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 8. JPV has facilitated improved communication in the local prevention network. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 9. There is clear-cut leadership that can bring together the community within the local violence prevention system. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 10. Where does this leadership stem from? a. Civil Society b. International Cooperation (JPV/USAID) c. State Government d. Municipal Government e. Academia f. Private Sector 11. The leaders in the local prevention system are effective in coordinating local system actors and using the existing resources. a. Strongly Agree 89 b. Agree c. Neutral d. Disagree e. Strongly Disagree 12. Your region has sufficient capacity to effectively address youth at risk of involvement in criminal or violent activity. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 13. How has this capacity changed since JPV/USAID involvement in your region? a. It has significantly increased b. It has increased c. It has remained the same d. It has decreased e. It has significantly decreased IV. Relationships The following questions pertain to the communications and relationships between local s: 1. How would you characterize your relationship with JPV/USAID? On a scale from 1 to 5, please indicate to what degree you would consider each adjective applicable, with 1 being not at all and 5 very much. a. Hierarchical (1 , 2 , 3 , 4 , 5) b. Reciprocal (1 , 2 , 3 , 4 , 5) c. Collaborative (1 , 2 , 3 , 4 , 5) d. Beneficial (1 , 2 , 3 , 4 , 5) e. Conflictive (1 , 2 , 3 , 4 , 5) 2. How would you characterize your relationship with similar organizations to your own that work in the local prevention system? a. Very strong b. Somewhat strong c. Neither strong or weak d. Somewhat weak e. Very weak 3. How would you characterize your relationship with Academia? On a scale from 1 to 5, please indicate to what degree you would consider each adjective applicable, with 1 being not at all and 5 very much. a. Hierarchical (1 , 2 , 3 , 4 , 5) b. Reciprocal (1 , 2 , 3 , 4 , 5) c. Collaborative (1 , 2 , 3 , 4 , 5) d. Beneficial (1 , 2 , 3 , 4 , 5) e. Conflictive (1 , 2 , 3 , 4 , 5) 90 4. How would you characterize your relationship with the municipal government? On a scale from 1 to 5, please indicate to what degree you would consider each adjective applicable, with 1 being not at all and 5 very much. f. Hierarchical (1 , 2 , 3 , 4 , 5) g. Reciprocal (1 , 2 , 3 , 4 , 5) h. Collaborative (1 , 2 , 3 , 4 , 5) i. Beneficial (1 , 2 , 3 , 4 , 5) j. Conflictive (1 , 2 , 3 , 4 , 5) 5. How would you characterize your relationship with the state government? On a scale from 1 to 5, please indicate to what degree you would consider each adjective applicable, with 1 being not at all and 5 very much. a. Hierarchical (1 , 2 , 3 , 4 , 5) b. Reciprocal (1 , 2 , 3 , 4 , 5) c. Collaborative (1 , 2 , 3 , 4 , 5) d. Beneficial (1 , 2 , 3 , 4 , 5) e. Conflictive (1 , 2 , 3 , 4 , 5) f. N/A 6. How would you characterize your relationship with local foundations, if you receive funds from one of them? On a scale from 1 to 5, please indicate to what degree you would consider each adjective applicable, with 1 being not at all and 5 very much. a. Hierarchical (1 , 2 , 3 , 4 , 5) b. Reciprocal (1 , 2 , 3 , 4 , 5) c. Collaborative (1 , 2 , 3 , 4 , 5) d. Beneficial (1 , 2 , 3 , 4 , 5) e. Conflictive (1 , 2 , 3 , 4 , 5) f. N/A 7. If the JPV/USAID program were terminated, how would the spaces for sharing ideas/experiences between other local system actors be affected? 8. Would you say that JPV/USAID’s feedback has been valuable? a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree *Please leave any additional comments here ____________ 9. Have you had opportunities to give JPV/USAID your feedback? a. Yes b. No Why or why not? 91 10. JPV has taken your feedback into account. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree * Please leave any additional comments here:_______ V. Rules The following questions pertain to the details of your collaboration with JPV/USAID: 1. What type of contractual relationship do/did you have with JPV/USAID? a. Subcontract b. Donation c. Cooperation Agreement d. No formal relationship e. Other: _________ 2. Who assumed more of the risk in this contractual relationship? By risks we refer to financial, programmatic risks, as well as risks to reputation. Please indicate a position on the line corresponding to who assumed more risk. _________________________________________________________________ 3. Please indicate, from 1 to 5, how relevant you think each criterion was for JPV/USAID to determine which organizations would receive a donation. On a scale from 1 to 5, please indicate how relevant would you consider each criterion, with 1 being not at all relevant and 5 very relevant. a. Evidence-based practices (1 , 2 , 3 , 4 , 5) b. Implementer’s organizational capacity (1 , 2 , 3 , 4 , 5) c. Implementer’s innovative capacity (1 , 2 , 3 , 4 , 5) d. Low risk associated with the donation (1 , 2 , 3 , 4 , 5) 4. The contractual terms of the relationship with USAID allowed for the flexibility necessary to implement your program. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree f. N/A * Please leave any additional comments here:________________________________________ JPV/USAID Risk equally shared between parties My organization 92 5. The adjustments made to your program design as a result of cooperation with JPV/USAID improved the effectiveness of its implementation. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree f. N/A *Please leave any additional comments here:_______ 6. Who had more decision-making power throughout the implementation of the program? Please indicate a position on the line corresponding with who had more decision-making power. _________________________________________________________________ 7. Implementation plans commonly have to make adjustments as public policy advances. In your opinion, how receptive were JPV/USAID to the necessity of these adjustments? a. JPV/USAID was very receptive and facilitated the necessary changes b. JPV/USAID was somewhat receptive and facilitated some of the necessary changes c. JPV/USAID was little receptive, and facilitated few of the necessary changes d. JVP/USAID was not receptive. 8. How would you classify the reporting requirements on the progress of your program to JPV / USAID? a. Minimal b. Less than sufficient c. Sufficient d. More than sufficient e. Excessive *Please leave us a comment regarding your perception of the reporting requirements: _______ 9. The indicators you must report to JPV/USAID are: A. Useful a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree B. Reflective of the advances made by the program a. Strongly Agree b. Agree JPV/USAID Same decision￾making power My organization 93 c. Neutral d. Disagree e. Strongly Disagree C. Relevant a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 2. The time dedicated to completing the required reports for USAID was: a. Very useful b. Somewhat useful c. Neither useful or useless d. Somewhat useless e. Very useless *Please leave any additional comments here:___________ 2. In your opinion, the objectives established in the beginning of your collaboration with JPV/USAID were: a. Pessimistic b. Somewhat pessimistic c. Neither pessimistic or optimistic d. Somewhat optimistic e. Very optimistic 3. You had to commit yourself to some unrealistic goals in order to receive funding from JPV/USAID. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 4. The implementation calendars committed to JPV are realistic a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree VI. Results The final questions pertain to the results that the program with JPV/USAID has achieved: 94 1. To what extent do you agree or disagree with the following statements related to local crime and violence prevention systems? A. JPV/USAID contribution led to local system actors identifying, designing, and implemented programmatic high-potential prevention policies a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree B. JPV/USAID contributed to local actors in the violence prevention system assigning more resources to effective prevention strategies. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree C. JPV/USAID contributed to a diverse set of actors being more sensible about the importance of crime and violence prevention. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree D. JPV/USAID made it feasible for all actors to participate and work as a local system. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 2. On a scale from 1 to 5, with 1 being very little change and 5 being drastic change, how much do you consider JPV/USAID was capable of positively influencing the institution below in their crime and violence prevention work? a. State government (1 , 2 , 3 , 4 , 5) b. Municipal government (1 , 2 , 3 , 4 , 5) c. Civil society (1 , 2 , 3 , 4 , 5) d. Academia (1 , 2 , 3 , 4 , 5) e. Private sector (1 , 2 , 3 , 4 , 5) * Please leave any additional comments here: _________________________ 3. To what extent do you agree or disagree with the following statements related to the role of your organization as it compares to JPV/USAID and other system actors? A. JPV/USAID made it easier for your agency to assume a local leadership role vis-à-vis other actors. a. Strongly Agree b. Agree c. Neutral 95 d. Disagree e. Strongly Disagree B. JPV/USAID made it easier for your agency to gain local recognition vis-à-vis other actors. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree C. JPV/USAID made it easier for your agency to implement the practices, tools, and initiatives developed in conjunction with JPV/USAID. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree D. JPV/USAID made it easier for your organization to develop a broader network of contacts that work in crime and violence prevention. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 4. Has your organization managed to have influence on local regulations regarding crime and violence prevention and/or has influenced in the local government proposed public policy during the last 4 years? Influence here refers to actions that organizations have taken with the intention of directly altering political decision, actions, and policies that are designed, adopted, or implemented by Governmental a. Yes b. No c. N/A. Please explain:__________________ 5. Do you know about any organization besides yours that has influenced in local violence prevention regulations and/or adoption of proposed public policies during the last four years? a. Yes (If so, which one?) b. No 6. Would you say that JPV/USAID created the conditions for your organization an impact on public policies for crime and violence prevention? a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 96 7. To what extent do you agree or disagree with the following statements related to the implementation of best practices? A. JPV/USAID contribution led to implementing organizations from the local crime and violence prevention system to elaborate diagnosis with disaggregated data that take sociodemographic characteristics of the region and risk factors of the population being served. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree B. JPV/USAID contribution provided implementer organizations with more tools to identify and follow up with the beneficiaries of the interventions. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree C. JPV/USAID contributed to implementer organizations' improved capacity and more tools to monitor and evaluate the success of interventions. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree D. JPV/USAID contributed to implementer organizations implementing best practices for prevention in a sustainable way. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree E. JPV/USAID contributed to local organizations implementing prevention strategies that will have continuity in the long run. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree * Please leave any additional comments here:_________________________ Only a few questions left. 8. To what extent do you agree or disagree with the following statements related to local innovation and transference of best practices? A. JPV/USAID made it easier for local system actors to access best practices materials developed by other actors. a. Strongly Agree 97 b. Agree c. Neutral d. Disagree e. Strongly Disagree B. JPV/USAID contribution led to local systems catalyzing innovations at a local level. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree C. JPV/USAID made the transfer of knowledge between actors easier. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree ¿ Which innovations you know produced at a local level have been replicated elsewhere or scaled by other actors? ______________________ 9. To what extent do you agree or disagree with the following statements related to monitoring and evaluation processes? A. JPV/USAID contributed better indicators for monitoring the progress of the implemented programs' activities. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree B. JPV/USAID made it easier for your organization to develop the capacity to carry out monitoring and evaluation by themselves. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree C. JPV/USAID facilitated cooperation with organizations of process evaluation experts. a. Strongly Agree b. Agree c. Neutral d. Disagree e. Strongly Disagree 10. If you did not have support from USAID, what would you consider to be the principal challenges facing your organization? Why? 11. Notwithstanding support from USAID, what would you consider to be the principal challenges facing the local prevention system? Why? 98 12. Would you say that the local prevention system is adaptable and capable of responding to political changes? Why or why not? 13. What could the program have done differently/ what would you recommend for future initiatives? 99 II. 2 GENERAL INSTRUMENT FOR SEMI-STRUCTURED INTERVIEWS (RELEVANT STAKEHOLDERS) Initial Questions: 1. To begin, we would like to hear more about your experience working with JPV/USAID. When did you begin collaborating with them? How closely did you work together? 2. Which aspects of your collaboration with JPV/USAID were the most positive? What has worked well? 3. Which aspects of your collaboration could be improved? What has not worked well? 4. What are the principal limitations the JPV/USAID confronted in the local prevention system? Questions pertaining to local prevention systems: I. Resources 1. Did you receive resources or materials from JPV/USAID? (Y/N) i. Do you consider the resources and supplies provided by JPV/USAID to be consistent and predictable enough to plan for their efficient use in the future? Why or why not? ii. Which resources were unreliable or scarce and how did your organization confront this challenge? 2. Did your organization receive capacity training from JPV/USAID? (Y/N) i. Was the follow-up to the trainings appropriate and sufficient to implement the capacities learned in the training(s)? ii. In what way(s) has your organization adopted or institutionalized the capacities developed as to not depend on the resources of international cooperation? iii. What challenges does your organization face in accessing resources for capacity training in the absence of JPV/USAID support? II. Roles 1. Could you tell us more about the relationship between the government, civil society and the private sector in your region? Has this changed in light of the JPV/USAID program? If so, in what way? What are the communication channels that you rely on? 2. Have JPV/USAID been effective in identifying relevant actors in the local system? What were some of JPV/USAID’s major successes/failures in identifying/involving these relevant actors? 3. Is there a clear leadership in organizational authority in the local system? Who is in charge of this? Is it adequate? Why or why not? 4. Would you say that JPV/USAID have been capable of empowering relevant actors to take more prominent roles in violence prevention? i. For example, would you agree that selecting _____ as a key actor/ strategic partner was a success? [This question changes for each region] 5. Since the beginning of the JPV/USAID program, which actors have increased their participation in the local violence prevention system the most? What kind of activities are they involved in? i. Does your organization have the power/influence to impact the direction of the local prevention system? ii. Which organizations have the most influence in the local prevention system? Why? 100 III. Relationships 1. Did the priorities of the JPV/USAID program take the needs and aspirations of yours and other local organizations into account? i. How do their priorities change in response to the demands of local actors and how do they adapt to them? 2. Was there adequate space for giving JPV/USAID feedback from your organization regarding your collaboration with them? Did they take that feedback into account? i. How receptive did you perceive the program to be? Were there perceptible differences between different staff positions? 3. Have JPV/USAID been successful in bringing about positive interactions between local actors where before there were none? i. What are JPV/USAID’s most important achievements in this respect? Which were the most important shortcomings? ii. Has your organization extended its network of organizations and contacts as a result of the JPV/USAID program? 4. Do you consider there are instances for accountability within the local prevention system? IV. Rules 1. What kind of contractual relationship did your organization have with JPV/USAID? Was it appropriate? Why or why not? 2. How realistic were the goals your organization committed to in order to receive financial/technical assistance from JPV/USAID? Please elaborate on JPV/USAID’s successes/failures on in developing the contract/program with your organization. 3. In what way did JPV/USAID’s proposed time horizons impact your organization’s work? 4. Was your organization given the flexibility needed to make efficient use of JPV/USAID’s resources? Why or why not? 5. How much risk did USAID assume in deciding to open its investment portfolio to organizations that could act as catalysts for promising operations? i. What was balance between risk/reward in terms of returns? ii. Which criteria were most relevant for JPV/USAID to award/renew its donations to your organization? V. Results 1. Has your organization been strengthened since its cooperation with JPV/USAID? In what way have donations/technical assistance contributed to this end? 2. Do you think that your organization through has more tools and capacity to identify and design effective and promising models informed by evidence as result of collaboration with JPV/USAID? i. How did JPV/USAID contribute to this result? 3. Was JPV/USAID effective in conducting knowledge transfer between actors in different regions? i. Were the results generated by other actors (implementation guides, resources, publications, research, etc.) effectively translated into usable materials for your organization? 4. Has JPV/USAID been effective in transferring effective models between actors? i. In particular, what do you believe accounts the success of models that local actors have explicitly requested? (Ex. CBT o Civic Justice) 5. What innovations generated by other actors (national or international) did your organization have access to? 101 i. What do you consider to be the advantages and disadvantages to focusing on funding institutions that can act as catalysts for contextualized innovations that are scalable in the long-term as opposed to adopting models of famous foreign franchises? 6. In the absence of support from JPV/USAID, what are the principal challenges that the local prevention system faces? Why? i. Is the local prevention system capable of adapting and responding to political changes? Why or why not? 7. In the absence of support from JPV/USAID, what are the principal challenges that your organization faces? Why? 8. From your experience with JPV/USAID, what role do you think international cooperation in crime and violence prevention should occupy? Which activities should it direct its limited resources to? i. What resources should be provided by international cooperation that are necessary for the implementation of best practices in prevention? (Note for interviewer: We are seeking to understanding more about resources that the local system has no substitutes for). 9. How have JPV/USAID contributed to the reduction of crime and violence in the region? How have they achieved results? 102 II. 3 INSTRUMENT FOR SEMI-STRUCTURED INTERVIEWS WITH CSOS PROGRAM COORDINATORS I. Targeting a. What is the target population of your organization? i. Which mechanisms or strategies do you use to reach this population? Which are the reasons for rejecting a case? ii. Which tools do you utilize to determine youth's profiles? What do you measure with these instruments? How did you choose or develop these instruments? (inquire if these are instruments that they built and validated or if they took them from somewhere else) iii. How did JPV/USAID contribute to improving these profiling tools? (YTT, generation of disaggregated crime incidence data by neighborhoods, etc.)? b. Which risk factors you are trying to mitigate with your interventions with youth? i. How effective do you think your interventions are in achieving this goal? Why? ii. How did JPV/USAID help your organization to better understand and identify risk factors and local dynamics of violence? Which resources and/or tools did JPV/USAID provide to your organization to better understand and identify risk factors and local dynamics of violence? c. Which are the protective factors that you seek to strengthen in young people? i. How do you enhance them? ii. How did JPV/USAID help your organization to better understand and identify youth’s protective factors? II. Attention to young people and families a. Which services does your organization offer to young people and their families? Which of them are under your coordination? i. Which changes have you made since the beginning of the collaboration with JPV? ii. In your area, how did you define this offer of services? How do you update it? How often? iii. Which opportunities for improvement do you identify in working with young people in the area you coordinate? iv. How do you work with families? b. How do you determine the activities in which each youth will participate? i. Which are the main difficulties you face in engaging young people with these activities? What about engaging families? ii. Which strategies do you follow when youth lack of commitment? What helps youth engage and stay in the program? c. Which are the tools you offer young people to reduce their recidivism likelihood after they finish the program in your organization? i. How do you determine when a youth is ready to leave the program? ii. How often do you have youth who had already been with you before? III. Implementation capabilities a. With which government institutions does your organization collaborate with? i. How did JPV/USAID facilitate coordination with these institutions? ii. How are the dynamics of work with these institutions? How does this dynamic affect the work with youth? b. Specifically, in the case of educational institutions, which challenges do you identify to ensure that youth continue with their education? 103 i. How do educational modalities and study plans help this youth to keep up with their studies? c. Which companies and private sector organizations are you working with you? i. How do companies or private sector organizations support youth’s reintegration processes? What motivates companies or private sector organizations to work with young people? ii. How effective are linkages with these companies? iii. How often do they offer formal job opportunities to youth? Are offered wages sufficient? d. In which levels of prevention does your organization work? Which are your organization’s capabilities to implement strategies at all prevention levels? i. How has JPV/USAID contributed to strengthen these capacities? ii. Which capacities remain to be generated to implement the strategies more effectively? Which are the biggest challenges, in terms of capabilities, that your organization faces to effectively implement these strategies? (human capital, inter￾institutional financial coordination or technical knowledge)? iii. How did JPV/USAID contribute to meet these challenges e. Which actors have been involved in the design, implementation, and evaluation of your organization's interventions? i. Did JPV/USAID strengthen cooperation with other specialized actors in the design, implementation, and evaluation of interventions? ii. Which relationships are yet to be developed with key stakeholders to more effectively implement prevention strategies? f. Do you consider that your organization has the necessary resources to effectively carry out its work? i. How has JPV/USAID contributed to guaranteeing sustainable access to these resources? ii. Which additional resources would be necessary to carry out the implementation more effectively? IV. Monitoring and evaluation a. How do you follow-up on cases after program conclusion? (Probe on family follow-up) b. Which monitoring tools do you use in your organization to follow-up on interventions’ implementation? i. How have these tools changed and/or been strengthened through cooperation with JPV / USAID? c. How do you evaluate the impact and/or success of interventions in your organization? i. How have measurement/evaluation tools changed and/or been strengthened through cooperation with JPV / USAID? d. Do you consider that due to the work done with JPV/USAID, the perception of the importance of evaluating and informing by evidence the work you do, has changed within your organization? V. Institutionalization a. Is/are the intervention(s) implemented by your organization institutionalized? b. Which factors pose a threat to the sustainability and institutionalization of your interventions? (change of government, lack of financing, etc.)? c. In your organization, have you considered expanding to other contexts or transferring the practice to other actors? d. How has JPV/USAID enhanced the scale potential of your interventions 104 ANNEX III. RESULTS ACHIEVED BY MEASSURMENT INSTRUMENT Table 1. GOPI-P Scores for JPV Supported Municipalities Municipality 1st measurement (2016-17) 2nd measurement (2018) 3rd measurement (2019) % Change Pending improvement areas Chihuahua 37% Willing to Participate 54% Basic 69% Basic 86% - Design a communication strategy for C&V prevention Ciudad Juárez 41% Willing to Participate 59% Basic 77% Acceptable 88% - Institutionalize the multisectoral group - Mesa del Suroriente, in the municipal government Escobedo 48% Willing to Participate 68% Basic 65% Basic 35% - Improve citizen involvement - Strengthen use of C&V prevention tools - Update crime and violence prevention strategy Guadalupe 57% Basic 64% Basic 52% Basic - 9% - Train public servants on strategic planning - Reestablish multisectoral group Monterrey 33% Willing to Participate 69% Basic 84% Acceptable 155% - Establish the Municipal Council for C&V Social Prevention - Trainings Morelia 38% Willing to Participate N/A74 58% Basic 53% N/A Tijuana 51% Basic 65% Basic 65% Basic 27% - Improve the multisectoral group’s structure - Develop specialized studies on C&V Tonalá 53% Basic 60% Basic 52% Basic -2% - Improve citizen involvement - Develop specialized diagnoses - Generate a communication strategy Zapopan 59% Basic 57% Basic 69% Basic 17% - Improve M&E capacity - Build multisectoral alliances - Design V&C prevention public policy 74 The second meassurment of the GOPI-P in Morelia was focused on Civic Justice and not in the overall crime and violence prevention agenda, and therefore it is not considered. 105 Table 2. OPI-P Results for Organizations that Received Strengthening Assistance Plan Municipality CSO Initial OPI Final OPI Change Chihuahua Busuleba A.C. Low Low - Centro de Liderazgo y Desarrollo Humano Basic N/A - Fomentando bienestar y cohesión comunitaria – Programa Espere Low Basic + Red de Cohesión Social 1,2,3 por mí y mi comunidad** High High - Ciudad Juárez Casa Triunfo de México, A.C.* Low Low - Sin Violencia A.C. Low N/A - Techo Comunitario Basic Moderate + Guadalajara (metropolitan region) Aluzarte Basic Basic - Corpocreativo cultura en movimiento A.C. Basic Moderate + México Me Necesita A.c. Moderate N/A - Paz y Reconstrucción del Tejido Social A.C. Basic Moderate + Seguimiento y Detección del Intento Suicida y Abuso Sexual A.C. (SYDISAS) Moderate Moderate - Monterrey (metropolitan region) Alternativas Pacíficas A.C. Moderate Moderate - Despierta, Cuestiona y Actúa A.C. Moderate N/A - Fundación Jomar Low Basic + Mariposas de la Felicidad Basic Basic - Raza Nueva Low Basic - Pastoral Juvenil en Situaciones Críticas PJSC A.C. Low Basic - Morelia Centros de Integración Juvenil* Basic N/A - Fundación Crece Low N/A - Tijuana Casa de las Ideas Basic Moderate + Centro de Arte y Cultura El Globo A.C. Basic Moderate + Fundación Tú + Yo A.C. Basic Moderate75 Moderate + Promotora de las Bellas Artes A.C. Basic Moderate + * Organizations that received funding from JPV. ** Organizations that also have a Best Practice Scale (EBP) and received funding from JPV 75 This CSO has different results reported in DevResults and in its report. The results are Basic in the former and Moderate in the Later. 106 Table 3. Best Practices Score (EBP, for its acronym in Spanish) Municipality Name of intervention Implementing organization Initial EBP Category Final EBP Category Change Received JPV funding Chihuahua Modelo de familias inteligentes CIFAC Not informed by evidence Promising practice + - Mi Colonia es mi Casa Instituto Municipal de Planeación (Comisión CPTED Urbano) Informed by evidence Promising practice + - Comisión de Juventud en Riesgo Comisión de Juventud en Riesgo Informed by evidence Promising practice + - Modelo de intervención comunitaria, 1, 2 y 3 por mí y por mi comunidad Red de Cohesión y Participación Social Informed by evidence Promising practice + Yes Ciudad Juárez Autonomía y libertad interior transformando sujetos violentos CASA Not informed by evidence Informed by evidence + Yes Pandillas: Clubs de Convivencia Armónica Sumando Esfuerzos por Juárez Informed by evidence Promising practice + - Yo Soy Rediseño Social Centro Caritativo para Atención a Enfermos de Sida, A.C. Not informed by evidence Promising practice + Yes Desafío: Modelo de Formación Juvenil con Enfoque en la Reducción de Riesgos Fundación Comunitaria de la Frontera Norte, A.C. Informed by evidence Promising practice + Yes Casa Equidad* Casa Amiga Not informed by evidence Informed by evidence + - Cure Violence / Del Barrio a la Comunidad* Centro Caritativo para Atención a Enfermos de Sida, A.C. Informed by evidence Promising practice + Cuauhtémoc (Mexico Federal District) Atención a adolescentes en conflicto con la ley Reintegra, A.C. Not informed by evidence Promising practice + Yes Jóvenes en conflicto con la ley: acción, formación y ocupación Acciona, A.C. Informed by evidence Promising practice + Yes Escuela de Cultura Urbana La Victoria Emergente Informed by evidence Promising practice + Yes Guadalajara (metropolitan region) Grupos de ayuda mutua Instituto de las Mujeres Guadalajara Informed by evidence Informed by evidence - - Servers Grupo RE Informed by evidence Informed by evidence - - Mindfulness CEDAT Informed by evidence Promising practice + Yes Arte en prisión Aluzarte Informed by evidence Informed by evidence - - Más allá de los miedos Paz y Reconstrucción Informed by evidence Promising practice + - 107 Table 3. Best Practices Score (EBP, for its acronym in Spanish) Municipality Name of intervention Implementing organization Initial EBP Category Final EBP Category Change Received JPV funding Monterrey (metropolitan region) Renace Jóven Renace ABP Not informed by evidence Promising practice + Yes Tijuana Modelo de Abordaje Comunitario e Integral Tú + Yo Fundación Tú + Yo Informed by evidence Promising practice + - Orale ProSalud Informed by evidence Promising practice + Yes Casa de las Ideas Casa de las Ideas Informed by evidence Informed by evidence - - Club de niños y niñas Club de niños y niñas Not informed by evidence Informed by evidence + - Modelo de Desarrollo Comunitario Grupo Ruba Not informed by evidence Informed by evidence + - 108 ANNEX IV. COMPARATIVE MUNICIPALITIES MATRIX Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana Institutionali￾zed multisector collaboration Council for Crime and Violence Social Prevention of the Municipality of Chihuahua (COPREV). Operates with little oversight from JPV. In 2019, Mesa Suroriente was established under the leadership of the Municipal Women’s Institute to work on topics of gender violence. The anchor actor and JPV supported this initiative to bring the multisector perspective. The anchor actor finances many of the participating organizations. In this group, members of the Executive Committee of the Municipal Cabinet of Social Prevention also participate. 76 There is an additional working group for topics of child sexual violence prevention A metropolitan violence prevention committee was promoted in 2019 with support from the anchor actor and in alliance with the Metropolitan Security Agency and the University of Guadalajara (this group was formed for the implementation of civic justice). At the state level, the Jalisco Attorney General promoted the creation of a Youth Reintegration Committee (members of the committee include the government of Jalisco, the municipalities of Zapopan, Guadalajara, and Tonalá, the Office of the Attorney General, and Corporativa de Violence Prevention Multisectoral Network. A varied set of actors participate in this agenda, but coordination and articulation remain a challenge. Escobedo has its own Multisectoral Cabinet for the Prevention of Violence. The local system is organized around the implementation of the civic justice. Although this is not a multisectoral broad committee, we deem this to be sustainable over time. Municipal Committee for Violence Prevention. Operates with little oversight from JPV, but is highly dependent on JPV’s funding. This committee was formed since the PCC program. After the administration change in 2019, giving continuity to the work with authorities was deemed complex. Therefore, members kept meeting under the LPS umbrella (also adding additional relevant actors). In 2020, the 76 This committee existed since the former Program for Community Violence Prevention (PCC for its acronym in Spanish) and responded to the PRONAPRED logic. JPV tried to reactive this cabinet, but priorities were different. Nevertheless, this committee acts as a counterpart for the GOPI-P application. 109 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana with participation of representatives from the local legislature and SIPINNA,77 the anchor actor, and other local organizations. The Social Reintegration Committee is an additional relevant group. This started under the leadership of a court magistrate and in collaboration with local NGOs. One organization that is funded by JPV and the anchor actor leads this group. Given the collaboration, the anchor actor has also started systematizing the work of this initiative. Fundaciones, the local anchor actor). The municipalities of Tonalá and Zapopan also achieved institutionalization of multisector groups. collaboration with the government has been slowly resumed. Shared and agreed agenda and clear working plans The group has been able to establish priorities and goals around a crime-and-violence￾prevention agenda, Only as related to gender strategy. The Mesa Suroriente agenda is currently being developed. The recently established prevention committee has made progress in defining some objectives. It is still unclear whether this have become actionable. While the multisector group was able to attract relevant actors, it could not coordinate and articulate the work under a Only around the implementation of civic justice The group has been able to establish priorities and goals around a crime-and￾violence￾prevention agenda. 77 National System for the Protection of Girls, Boys, and Adolescents (SIPINNA for its acronym in Spanish) 110 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana organized in five strategic areas. COPREV has an internal regulations and legal framework. Zapopan and Tonalá have developed crime and violence agendas within each municipality. common agenda and define priorities. Escobedo’s cabinet meets regularly, but it is unclear whether setting a clear agenda (beyond the implementation of civic justice) has been possible. Mesa Metrópoli is coordinating a metropolitan network to implement civic justice in eight municipalities of the region. Active government involvement in the multisector group Although recently created, the Mesa Suroriente focused on the gender strategy is fully supported by the local government. This group was recently created. It is unclear whether the government participation is sustainable. However, the Metropolitan Security Agency is leading the civic justice implementation, which is promising. In Escobedo the participation is active In Monterrey the participation is mainly at a state level (Nuevo León) through the state attorney. In Guadalupe there is no participation. After the administration change in 2019, government participation was significantly reduced. 111 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana Local system impact in public policy (both through the multisector group or outside of it) COPREV resulted in the alignment between the municipal development plan and the municipal prevention plan, ensuring coherence in public policy regarding crime and violence prevention In 2019, the municipal government approved a civic justice regulation to substitute the municipal edict.78 This substitutes some penalties, from punitive actions— such as fines or detentions—to alternative measures that promote the restauration of community relations. The Municipal DIF79 and the Municipal Women’s Institute adopted the Care Model for Child and Adolescent Victims of Family Violence. With this model, these institutions incorporate an evidence-based strategy that allows them to methodologically serve children, adolescents, and mothers who have been exposed to violence. The government of Jalisco institutionalized the ROLE model for its implementation in private schools. The government of Zapopan, after seeing the benefits from the Zapopan Rifa project, published program operating rules for its institutional permanence. The attorney general adopted the implementation of a CBT-informed intervention to support youth undergoing processes with Jalisco’s adolescent criminal justice system Monterrey’s government partnered with RENACE to provide psychosocial support to youth detained for crimes committed in, or by people living in, Monterrey. The government of Morelia included USAID’s local systems framework in its municipal development plan for 2018–2019 The program supported the development of the Directorate for the Execution of Administrative Sanctions that gives the institution the power to bring together other actors in the LPS. In this capacity, the directorate has enabled the catalog of alternative solutions to the conflict that will form part of the civic justice system. The program supported the development of the The government approved a reform to the municipal edict in matters of family and gender violence.80 The LPS independently proposed and got approved the integration of a crime-and￾violence￾prevention subject in all school levels of the municipality. The Violence Prevention Municipal Committee was selected to administer federal funds to finance projects for at-risk youth. The anchor actor settled an agreement with 78 Bando de Policía y Buen Gobierno in Spanish. 79 National System for the Family Integral Development (DIF for its acronym in Spanish) 80 This regulation now forces violence perpetrators to attend workshops on new masculinities delivered by the Crime Prevention Department of the Municipal Public Security Secretariat in alliance with the Municipal Women’s Institute. 112 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana The government stated in its municipal development plan for 2019–2021 that it would expand the program “My Neighborhood, My House” because of its effectiveness. An adapted YTT was adopted by the criminal justice system for adolescents in the state of Chihuahua. Offender Monitoring System (SIMOPIN). The objective of this system is to generate criminals from statistical data brought to the civic courts to inform decision-making and generate evidence on the effectiveness of the programs that are part of the catalog of solutions in Morelia. UVAQ and Morelia’s municipal security commission formed a partnership that aims to promote the civic justice model and provide alternative measures for conflict resolution. In the framework of such alliance, UVAQ offered to finance 60% of tuition fees for police officers and their families for undergraduate and graduate programs. Additionally, UVAQ offered stress management workshops for police the local government to operate public resources from FORTASEG to implement the project “Youth Building Prevention,” targeting at-risk youth, valued at 4,800,000 MXN (about 251,300 USD). 113 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana officers and civic judges and participated in the provision of an 18- hour-long seminar for Morelia’s Municipal Security Commission and the Municipal Receivership. Private sector participation in the multisector group FICOSEC is also an anchor actor and co-leads the group. FICOSEC is involved in all of the multisector collaborations, and FECHAS is incrementally increasing its participation in the crime-and-violence￾prevention agenda. The private sector participation is active in Monterrey, but not necessarily aligned with the LPS Through the Pro￾Escobedo trust, businessman seek to contribute to the civic justice model by generating employment for offenders in companies that belong to this association. COPARMEX The anchor actor is financed by businessman. Other organizations from the private sector are active in the group (BBVA, Grupo Ruba, etc.) but do not necessarily finance its activities. Anchor actor interested in and capable fn furthering the LPS Seguridad y Justicia, FICOSEC, and FECHAC are all interested in this agenda. It is unclear whether the work will continue the local It is unclear whether the work will continue the local system as defined by JPV. (The municipal government) But faces a financial sustainability risk 114 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana system as defined by JPV, but there are natural leaders for this agenda. Number of at￾risk youth direct beneficiaries of reintegration programs Total: 108 Male: 76 Female: 32 Total: 939 Male: 548 Female: 391 Total: 106 Male: 97 Female: 9 Total: 554 Male: 416 Female: 138 Total: N/A Male: : N/A Female: : N/A Total: 316 Male: 171 Female: 145 Total co￾investments81 11,234,201 MXN (59% was invested in CPTED, 34% in sexual violence, and the rest in at-risk youth and civic justice) 9,451,764 MXN (67% was invested in at-risk youth, 23% in sexual violence, and 11% in civic justice) 32,880,863 MXN (75% was for CPTED, 21% for at-risk youth, and the rest for civic justice and a co￾investment from the anchor actor) 13,136,413 MXN (88% was for civic justice and 12% for at-risk youth. There is also a little co￾investment from the anchor actor that represents less than 1%) 19,731,789 MXN (100% was for civic justice) 6,346,613 MXN (88% for at-risk youth, 6% for gender violence, and the rest co￾investments from the anchor actor and other small donations) Public sector co￾investment82 6,837,193 MXN (93% was invested in CPTED, the rest in civic justice) 1,000,000 MXN83 (100% was invested in civic justice) 31,063,632 MXN (80% of this resource was in the municipality of Zapopan for CPTED, 17% for at-risk youth, and the rest for civic justice) 11,537,790 MXN (100% was for civic Justice) 19,363,222 MXN (100% was for civic justice) 4,891,956 MXN84 (100% for at-risk youth) 81 There is an additional co-investment of 21,519,181 MXN that happened outside the regions, mainly from actors based in Mexico City. 82 There is an additional investment of 18,251,943 MXN coming from FORTASEG made to other regions. 83 FORTASEG resources 84 FORTASEG resources 115 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana Private sector co￾investments85 3,977,908 MXN (96% was invested in sexual violence and the rest in CPTED and family violence) 1,800,000 MXN (100% invested in sexual violence) 26,399 MXN (100% invested in at-risk youth) - 63,568 MXN (100% in civic justice) 981,636 MXN (59% invested in at-risk youth, the rest in gender violence) Civil organization co￾investment86 416,100 MXN (53% invested in youth at risk and the rest in CPTED) 6,651,764 MXN (95% invested in at-risk youth, the rest in sexual violence) 1,791,162 MXN (88% invested in at-risk youth, the rest is a co￾investment from the anchor actor) 1,598,624 MXN (97% was for at-risk youth, the rest a co￾investment from the anchor actor) - 473,020 MXN (70% is a co￾investment from the anchor actor, the rest was invested in at-risk youth) Academia co￾investments 3,000 (invested in CBT) - - - 305,000 MXN (100% in civic justice) - JPV donations 3,132,751.38 MXN (49% invested in territorial strategies and 51% in focalized strategies, specifically CBT￾informed interventions, as part of the civic justice model) 6,451,630.16 MXN (42% invested in focalized strategies for youth in conflict with the law, and 58% in territorial strategies) 5,864,589.04 MXN (84% invested in focalized strategies, in particular CBT for youth in conflict with the law and adapted to be implemented in schools, and the rest in territorial strategies)87 5,716,784.04 MXN (65% invested in focalized strategies— 35% for the adaptation of a CBT￾informed intervention for civic justice, 17% for a CBT-informed intervention in schools, and 17% for an intervention for youth in conflict with the law. The rest was invested in territorial 1,630,558.45 MXN (43% invested in focalized strategies for civic justice, in particular a CBT intervention and an addiction program. The rest was invested in a territorial strategy) 4,999,580.74 MXN (62% invested in focalized strategies for youth in conflict with the law, 20% for a focalized strategy of a CBT intervention in schools, and 18% in a territorial strategy) 85 Additional 1,690,866 MXN was leveraged in Mexico City 86 Additional 1,576,372MXN was leveraged in Mexico City 87 This number considers the subcontract with ProSociedad that was aimed at developing a CBT intervention in schools in collaboration with the government. 116 Table 1. Comparative Municipalities Matrix Feature / region Chihuahua Ciudad Juárez Guadalajara (metropolitan region) Monterrey (metropolitan region) Morelia Tijuana strategies related to new masculinities) Local trust A local trust was established in 2010 for crime and violence prevention—FECHAC and FICOSEC. In 2019 a fund was inaugurated by the anchor actor to finance crime-and-violence￾prevention initiatives in the region—with a communication strategy to attract more funds. The United Businesspeople, a nonprofit in Escobedo, was inaugurated in 2019, but the financial sustainability plan is still developing. Trust was established in 2019 but given the COVID-19 crisis funds are unavailable (the payroll tax—the source of funding— was suspended). Efforts to build a local trust haven’t been successful yet. 117 ANNEX V. DONATIONS PROGRAM Table 1. Donations Program Grants and Products RFP Technical agenda Grants (region) Systematization/ Publication 001 2016-17 Evidence-bases models, including: CBT, Parental skills development, alcohol and drug abuse, mentorship, restorative justice, social cohesion, new masculinities, social reinsertion, focused deterrence, procedural justice, gender￾based violence, environmental prevention policies. This call invited programs with implementation timelines of maximum one year, with the opportunity to renew for two additional phases for a maximum, each of one year (2017 and 2018) Total granted amount: MXN 11,310,226 CEDAT (Guadalajara) Strengthen the intervention model based on mindfulness and CBT components aimed at reducing the incidence of juvenile delinquency. * This grant had a follow-up award to replicate and adapt the model for the adoption by other organizations, including a treatment option for the Judicial System in the State of Jalisco, with a total amount grated MXN 524,5050 (not included in the total amount in the right column). Model systematization Implementation manual Impact evaluation ITESM (Morelia) Feasibility study of evidence-based programs/ study on the ecosystem development based on results and implementation opportunities. Diagnosis publication TELAR DEL SARAPE (Guadalajara) Strengthening of the social transformation model, based on creativity and neuro￾education as tools for social transformation Model systematization ACCIONA (Morelos) Strengthening of the violence prevention model based on mentorship provision aimed at finding alternative activities options according to youth’s needs and capabilities. Model systematization GENTE DIVERSA (Baja California – including Mexicali, Rosarito, Tecate, Tijuana, and Ensenada) Develop guides for the application of the Safe Cites Model aimed at preventing and eradicating violence against women in urban areas with high crime incidence Results evaluation ODP (Escobedo) Develop a curriculum and Implement CBT interventions for young people in situations of vulnerability of violence. Develop Model systematization Implementation manual IPAP (Morelia, Tijuana, Mexico City, Chihuahua, Guadalajara, Monterrey, Ciudad Juárez) Implement the project “Best local prevention practices for gender based violence” to identify practices implemented at the local level. Model systematization (not public) 118 Table 1. Donations Program Grants and Products RFP Technical agenda Grants (region) Systematization/ Publication SIMO (Chihuahua) Support the Unit of Criminal Analysis develop their data management system to be used as an input for prevention policies. - SIMO (Tijuana) Design, implement, and evaluate a CBT intervention targeted at youth in urban schools to prevent violence. Model systematization (not public) Results evaluation (not public) REINTEGRA (Mexico City) Strengthen the reintegration model for youth in conflict with the law under non-custodial measures Model systematization Implementation manual RENACE (Monterrey) Strengthen the model aimed at social reintegration of youth un conflict with the law through counseling and services, such as accompaniment with youth and their families during their trial and sentencing, as well subsequent follow-up activities. Model systematization (identified as good practice) RED DE COHESION (Chihuahua) Strengthen the interventional model aimed at addressing problems of violence through the rescue and recovery pf public spaces, by introducing the YTT and activities to work with youth displaying high-risk behaviors- - 002 2017 Models focused on social reinsertion through programs focused on generating employment opportunities for youth in secondary or tertiary prevention and in conflict with the law. Total granted amount: MXN 5,843,412 VICTORIA EMERGENTE (Mexico City) To promote the economic and social inclusion of youth who are part of cultural groups by implementing an alternative pedagogical model that strengthen s the technical and psychological capacities with a focus on violence prevention through socio￾entrepreneurship productive projects. Model systematization (not public) CASA (Ciudad Juárez) Implement a model aimed at interrupt cycles linked to violence and acts of crime committed by at-risk youth, by developing resilient identities, options for educational and job reinsertions, and prevention capacities in communities. Model systematization (not public) SUPERA (Monterrey) Results evaluation 119 Table 1. Donations Program Grants and Products RFP Technical agenda Grants (region) Systematization/ Publication Implement the New Masculinity “Youth Transforming Youth” program aimed at reducing gender based violence. (not public) 004 2017 Models focused on social reinsertion through programs focused on generating employment opportunities for youth in secondary or tertiary prevention and in conflict with the law, or in advanced processes of addiction treatment. Total granted amount: MXN 5,685,349 FUNDACION COMUNITARIA DE LA FRONTERA NORTE (Ciudad Juárez) Implement the model “Desafío” aimed at engaging at-risk youths in productive and participative activities while strengthening protective factors against violence and delinquency, though psychological support, the development of interpersonal skills through sports, technical in-job training. Model systematization Results evaluation (not public) TENDA DI CRISTO (Ciudad Juárez) Implement the model “Yo Soy Rediseño Social” aimed at reducing recidivism in youth in conflict with the law. Based on multi-actor collaboration processes seeking to increase social inclusion (socio-educational and employment). Results evaluation (not public) PROSALUD (Monterrey) Implement the model “Orale” which is an adoption of the International Youth Foundation’s Youth Employment Model, aimed at increasing employment and educational opportunities for at-risk youth, teaching soft skills. This model also added CBT components to enhance impact. Results evaluation (not public) Total granted amount: MXN 2,273,109 FUNDACION EDUCACION SUPERIOR EMPRESA (Chiapas and Tijuana) Implement care models aimed at the social reintegration of youth in conflict with the law and or at-risk to create opportunities for labor inclusion. - 006 2019 Development and implementation of a CBT model for youth between 18 and 29 years old that are derived from the Civic Justice System in the cities of Chihuahua and Escobedo (one on one), with the objective of reducing their impulsivity and improve self-control. Total granted amount: MXN 3,298,431 LIBRES POR AMOR (Chihuahua) Develop or adapt a model with CBT components to assist youth from 18 to 29 years old derived from the Municipal Civic Court in Chihuahua. In progress SUPERA (Monterrey) Develop or adapt a model with CBT components to assist youth from 18 to 29 years old derived from the Municipal Civic Court in the Metropolitan Area of Monterrey. In progress 007 2019-20 1) Model of addiction treatment 2) CBT model To work with youth between 18 and 29 years derived from the Civic Justice System in the City of Morelia Universidad Vasco de Quiroga - UVAQ (Morelia) Develop or adapt a model with CBT components to assist youth from 18 to 40 years old derived from the Municipal Civic Court in Morelia. In progress 120 Table 1. Donations Program Grants and Products RFP Technical agenda Grants (region) Systematization/ Publication Total granted amount: MXN 1,818,313 CIJ (Morelia) Develop or adapt a model with addiction treatment to assist youth from 18 to 40 years old derived from the Municipal Civic Court in Morelia. In progress 008 2019 Reintegration model, at the community level, of men and women at risk to generate violent behaviors, in conflict with the law, or substance consumption, in Tijuana Total granted amount: MXN 899,560 PROSALUD (Tijuana) Implement a model for the reintegration of young people, though the development of a diagnosis of main risk factors, development of a canalization protocol, implement an incorporation program, and generation of strategic alliances to channel youth for other services. Model systematization 009 2019 Care model to accompany the process of comprehensive social reintegration of young people in conflict with the law on non-custodial sanctions in two municipalities of the Guadalajara Metropolitan Area, through the accompaniment one on one mentoring to develop their educational, work / production environment, personal and family. Total granted amount: MXN 1,690,945 CEDAT (Guadalajara) Implement a program of accompanying young people in conflict with the law on non￾custodial sanctions. See above on CEDAT’s row 010 2019-20 To give continuity to the care model aimed at the social reintegration of young people in conflict with the law and / or at risk through programs to generate opportunities for job inclusion (Hope Truck) – in Tijuana. Total granted amount: MXN 1,328,407 TIJUANA INNOVADORA (Tijuana) Give continuity to the care model aimed at the social reintegration of young people in conflict with the law and / or at risk through programs to generate opportunities for job inclusion (Hope Truck) – in Tijuana. In progress 011 2020 For the elaboration of an Intervention Manual of the Model of the Supervision System of Adolescents in non-custodial measures the State of Chihuahua, as well as the Manual for the use of YTT tool as part of this model. Total granted amount: MXN 800,323 TENDA DI CRISTO (Ciudad Juárez) A follow up grant to “Yo Soy Rediseño Social” to meet the demand for channeling youth with non-custodial measures. In progress 121 ANNEX VI. JPV ORGANIZED EVENTS AND STUDY TOURS Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 At-risk youth Events Mexico City: In collaboration with J￾PAL, JPV organized the workshop “The Use of Evidence in Preventing Youth Violence: Challenges and Opportunities in Latin America" in Mexico City. More than 100 attendees shared experiences and lessons learned from implementing CBT interventions in Mexico, El Salvador, Honduras, the United States, and Liberia (2017). Mexico City: In collaboration with J￾PAL, JPV organized the seminar “Prevention of Youth Violence: Challenges and Opportunities in Mexico and Latin America” in Mexico City to present the results of What Works to Prevent Violence Among Youth? Speakers included professor Thomas Abt from Harvard University, Beatriz Magaloni from Stanford University, and staff from the University of Chicago’s crime lab. Ciudad Juárez: FICOSEC Juárez organized a seminar on program evaluation for interventions focused on violence prevention during the evaluation week for Latin America and the Caribbean. Relevant actors of other regions such as CEDAT, RENACE, and COLEF88 participated as speakers. Tijuana: JPV and Tijuana Innovadora organized the first International Summit on Citizen Security to discuss promising crime-and-violence-reduction policies such as CPTED, focused deterrence, 88 The College of the Northern Border (COLEF for its acronym in Spanish) 122 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 and civic justice. The summit gathered more than 150 participants. Ciudad Juárez and Guadalajara: FICOSEC traveled to Guadalajara to learn about CBT interventions implemented by CEDAT and ProSociedad. This study tour led to the future transfer of CEDAT’s model. Guadalajara and Ciudad Juárez: A group of actors from Guadalajara traveled to Ciudad Juárez to meet and learn from the Ciudad Juárez Social Reintegration Committee; this study led to the creation of a Social Reintegration Committee for Youth within the local system in 2019. Civic justice (or related, including police reform) Events Guadalajara: JPV, the Metropolitan Security Agency, and the University of Guadalajara organized the Civic Justice Homologated Model, Good Governance and Culture of Legality of the Guadalajara Metropolitan Area Forum. Speakers included municipal authorities from Escobedo and Morelia to share their experiences in the implementation of the civic justice model; around 100 people attended. Morelia: JPV and the municipal government of Morelia organized the first International Security and Civic Justice Forum. More than 300 participants attended the event to share results and challenges faced implementing the civic justice model. Monterrey: JPV, Mesa Metrópoli and the municipal government of Escobedo organized the International Forum for Violence Prevention and Citizen Security in Escobedo. Speakers included Julius Lang, Senior Advisor for Training and Technical Assistance at the Center for Court Innovation; David M. Kennedy, Director of the National Network for Safe Communities at John Jay College of Criminal Justice in New York; and government authorities. Around 400 participants attended the event to discuss evidence-based violence prevention strategies Guadalajara: JPV, the municipal government of Morelia, and the municipal government of Tonalá organized the International Forum of Problem-Oriented Policing and Civic Justice (2020). During the event, representatives of local governments exchanged experiences to strengthen the implementation of the Civic Justice Homologated Model. Monterrey: JPV and Mesa Metrópoli organized the forum 123 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 in Mexico and the world, such as focused deterrence, civic justice, and proximity police. Promising Practices in Security and Justice: From a Local Perspective. Speakers included representatives of local governments such as Nuevo León, San Pedro, and Allende, among others. Study tours Ciudad Juárez and Morelia: A delegation of public servants from the Municipal Government of Juárez traveled to Morelia to participate in the Civic Justice and Security Forum. They took advantage of the opportunity to learn more about the implementation of the Civic Justice Homologated Model. Monterrey: Representatives of the Municipal Government of Escobedo traveled to New York to learn from the Courts Innovation Center about the different models that exist on community justice courts to adopt practices in the Mexican context. Gender strategy Events Chihuahua: The Social Prevention of Violence and Crime Council of Chihuahua (COPREV being its Spanish acronym), with support from FICOSEC and USAID organized the Forum on Best 124 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 Practices and the Prevention of Domestic Violence. Ciudad Juárez: JPV, the Municipal Women’s Institute, and the Autonomous University of Ciudad Juárez organized the Forum on Strategies to Reduce Gender￾Based Violence to share experiences and challenges with local actors. Study tours Ciudad Juárez: Representatives of Promotora Social México, the Corporate Social Responsibility branch of Gentera Bank, went on a visit to Ciudad Juárez to learn about the work of FICOSEC, Casa Amiga, La Tenda de Cristo, and Fundación Paso del Norte para la Salud y Bienestar. After this visit, Promotora Social México agreed to finance two of these organizations in their efforts to eradicate sexual violence among children. Fundación Paso del Norte para la Salud y Bienestar committed to invest an additional 50% of the funding provided by the bank. Monterrey: During the International Forum for Violence Prevention and Citizen Security in Escobedo, a study tour was conducted to the local security center (C4) and to a care center that treats women who are victims of violence. Environmental design Events Guadalajara: JPV, the Metropolitan Security Agency, and the University of Guadalajara organized the Urban Design and Violence Prevention Forum: Experiences in Mexico. Speakers included representatives from interventions in Chihuahua, Juárez, and 125 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 Tijuana. More than120 participants attended the event. Institutional￾strengthening strategy Events Mexico City: JPV and the Monterrey Institute of Technology organized an event in Mexico City to launch the virtual learning community. Mexico City: JPV organized a meeting in Mexico City for representatives from local organizations based in Morelia, Ciudad Juárez, Jalisco, Monterrey, Chihuahua, and Tijuana to exchange best practices on violence prevention. Study tours Ciudad Juárez: Juárez Mágico was a study trip organized by JPV and FICOSEC that brought businessmen and representatives of CSOs and business chambers from Tijuana, Nuevo León, Irapuato, and Morelia to Juárez to learn about FICOSEC’s work. It sought to increase collaborations in Morelia and Chihuahua: Members of Morelia’s municipal security commission traveled to Chihuahua to learn from FICOSEC’s crime observatory to foster knowledge generation on public security and civic justice. Morelia and Chihuahua: A group of entrepreneurs traveled to Chihuahua to visit FICOSEC and FECHAC to learn from its model to fund prevention strategies and replicate it. 126 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 crime and violence prevention, strengthen LPS capacities, and enhance private sector participation in this agenda. Participants then visited the project Juárez Mágico, an environmental design prevention project. Several other exchanges followed this tour: - Toks89 visited FECHAC to share work strategies - Pro-Irapuato visited Tijuana Innovadora to learn from its organizational structure. - CINLAAC90 funded by FEMSA,91 visited FICOSEC in Chihuahua to learn from its model. 89 Restaurant chain 90 Civic Council of the Institutions of Nuevo León (CINLAAC for its acronym in Spanish) 91 Fomento Económico Mexicano (a Mexican multinational beverage and retail company headquarteed in Monterrey, Mexico) 127 Table 6. Events and Study Tours, by Transversal Strategy Strategy Type of activity 2017 2018 2019 2020 International sharing of experiences Study tours Monterrey: Public servants from Escobedo’s municipal government presented their experience implementing the Civic Justice Homologated Model in the Public Safety and Community Courts Conference organized by the Courts Innovation Center in Alabama. Morelia: Representatives of the federal government of Guatemala traveled to Morelia to identify and learn from promising interventions with the intention to replicate them in the Guatemalan context, one of them being civic justice. Monterrey: Representatives of the federal government of Jamaica and USAID’s staff in Jamaica visited Guadalupe to learn from Supera’s Youth Transforming Youth model. 128 ANNEX VII. JPV’S RESOURCES DEVOTED TO TRANSFER AND SCALE-UP AND RESULTS Table 1. JPV’s Resources Devoted To Transfer and Scale-Up and Results Best practices in violence and crime prevention Knowledge development Implementation of Solutions & Improve the practice (policy influence) Improve the practice (Publications to support adaptation/transfer) Solutions for at￾risk youth Main result: JPV invested significant effort in developing, systematizing, scaling, and transferring CBT-informed interventions. CBT-informed interventions are now being implemented by several organizations across the regions. JPV financed five different CBT interventions (SIMO, 92 ODP93, Reintegra, CEDAT, and ProSociedad). Two received special assistance for the scale-up and adaptation to different settings: CEDAT’s Mente Libre and ProSociedad’s ROLE. - CEDAT developed the Mente Libre model based on CBT and mindfulness components. It is focused on reducing young felons’ propensity to committing crimes. - ProSociedad developed the ROLE model, focused on reducing young men’s exposure to violence in urban contexts through tools that change cognitive and behavioral processes, along with mechanisms that generate more functional, prosocial, and healthy lifestyles. JPV established a collaboration with J-PAL to facilitate knowledge exchange with international organizations implementing CBT and academics to learn from existing experiences and evidence. Through this collaboration, local CSOs also received CEDAT transferred its Mente Libre model to Jalisco’s internment centers and is currently working to transfer the model to organizations in Chihuahua and Ciudad Juárez, and 10 additional municipalities in Jalisco. - In particular, Libres por Amor with technical assistance from CEDAT developed an intervention model with CBT and mindfulness components to assist at-risk youth derived from the municipal civic court in Chihuahua. - CEDAT and FICOSEC launched a call for proposals invited organizations to participate in a program aimed at transferring the model to interested organizations. The programs include in-person training and online accompaniment. Mente libre Implementation materials: Instructor's Manual: Mente Libre Protocol Youth and violence in Mexico: Dialogue between neurosciences, civil society, and social sciences Evaluation: Has an internal experimental evaluation (caveat: sample size is very small). ROLE: Iimplementation materials ROLE Implementation Manual M&E ROLE Manual Evaluation: Has an external process and results (quasiexperimental) evaluation of the model implemented in schools, but 92 Intelligence Systems in Markets and Opinion (SIMO being its acronym in Spanish) 93 Observatory of Regional Development and Social Promotion (ODP being its acronym in Spanish) 129 technical assistance for their program design. CEDAT and ProDociedad received further support to systematize and rigorously evaluate their models and develop manuals for their adoption by other organizations. ProDociedad’s model was evaluated by an external evaluator, funded by JPV. Once the models were operating, JPV organized events and study tours to facilitate knowledge exchange between regions. ProSociedad’s ROLE model was transferred to high schools in Jalisco. The government mandated a statewide institutionalization of the model in coordination with the Ministry of Education, and ProSociedad supported the process and results evaluation of the adaptation process. - In particular, ProDociedad transferred its model to Supera in Monterrey to assist youth at risk derived from the municipal civic court in Chihuahua. - ProSociedad is also collaborating with UVAQ in Morelia to transfer the ROLE model to its CAS in Morelia, with the objective of strengthening the civic justice system in the city by increasing its capacity to receive and treat offenders. not for the models being replicated for youth in conflict with the law. Civic justice Main result: JPV promoted the Civic Justice Homologated Model based on the innovative approach of public officials in Morelia to deescalate violence and conflict and The Civic Justice Homologated Model was promoted by JPV based on the innovative approach of public officials in Morelia to deescalate violence and conflict and offer an alternative for everyday conflict resolution. In addition, JPV incorporated the PoP approach to strengthen the role of the police within the framework of civic justice. JPV worked with the Mexican government, through the National Commission of Public Security, to design the Civic Justice Homologated Model, which was approved in 2017. From this Because of high demand, JPV started developing civic justice model implementation manuals for judges, mediators, public security officers and other local judicial workers. In addition to supporting the materialization of the idea into a replicable and standardized intervention model, JPV worked jointly with the federal government to lead working groups with 27 municipalities. Furthermore, JPV organized seminars and exchange opportunities with key international actors in community Implementation materials: Guide I. Diagnosis Guide II. Regulatory adequacy Guide III. Public hearings Guide IV. Convivencia Cotidiana Model: Police Oriented to Problem Solving for Civic Justice Implementation Guide: Problem￾Oriented Police for Civic Justice: 130 offer an alternative for everyday conflict resolution. JPV worked to developed and systematize the system, also incorporating the PoP approach to strengthen the role of the police within the framework. The Civic Justice Homologated Model is now being scaled up to more than 15 municipalities in the country with additional demand for replication in others. initiative, the Mexican government determined that federal funds will be allocated to the FORTASEG to implement the Civic Justice Homologated Model. In addition to the work at the national level, JPV worked with and monitored the progress of, the municipal governments of Morelia and Escobedo to implement pilots of this model. JPV’s technical assistance, which was based on the recruitment of specialized consultancies, culminated in the development of an institutionalized mechanism, which then provided the knowledge basis to transfer the model to other areas. justice, developed exhaustive guides that detail the operational requirements to implement the model, and developed focalization tools to assess the risk level of offenders. Besides Morelia and Escobedo, JPV provided technical assistance and recruited specialized consultant services to provide support to Chihuahua, Ciudad Juárez, the Guadalajara metropolitan area, the Monterrey metropolitan area, and Los Cabos to design, implement, and systematize the intervention model. In 2018, after the change in the federal administration, the Mexican government included the Civic Justice Homologated Model and CBT in its list of priorities for the allocation of funds of FORTASEG and the Public Security Contribution Fund (FASP94 being its acronym in Spanish). In July 2019, the Mexican government launched the new National Police and Civic Justice Model, which incorporates in its entirety the Civic Justice Homologated Model and the PoP model supported by JPV. By 2020, 36 municipalities had allocated FORTASEG resources to the Civic Justice Homologated Model. Adjustments to the Regulatory Framework Module 1. Participant's Guide: Police Oriented to Conflict Resolution in Communities Module 1. Instructor's Guide: Police Oriented to Conflict Resolution in Communities Module 2. Participant's Guide: Police Oriented to Problems Solution with Mediation Capacity Module 2. Instructor Guide: Participant's Guide: Police Oriented to Problems Solution with Mediation Capacity Module 3. Participant's Guide: PoP model tools 94 Fondo de Aportaciones para la Seguridad Pública in Spanish 131 In late 2019 and 2020, the Mexican government and JPV have agreed to provide technical assistance to ten more municipalities on the Civic Justice Homologated Model: Acapulco, Cancun, Ciudad Juárez, Hermosillo, La Paz, Los Cabos, Manzanillo, Playa del Carmen, Puerto Vallarta, and Tijuana. Environmental prevention strategy JPV provided technical assistance and support to CSOs and municipal authorities to implement interventions under the CPTED methodology in Chihuahua, Guadalupe, Mexicali, Tijuana, and Zapopan with a focus on at-risk youth and GBV. JPV’s technical assistance consisted primarily of grants to implementing organizations, the systematization of promising interventions following the CPTED methodology, and the training and CPTED certification by specialized service providers of local actors of Tijuana, Ciudad Juárez, Chihuahua, Monterrey, and Guadalajara. This certification will enable participants to apply the CPTED approach in their LPSs to implement sustainable crime-and-violence-prevention strategies. As a result of the evaluation of the CPTED model My neighborhood, My House, Chihuahua’s municipal government decided to expand and replicate the model in four new neighborhoods. In order to replicate the model, public officials from Chihuahua participated in a series of practical and theoretical trainings on the CPTED model by JPV. The replication of this intervention model has resulted in each neighborhood applying a participatory diagnostic, establishing a baseline, developing an annual work plan with indicators and goals, and implementing actions to enhance trust with the local stakeholders. In Tijuana, BBVA-Bancomer, in collaboration with Tijuana Innovadora (the anchor actor) developed the model Mejora Mutualista, which is informed in CPTED principles. Also, Gente Diversa developed the model Secure Cities for Women for Implementation materials: Implementation guide of the CPTED strategy from a prevention perspective for GBV, at-risk youth, and the community asset construction model Implementation manual: Mejora Mutualista—BBVA Situational prevention: Safe Corridor for Ciudad Juárez women Evaluation: External process/ pre- postevaluation (in collaboration with FICOSEC, the anchor actor) 132 several cities in Baja California, which is informed by CPTED principles. Also , Fundación Tú+Yo has developed the model Abordaje Comunitario, informed by CPTED principles. It is implemented in Tijuana, Mexicali, Chihuahua, and Ciudad Ju árez In Zapopan , a public -private partnership emerged between the Zapopan’s Public Space Authori ty and CEMEX—a multinational company that specializes in the production and sale of concrete and similar materials —to implement the workshop “I build ,” which develops construction skills among vulnerable populations of the Miramar neighborhood for the urba n design of safer communities. Given the success of the project, CEMEX financed a second implementation, valued at 40,000.00 MXN (about 2 ,000 USD). Gender strategy Main result: JPV developed the Care Model for Child and Adolescent Victims of Family Violence Through the recruitment of expert consultants and in collaboration with women’s justice centers, JPV supported the piloti ng and systematization of the care model in Ciudad Juárez and Pachuca, which includes CBT components, group therapy methods, standardized tools for monitoring service users , and program evaluation methods. As a result of the piloting of the model, the care model was transferred to institutions in three additional local systems: Ciudad Ju árez, Monterrey, and Gu adalajara. JPV organized trainings for psychologists and public officials with local allies that can further increase the reach and impact of the strategy. A total of 16 women’s center s were trained on the model. 133 The program also launched Chihuahua’s Sexual Violence Prevention Strategy, which got the commitment of the state of Chihuahua. In alliance with the National Pedagogical University of the State of Chihuahua, JPV developed the Diploma on Comprehensive Care for Sexual Abuse during Childhood and Adolescence and provided specialized training to 200 public servants. It leveraged 6,000,000 MXN from local investment to implement related strategies. 134 ANNEX VIII. TARGETING ANALYSIS 1. HOMICIDE TOTALS BY SOURCE Table 1: Comparison of Intentional Homicide Data from INEGI, SESNSP, and Local Police Departments Year / City Chihuahua Escobedo Guadalupe I P S I P S I P S 2015 253 188 142 21 29 24 24 32 27 2016 409 300 275 22 30 23 45 73 55 2017 535 389 363 59 78 62 52 71 62 2018 479 313 282 42 56 53 66 78 77 2019 160 332 44 62 78 83 1676 1350 1394 144 237 224 187 332 304 Year / City Monterrey Ciudad Juárez Tijuana 2015 181 161 130 440 383 269 644 612 612 2016 246 1014 209 680 549 470 857 871 871 2017 193 156 145 805 737 642 1669 1618 1618 2018 323 283 236 1441 1139 1032 2302 2252 2253 2019 178 239 644 1293 2012 2013 943 1792 959 3366 3452 3706 5472 7365 7367 Source KEY: I = Mexico’s National Institute for Statistics and Geography (INEGI, for its acronym in Spanish), S= Mexico’s Secretary General of National Public Security (SESNSP, for its acronym in Spanish), INEGI, P = Local Police. 1I. COVERAGE BY MUNICIPALITY II.1 CHIHUAHUA Table 2: Most Common Neighborhoods Registered in Chihuahua 11 de febrero 18 Riveras de Sacramento 16 Villa Juárez 15 Punta Oriente 14 Chihuahua 2000 13 Popular 4 135 Table 3: Intentional Homicide, Top 10 Neighborhoods in Chihuahua Neighborhood / Year 2014 2015 2016 2017 2018 2019 Grand Total Centro 10 13 9 16 19 67 Sin Datos Específicos 3 1 8 10 20 23 65 Vía Pública 1 11 21 33 Obrera 5 6 11 6 2 1 31 Granjas del Valle1 8 5 8 7 3 31 Revolución 6 3 6 6 2 3 26 Cerro de la Cruz 3 3 8 5 3 1 23 Villa Juárez 2 4 3 7 2 2 20 Aeropuerto 3 2 1 6 2 5 19 20 Aniversario 1 1 4 2 6 4 18 Total Homicides (all neighborhoods) 209 188 399 389 313 160 1,350 II.2 CIUDAD JUÁREZ Table 4. Most Common Beneficiaries’ Neighborhoods Registered in Ciudad Juárez Díaz Ordaz 117 Adolfo López Mateos 81 La Reforma 67 Cersai 3 56 16 de Septiembre 52 Riveras del Bravo 22 Guadalajara Izquierda 22 Plutarco Elías Calles Alta 17 Table 5. Intentional Homicides in Ciudad Juárez by Neighborhood Neighborhood / Year 2014 2015 2016 2017 2018 2019 Grand Total Margaritas 16 6 7 27 39 9 104 Centro 15 2 4 7 22 15 65 Salvarcar 8 6 8 9 16 4 51 Riveras De l Bravo 9 1 4 6 22 9 51 Infonavit Casas Grandes 5 7 4 8 18 5 47 Paseo De Las Torres 2 1 4 10 15 7 39 Chaveña 6 1 3 3 14 11 38 Zona Pronaf 18 2 2 3 10 2 37 La Cuesta 4 3 7 1 16 6 37 Aztecas 6 2 10 11 6 35 Total Homicides (all neighborhoods) 520 383 549 747 1,139 644 3,452 136 II.3 ESCOBEDO Table 6: Most Common Neighborhoods Registered in General Escobedo N/A 62 Alianza Real 17 Table 7: Intentional Homicides in General Escobedo by Neighborhood Neighborhood / Year 2015 2016 2017 2018 2019 Grand Total Villas de San Francisco 3 2 7 6 1 19 Andrés Caballero 1 2 7 4 14 Carretera 1 9 1 2 13 Alianza Real 6 2 2 10 Nueva Esperanza 1 2 2 2 7 Praderas de San Francisco 1 3 3 7 Nuevo Escobedo 2 4 6 La Unidad 2 1 2 1 6 Paso Cucharas Progreso 1 1 2 4 Total Homicides (all neighborhoods) 29 30 78 56 44 237 II.4 GUADALUPE Table 8: Most Common Neighborhoods Registered in Guadalupe Valle del Sol 119 Independientes 55 (blank) 32 Paseo de Guadalupe 31 Los Independientes 22 Santa Cruz 14 Jardines de Xochomilco 14 Table 9: Intentional Homicides in Guadalupe by Neighborhood Neighborhood / Year 2015 2016 2017 2018 2019 Grand Total Valle Soledo 1 5 3 5 14 Nuevo San Rafael FOM 7 1 1 6 8 Cañada Blanca 2 3 1 6 Centro de Guadalupe 2 2 2 6 Nuevo México 1 4 5 2 de Mayo FOM 20 3 2 5 Paseo de Guadalupe 1 1 2 1 5 Valle del Sol 1 2 1 1 5 Ciudad CROC 1 3 1 5 Total Homicides (all neighborhoods) 32 73 71 78 78 332 Santa Cruz ranks #18 on the list, not shown here.* 137 II.5 TIJUANA Table 10. Most Common Beneficiaries’ Neighborhoods Registered in Tijuana N/A 184 La Mesa 23 Los Álamos 23 Col. Luis Echeverria 22 Cañadas del Florido 19 Ejido Ojo de Agua 17 Camino Verde 14 Santa Fe 14 Table 11. Intentional Homicides in Tijuana by Neighborhood Neighborhood / Year 2014 2015 2016 2017 2018 2019 Grand Total Camino Verde 12 11 27 75 65 42 232 Zona Norte 7 26 43 49 43 41 209 Zona Centro 10 19 11 31 33 31 135 Sanchez Taboada 10 14 11 17 22 42 116 Tres De Octubre 3 4 6 12 38 36 99 Nueva Tijuana 5 7 7 19 29 18 85 Ejido Francisco Villa 7 3 8 16 25 21 80 El Florido Primera Y Segunda Seccion 1 3 5 25 29 15 78 Mariano Matamoros Centro 6 7 10 14 21 15 73 El Pipila 2 4 4 12 25 25 72 Altiplano 5 3 5 18 15 10 56 Total Homicides (all neighborhoods) 462 612 871 1,618 2,252 2,012 7,365 138 ANNEX IX. PRELIMINARY STATISTICAL ANALYSIS Figure 1. Homicide Rates by Municipality per 100,000 residents 2010-2018 (INEGI) Figure 2. Perception of Population 18 Years and Older that Feels Insecure (ENSU) 0 50 100 150 200 250 300 2010 2011 2012 2013 2014 2015 2016 2017 2018 Chihuahua Escobedo Guadalajara Guadalupe Juárez Monterrey Morelia Tijuana Tonalá Zapopan Nacional 40 50 60 70 80 90 100 2016m3 2016m6 2016m9 2016m12 2017m3 2017m6 2017m9 2017m12 2018m3 2018m6 2018m9 2018m12 2019m3 2019m6 2019m9 2019m12 2020m3 Tijuana Chihuahua Juarez Tonala Zapopan Morelia Escobedo Guadalupe Monterrey Mexico 139 Figure 3. Percentage of Population 18 Years and Older that Has Witnessed Alcohol Consumption in Public Spaces (ENSU) Figure 4. Percentage of Population 18 Years and Older that Has Witnessed Drug Sells or Consumption (ENSU) 30 35 40 45 50 55 60 65 70 75 80 2016m3 2016m6 2016m9 2016m12 2017m3 2017m6 2017m9 2017m12 2018m3 2018m6 2018m9 2018m12 2019m3 2019m6 2019m9 2019m12 2020m3 Tijuana Chihuahua Juarez Tonala Zapopan Morelia Escobedo Guadalupe Monterrey Mexico 0 10 20 30 40 50 60 70 80 2016m3 2016m6 2016m9 2016m12 2017m3 2017m6 2017m9 2017m12 2018m3 2018m6 2018m9 2018m12 2019m3 2019m6 2019m9 2019m12 2020m3 Tijuana Chihuahua Juarez Tonala Zapopan Morelia Escobedo Guadalupe Monterrey Mexico 140 Figure 5. Percentage of Population 18 Years and Older that Has Witnessed the Existence of Street Gangs (ENSU) Figure 6. Population 18 Years and Older that Has Engaged in Any Conflict (ENSU) 0 10 20 30 40 50 60 2016m3 2016m6 2016m9 2016m12 2017m3 2017m6 2017m9 2017m12 2018m3 2018m6 2018m9 2018m12 2019m3 2019m6 2019m9 2019m12 2020m3 Tijuana Chihuahua Juarez Tonala Zapopan Morelia Escobedo Guadalupe Monterrey Mexico 0% 10% 20% 30% 40% 50% 60% 70% 80% mar-16 jun-16 sep-16 dic-16 mar-17 jun-17 sep-17 dic-17 mar-18 jun-18 sep-18 dic-18 mar-19 jun-19 sep-19 dic-19 mar-20 Total general Tijuana Chihuahua Juárez Tonala Zapopan Morelia Escobedo Guadalupe Monterrey Mexico