PERFORMANCE EVALUATION OF ALIANZAS MUNICIPALES PARA LA PREVENCION DE LA VIOLENCIA (AMUPREV) FINAL REPORT JULY 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Democracy International, Inc. under Order No. AID-OAA-TO-13-00047, Contract No. AID-OAA-I-10-00004 PERFORMANCE EVALUATION OF ALIANZAS MUNICIPALES PARA LA PREVENCION DE LA VIOLENCIA (AMUPREV) FINAL REPORT JULY 2016 DISCLAIMER This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Democracy International. Members of the team included: Yemile Mizrahi, Team Leader; Erik Alda, Senior Crime Prevention Specialist; Alfredo Calderón, Local Expert; and Luis Enrique Amaya, Local Ex￾pert. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Photo: AMUPREV’s Webpage Contractor: Democracy International, Inc. 7600 Wisconsin Avenue, Suite 1010 Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com TABLE OF CONTENTS LIST OF ACRONYMS.....................................................................................................II EXECUTIVE SUMMARY ............................................................................................... III FINDINGS AND CONCLUSIONS ................................................................................ iii RECOMMENDATIONS ............................................................................................... vi EVALUATION REPORT ................................................................................................. 1 I. PURPOSE OF THE EVALUATION ........................................................................... 1 II. EVALUATION DESIGN, METHODS, AND LIMITATIONS ........................................ 2 III. CRIME AND VIOLENCE IN CENTRAL AMERICA .................................................. 4 IV. PROGRAM DESCRIPTION: AMUPREV ...............................................................10 THE ROLE OF MUNICIPAL GOVERNMENTS IN CRIME PREVENTION ..................12 V. FINDINGS ..............................................................................................................13 V. CONCLUSIONS AND LESSONS LEARNED .........................................................28 VI. RECOMMENDATIONS .........................................................................................30 ANNEX A: SCOPE OF WORK FOR AMUPREV EVALUATION ................................ A-1 ANNEX B: LIST OF PEOPLE INTERVIEWED ........................................................... B-1 ANNEX C: EVALUATION INSTRUMENTS ................................................................ C-1 ANNEX D: RESULTS OF SURVEY RESPONDENTS ................................................ D-1 ANNEX E: BIBLIOGRAPHICAL REFERENCES........................................................ E-1 ANNEX F: DISCLOSURE OF CONFLICT OF INTEREST .......................................... G-1 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT ii LIST OF ACRONYMS ANAM Asociación Nacional de Municipalidades AGAAI Asociación Guatemalteca de Alcaldes y Autoridades Indígenas AMUPREV Alianzas Municipales para la Prevención de la Violencia CAMCAYCA Confederación de Asociaciones Municipales de Centroamérica y el Caribe CARSI Central America Regional Security Initiative CBSI Caribbean Basin Security Initiative COMURES Corporación de Municipalidades de la República de El Salvador COMUPRE Comisión Municipal de Prevención CMPV Comisión Municipal de Prevención de la Violencia ICMA International City/County Management Association PNC Policía Nacional Civil PREPAZ Dirección General de Prevención Social de la Violencia y Cultura de Paz UNODC United Nations Office on Drugs and Organized Crime USAID United States Agency for International Development UPCV Unidad para la Prevención Comunitaria de la Violencia SICA Sistema de Integración para Centroamérica WHO World Health Organization PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT iii EXECUTIVE SUMMARY Democracy International, Inc. (DI) conducted a mid-term performance evaluation of a regional crime and violence prevention program, the Municipal Partnerships for Violence Prevention, or Ali￾anzas Municipales para la Prevención de la Violencia (AMUPREV) in Spanish. The program is im￾plemented by the International City/County Management Association (ICMA) in Central America and the Dominican Republic. The program is managed by the United States Agency for Internation￾al Development (USAID)’s Bureau for Latin American and the Caribbean’s Office for Regional Sus￾tainable Development, Democracy, and Human Rights Team in Washington, D.C. (LAC/RSD, DHR). At the request of USAID’s Bureau for Latin America and the Caribbean, DI evaluated AMUPREV’s results only in El Salvador and Guatemala, where the program concluded its activities in December 2015. Currently, AMUPREV is only conducting direct work with municipalities in Honduras and the Dominican Republic. AMUPREV continues its regional work with the RED AMUPREV, a regional network of municipal associations created by the program to disseminate information and promote the exchange of experiences implementing crime and violence prevention programs at the munici￾pal level. The main objectives of the evaluation were to assess whether the program had accomplished its overall goal of increasing municipal actors’ capacity to design and implement crime and violence prevention programs, to examine if it had achieved unexpected outcomes, to draw lessons learned, and to formulate a set of recommendations for improving the program, including through expan￾sion or contraction of specific program activities, as well as recommendations for replicating some of the program’s most effective components in other USAID-funded municipal programs addressing violence prevention. In the context of escalating levels of violence reaching epidemic proportions in several Central American countries, the AMUPREV program was launched in 2009 to empower and increase the capacity of municipal stakeholders to design and implement crime and violence prevention activities at the municipal level. Central American municipal governments for the most part have not been actively engaged in addressing the problem of crime and violence, even though they have been in￾creasingly recognized in national prevention plans and strategies as critical stakeholders responsible for taking measures to prevent crime and violence in their communities. As a regional program, AMUPREV had a limited budget to fund its activities. USAID initially allocat￾ed $1.5 million dollars for a three year project (2009-2012) and increased funding with a total of $5 million currently. The program has been extended several times and is now set to conclude in 2017. On average, the program has spent $500,000 dollars per year. In each country, AMUPREV worked in a targeted number of municipalities (five in El Salvador and six in Guatemala), providing training and technical assistance to municipal stakeholders. The program also invited a number of municipal officials from beneficiary municipalities and leaders of municipal associations to participate in learning exchanges with municipal counterparts in different cities in the United States. In an effort to scale up its impact and reach a larger number of individuals outside the beneficiary municipalities, AMUPREV created a regional network of municipal associations (RED AMUPREV) and a website (www.amuprev.org) to disseminate news on municipal crime and violence prevention interventions throughout Latin America and the Caribbean region. FINDINGS AND CONCLUSIONS AMUPREV’s support to local governments, even if limited, responded to a genuine need. Beneficiar￾ies in El Salvador and Guatemala greatly appreciated AMUPREV’s program and perceived the learn- PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT iv ing exchanges to be the single most interesting and inspiring activity of the program. All beneficiaries interviewed, however, observed that the program was too short and financially too limited to allow for results to materialize. Five of the 11 municipalities that participated in the exchange program replicated or adapted expe￾riences observed in the U.S. The difference between municipalities that managed to replicate expe￾riences and those that did not can be partly explained by the motivation and engagement of the mayors in each municipality involved in programming. Political will is the single most important de￾terminant behind a mayor’s decision to invest in crime and violence prevention initiatives. In all 11 municipalities assisted by the program in El Salvador and Guatemala, AMUPREV worked with the municipal violence prevention committee, a local entity recognized in national prevention plans and programs as responsible for coordinating with the central government for the design, funding, and implementation of violence and crime prevention initiatives. In each municipality, AMUPREV also supported the implementation of selective crime and violence prevention initiatives through small grants. AMUPREV established a regional network of municipal associations to maximize the results of its technical assistance and exchange programs, which only benefitted a few municipalities in each country. The network also has a website which disseminates news and information on experiences of municipal crime and violence prevention initiatives throughout Latin America and the Caribbean. In Guatemala, the two municipal associations, the Asociación Nacional de Municipalidades de la República de Guatemala (ANAM) and the Asociación Guatemalteca de Alcaldes y Autoridades Indígenas (AGAII), collaborated with AMUPREV during implementation in Guatemala. Once AMUPREV concluded its activities in the country and the leadership of these two organizations changed in early 2016, Guatemalan municipal associations became passive and ceased to collabo￾rate as actively with the AMUPREV network. This is particularly problematic for the future and sus￾tainability of the network, since AMUPREV is currently housed in a regional association of munici￾palities, the Confederación de Asociaciones Municipales de Centroamérica y el Caribe (CAM￾CAYCA), which has a rotating presidency and is currently led by ANAM in Guatemala. In El Salvador, AMUPREV did not succeed in engaging the municipal association, the Corporación de Municipalidades de la República de El Salvador (COMURES). Although it signed a memorandum of understanding with AMUPREV, COMURES has refused to collaborate with the program. AMUPREV’s chief of party claimed that COMURES leadership resists collaborating with USAID￾funded projects. The AMUPREV program was well-designed, combining good incentives to motivate and empower municipal stakeholders, offering technical assistance and training, funding projects through small grants, and establishing a network of municipal associations to share news on municipal prevention interventions and disseminate lessons learned. Aside from the initial resistance of mayors to engage in prevention interventions, the main problem AMUPREV confronted in El Salvador and Guatemala was that the program objectives were too ambitious for the level of funding and the time allocated to work in each country. IMPLEMENTATION CHALLENGES AMUPREV confronted administrative, political, and conceptual implementation challenges. Although AMUPREV succeeded in doing many activities with limited resources, financial limitations seriously constrained AMUPREV’s ability to accomplish its overall goal of empowering municipal administra￾tions to implement crime and violence prevention programs, especially considering the magnitude and severity of the problem of violence in these countries. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT v In most municipalities where AMUPREV worked, the existence of other and much larger donor￾funded crime and violence prevention programs made AMUPREV’s program more difficult to dis￾tinguish and its results more difficult to ascertain, especially by beneficiaries. Beneficiaries often could not identify which initiatives were funded by AMUPREV and which by other programs. However, in small and rural municipalities, especially in those that typically do not receive donor funds, benefi￾ciaries more easily recognized and appreciated AMUPREV. In these municipalities, a small grant can make a big difference, both for the municipality and for its citizens. Another administrative challenge was the limited period of time that AMUPREV was able to spend working in each municipality, which was, on average, two years. The promotion of improved munic￾ipal leadership and the support for comprehensive municipal-based violence prevention strategies and programs requires a long-term approach to consolidate, take root and demonstrate results. Politically, the most serious implementation challenge was mayors’ limited political will to support and embrace crime and violence prevention programs. AMUPREV worked hard to convince mayors and leaders of municipal associations to support these types of activities. While AMUPREV succeeded in engaging mayors, in Guatemala, virtually all mayors that collaborated by the AMUPREV program lost the elections in 2015. As a result, AMUPREV’s accomplishments appear to be lost as well. The new mayors and their municipal staff interviewed by the DI team did not demonstrate any interest in or knowledge of crime and violence prevention. AMUPREV left Gua￾temala precisely as new mayors came to office, a situation that only exacerbated sustainability chal￾lenges. Finally, many mayors and municipal stakeholders interviewed during this evaluation demonstrated difficulty understanding the concept of crime and violence prevention. The concept was often con￾fused with law enforcement—perceived as deterrence—or with broader social policies or pro￾grams municipalities regularly implement, such as building roads, paving streets, installing street lamps, or organizing sporting events. When everything can be regarded as prevention, it is difficult to evaluate the results of prevention interventions. Another related conceptual challenge is AMUPREV’s failure to identify appropriate outcome indicators to measure and evaluate the results of its program. A lack of appropriate indicators increases the risks of implementing activities without a clear strategic purpose. COORDINATION WITH OTHER BILATERAL PROGRAMS In Guatemala, AMUPREV managed to coordinate well with other USAID-funded programs imple￾mented by RTI and Mercy Corps, as the objectives of these programs were complementary. Pro￾grams managed by RTI and Mercy Corps focused on violence prevention at the community level while AMUPREV worked with the municipal government. In the case of El Salvador, however, while AMUPREV coordinated well with the program implemented by RTI in the municipality of Na￾huizalco, it did not coordinate well with the program implemented by Creative Associates in Suchi￾toto because this program also offered assistance to the municipal government, thus duplicating AMUPREV efforts. Moreover, in El Salvador, AMUPREV worked in two municipalities (San José Guayabal and San Bartolomé Perulapía) in response to a request from PREPAZ (Dirección General de Prevención Social de la Violencia y Cultura de Paz), the central government agency responsible for coordinating with municipal administrations for designing and implementing crime and violence prevention programs. AMUPREV did not coordinate with USAID activities in these two municipali￾ties because USAID did not have partners implementing programs in these cities. SUSTAINABILITY In municipalities where AMUPREV worked it contributed significantly to promoting a paradigm shift in how to tackle crime and violence at the municipal level, however it is unclear if mayors will main￾tain their level of commitment to prevention activities in the absence of AMUPREV. In Guatemala PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT vi and El Salvador, the program was too small and too short to develop a well-designed and realistic sustainability strategy. Many of the activities funded by AMUPREV were generally short-term and driven mainly by the availability of funding. The lack of commitment on the part of leaders of municipal associations in Guatemala and El Salva￾dor to engage with the program further affects the sustainability of AMUPREV, especially the net￾work, since municipal associations should be the owners of the AMUPREV network and become their principal stakeholders, users and promoters. RECOMMENDATIONS Based on the evaluation’s findings, DI has developed the following recommendations to improve the program and to replicate some of the program’s most effective components in other USAID￾funded municipal programs addressing violence prevention. I. Donor-funded municipal crime and violence prevention programs should be designed with longer time frames and adequate, secure, levels of funding. Mayors, even when initially commit￾ted to supporting violence prevention programs, tend to prefer investing in projects that have quick or visible results and that have electoral payoffs. Prevention interventions, which take time to demonstrate results and affect behavioral and/or attitudinal changes, are not politically expe￾dient in the short term and are disrupted by electoral cycles. For this reason, funding of crime and violence prevention activities should be sufficient to allow programs to plan strategically and for the longer term. II. In addition to designing appropriate training and technical assistance interventions, donor￾funded municipal crime and violence prevention programs like AMUPREV need to introduce the right incentives to motivate mayors to engage and remain committed to prevention. The CityLinks program was one of AMUPREV’s greatest accomplishments because it effectively mo￾tivated and encouraged municipal stakeholders to support prevention interventions. This type of incentive should be replicated in other municipal crime and violence prevention programs. III. To increase local buy-in and improve program sustainability, regional donor-funded crime and violence prevention programs should maintain a strong presence at the municipal level to sup￾port and mentor mayors and their administrations and to engage with civil society organiza￾tions. Programs should also invest in socializing crime and violence prevention initiatives with the population at large to increase social support for these programs, generating greater social demand for these types of initiatives, and thereby increasing the political payoffs for mayors to invest in crime and violence prevention programs. IV. To increase the sustainability of municipal association networks, leaders of municipal associa￾tions need to be fully committed to establishing and maintaining these networks. These types of networks depend on the ability of leaders to demonstrate concrete benefits to their partici￾pants. A program that seeks to establish such networks needs to maintain a longer presence in each country, engage directly with a larger number of municipal governments, and implement a strong communications campaign to disseminate information about the network and educate potential participants about its benefits. V. Political factors such as elections can significantly affect program sustainability and should be taken into account during program design. While electoral cycles are beyond the control of the international donor community, donor-funded programs can include measures to mitigate these risks. For example, in Guatemala AMUPREV could have decided to stay longer or executed a more gradual phase-out to prevent losing momentum on the work it had achieved with the municipal prevention committees. In addition, this would allow AMUPREV to work with the PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT vii newly elected mayors, educating them on previous accomplishments and motivating them to continue supporting the program. VI. Small and rural municipalities pose greater challenges to the sustainability of donor-funded pro￾grams because they have limited financial and human resources and weak institutional capaci￾ties. In these municipalities where programs like AMUPREV can become more apparent and appreciated by local stakeholders, it is even more critical to design measures to mitigate the risks of donor dependency. This can be accomplished by investing more strategically, ensuring that activities are related to the program’s overall goal, and enhancing municipal government of￾ficials’ capacity to generate local sources of income to fund prevention initiatives. VII. A regional municipal program working on crime and violence prevention should have a decen￾tralized decision making structure, granting technical directors in each country sufficient spend￾ing authority to respond more expeditiously to local needs and demands. VIII. Crime and violence prevention programs should develop explicit theories of change and identi￾fy better performance indicators. Robust monitoring and evaluation frameworks increase pro￾grams’ accountability and help implementers maintain a more strategic focus on the activities they support. To reach program indicator targets, activities have to be clearly linked to the re￾sults the program is seeking to achieve. Unfortunately, lack of appropriate indicators is a short￾coming prevalent among many donor-funded crime and violence prevention programs. In a re￾gion where there is poor understanding of crime and violence prevention, where the political will of government officials to support prevention interventions is uncertain, and where citizens understandably demand from their government quick and visible results, crime and violence prevention programs often support a host of ad-hoc activities without much strategic planning, focus or direction. Examples are numerous, but some examples include supporting one-time events like music concerts and sports tournaments, purchasing surveillance cameras for police officers, and financing arts and craft workshops not appropriately targeted to the right benefi￾ciaries, i.e. youth. IX. Donor-funded violence prevention programs should pay greater attention to the development of outcome, rather than only output indicators to assess whether they are accomplishing their overall objectives. Aside from increasing a program’s accountability, outcome indicators are a critical learning tool to identify what works and what does not and to assess the changes need￾ed to improve program effectiveness. X. USAID should consider whether regional programs could be managed from regional offices. USAID officials in Guatemala and El Salvador agreed that a regional program of municipal crime prevention should have been managed from the El Salvador regional office, not from Washing￾ton, D.C. Although officials agree that regional programs are innovative and can be effective, lack of direct management and supervision from the regional office limits the effectiveness of these programs to adapt to particular conditions, respond more expediently to country-specific demands, and coordinate more effectively with other bilateral donor-funded programs. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 1 EVALUATION REPORT I. PURPOSE OF THE EVALUATION At the request of the United States Agency for International Development (USAID)’s Bureau for Latin America and the Caribbean, Democracy International, Inc. (DI) conducted a performance evaluation of a regional crime and violence prevention program, the Municipal Partnerships for Vio￾lence Prevention, or Alianzas Municipales para la Prevención de la Violencia (AMUPREV) in Spanish, implemented by the International City/County Management Association (ICMA) in Central America and the Dominican Republic. This evaluation was conducted under Task Order #AID-OAA-TO￾13-00047, which provides analytical services to USAID’s Bureau for Latin America and the Caribbe￾an’s Office for Regional Sustainable Development, Democracy and Human Rights Team (LAC/RSD/DHR). AMUPREV is funded by LAC/RSD/DHR funds as well as by the Central America Regional Security Initiative (CARSI) and the Caribbean Basin Security Initiative (CBSI) and is managed by USAID/LAC/RSD/DHR in Washington DC1. AMUPREV focuses on two key objectives: (1) to pro￾mote comprehensive municipal-based violence prevention strategies and programs with key Central American stakeholders and foster development of regional peer knowledge networks; and (2) to provide training and technical assistance to local governments and community groups, in coordina￾tion with national police efforts and other municipal-based programs. The program began in 2009 and is currently in its seventh year of implementation. It has worked in Panama, El Salvador, Guatemala, Honduras and the Dominican Republic. At the request of USAID’s LAC/RSD/DHR Team, DI evaluated AMUPREV’s results only in El Salvador and Guatemala, where the program concluded its activities in December 2015. Currently, AMUPREV is only conducting direct work with municipalities in Honduras and the Dominican Republic. AMUPREV continues its regional work with the RED AMUPREV, a regional network of municipal associations created by the program to disseminate information and exchange experiences of implementation of crime and violence prevention programs at the municipal level. The main objectives of the evaluation were to assess whether the program had accomplished its overall goal of improving municipal actors’ capacity to design and implement crime and violence prevention programs, to examine if it had achieved unexpected outcomes, to draw lessons learned, and to formulate a set of recommendations for improving the program, including through expan￾sion or contraction of specific program activities, as well as recommendations for replicating some of the program’s most effective components in other USAID-funded municipal programs addressing violence prevention. The intended audience for this evaluation is USAID staff working on citizen security initiatives in Central America and other regions; local and national government officials in Central American and the Dominican Republic; regional government organizations such as the Central American Integra￾tion System; the U.S. Department of State, the National Security Council, Congressional stakehold￾ers and other USG agencies; municipal associations in each implementation country, including the Confederation of National Municipal Associations in Central American and the Caribbean (CAM￾CAYCA); and civil society and academic organizations interested in citizen security issues. 1 Interview with USAID/LAC/RSD/DHR staff, June 2016. According to USAID/LAC/RSD/DHR staff inter￾viewed for this evaluation, at the time the program was designed, they decided the program should be man￾aged from Washington, DC because the regional mission in El Salvador did not yet have the capacity to de￾sign and manage regional programs. This decision was taken jointly with staff from USAID/El Salvador. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 2 The intended uses of this evaluation are to guide USAID program management decision-making, including scope and focus of ongoing and future security programs; to disseminate lessons learned and promising practices for effective implementation, including those for addressing implementation challenges; and to help USAID identify programmatic and/or geographic gaps in the program that should be addressed by USAID or other actors. Please see Annex A for the Scope of Work of this evaluation. USAID formulated the following four evaluation questions: (1) To what degree has creation of a municipal-level crime and violence prevention network in the region facilitated design and implementation of new and/or improved crime and vio￾lence prevention programs by key stakeholders, namely national-level municipal associations and individual municipalities? (2) To what degree has municipalities’ participation in AMUPREV learning exchanges contrib￾uted to design and implementation of new and/or improved crime and violence prevention programs? (3) To what degree is AMUPREV complementary to existing bilateral Mission programs? To what degree is AMUPREV overlapping or working at cross-purposes with bilateral Mission programs? (4) What implementation challenges, if any, have negatively affected program implementation? This could include internal factors, such as challenges coordinating with several Missions, or external factors such as host countries’ legal frameworks, political will, capacity of local partners, availability of information, etc. II. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation was conducted by a team assembled by Democracy International and composed of Team Leader Yemile Mizrahi, DI’s Director of Research and Analytical Services, Erik Alda, Senior Crime Prevention Specialist, Alfredo Calderón, Local Expert in Guatemala, and Luis Enrique Amaya, Local Expert in El Salvador. The team leader was responsible for the overall guidance of the evalua￾tion, including the design of the evaluation methodology, formulation of interview protocols, data analysis and synthesis, and production of the evaluation report. The Senior Crime Prevention Spe￾cialist contributed to the evaluation design and methodology and was responsible for designing the survey instrument and analysis of the quantitative data. The two local consultants, recruited for their expertise in crime and violence prevention and experience working with local governments, con￾tributed with substantial contextual information on crime and violence dynamics in each country, participated in the data collection process and provided invaluable analytical feedback on infor￾mation collected during field work. Before traveling to the field, the DI team reviewed AMUPREV’s quarterly reports and other project documents, and conducted a desk study of academic and development practitioners’ reports on the situation of crime and violence in Guatemala and El Salvador and the role of municipal governments in the design and implementation of crime and violence prevention interventions. The team also reviewed national prevention plans and strategies in Guatemala and El Salvador. Please see Annex E for a full bibliographical list of these sources. The team conducted field work in Guatemala and El Salvador from April 25 to May 7, 2016 and interviewed a total of 97 individuals (63 men and 34 women). AMUPREV provided a list of benefi￾ciaries in all the municipalities and the municipal associations where it worked in El Salvador and Guatemala. Additionally, the team interviewed USAID officials in Guatemala and El Salvador who manage crime and violence prevention activities in their countries, a few central government officials PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 3 responsible for coordinating with municipal authorities in the implementation of national prevention plans and programs, and with academics recommended to the team by the local experts. Annex B presents the list of people interviewed for this evaluation. In each country, the team visited every municipality where AMUPREV implemented activities. In Guatemala, the team visited Mixco, Palen￾cia, Santa Catarina Pinula, Cobán, Tac-Tic and San Juan Chamelco. In El Salvador, the team visited Nahuizalco, Sonsonate, Suchitoto, San José Guayabal and San Bartolomé Perulapía. LIMITATIONS AND APPROACHES TO MITIGATING BIAS The evaluation was originally designed as a mixed methods evaluation using qualitative and quantita￾tive methods of data collection and analysis. The qualitative component relied on personal and/or group interviews with program implementers and beneficiaries as the main source of data. Annex C presents the list of interview questions developed for program implementers and beneficiaries. For the quantitative component, DI developed a survey instrument to assess the effect of AMUPREV’s activities on beneficiaries’ capacity to design and implement crime and violence prevention pro￾grams. The purpose of this survey was to collect data from a broader group of direct beneficiaries of AMUPREV’s programs and from individuals who consult AMUPREV’s website outside Guatemala and El Salvador. Results of the survey were also going to be used to triangulate findings obtained from qualitative data. AMUPREV’s communication director sent the survey online to all of (about 2,500) AMUPREV’s contacts in its listserv. The email requesting individuals to respond to the survey was signed by AMUPREV’s chief of party and contained a brief note explaining the purpose of this survey and eliciting individuals’ responses. Unfortunately, and despite two reminders, only 48 people responded to the survey, and therefore it could not be used for this evaluation as results are not statistically representative. Annex D presents analysis of data collected from the 48 survey respons￾es. As a qualitative evaluation, the evaluation was subject to several types of bias, including acquies￾cence bias, a type of cognitive bias where respondents answer questions in the way they think the interviewer wants them to rather than according to their true beliefs. Although it is difficult to de￾termine the extent of this bias, it is conceivable that respondents may have been either reluctant to criticize the program or prone to overstating the effectiveness of its activities, believing that a posi￾tive evaluation could lead to follow-on programs. To mitigate this bias, the team repeatedly empha￾sized their neutrality, assuring respondents that they had no influence on funding decisions, and con￾firmed findings by probing similar topics with various stakeholders to see whether findings con￾verged. Moreover, this evaluation was also subject to selection bias, due to the nonrandom nature of the sampling of beneficiaries interviewed. Findings reflect the opinions of the stakeholders interviewed, but are not generalizable to the entire population that participates in the AMUPREV program or that reads AMUPREV’s bulletins or visits the AMUPREV website. To mitigate this bias, the team compared information it obtained across multiple interviews, and triangulated information by dis￾cussing its findings with USAID representatives in El Salvador and Guatemala as well as with AMUPREV’s implementers. Finally, in Guatemala, the team was unable to interview several former mayors of Cobán and Mixco, who were beneficiaries of AMUPREV’s program. In El Salvador, officials from the municipal associa￾tion COMURES refused to grant interviews to the DI team.2 2 The team contacted COMURES representatives more than ten times to schedule an interview. AMUPREV provided the names of these representatives and their contact information. COMURES alleged having scheduling problems. In conversation with AMUPREV’s team leader, however, he acknowledged that PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 4 III. CRIME AND VIOLENCE IN CENTRAL AMERICA Over the past fifteen years, El Salvador, Honduras, and Guatemala have experienced an unprece￾dented escalation of crime and violence, including domestic violence, extortions, theft, trafficking in persons, drug trafficking, and homicides. Although citizens in these countries suffer from varied man￾ifestations of crime and violence, the high homicide rate has captured international attention and is commonly used to indicate the severity of the problem. According to the United Nations Office on Drugs and Crime (UNODC), the 2011 homicide rate in Central America was 25 per 100,000 in￾habitants, which was four times higher than the global average of 6 homicides per 100,000 inhabit￾ants. According to the World Health Organization (WHO), any country with a homicide rate above the threshold of 10 per 100,000 inhabitants is suffering a violence epidemic.3 In 2015, El Sal￾vador surpassed Honduras as the most violent country in the region with a homicide rate of 103 per 100,000 inhabitants. Homicide rates in Honduras and Guatemala that year were 57 per 100,000 and 29.5 per 100,000, respectively.4 Of particular concern in the Central American Region and the Dominican Republic are crime and violence related to criminal gangs and transnational criminal organizations. Perpetrators of these types of crimes are typically young men. In 2010, the United Nations and the U.S. Southern Com￾mand estimated that there were about 70,000 gang members concentrated in the Northern Trian￾gle: 36,000 in Honduras, 10,500 in El Salvador and 14,000 in Guatemala. Most of these are concen￾trated in two gangs: the Mara Salvatrucha-13 (MS-13) and Barrio 18.5 The depth of the relationship between gangs and international drug trafficking organizations is not completely understood, but there are apparent commonalities in the forces that compel young men to join these criminal organizations, including high levels of marginalization, disintegration of the family structure, income inequality, lack of opportunities (especially for youth), limited police capaci￾ty to contain crime, and severe weaknesses in the justice system (such as extremely high levels of impunity). Guatemala and El Salvador have a long history of violent conflict derived from prolonged and bloody civil wars lasting twelve years in the case of El Salvador (1980-1992), and of more than thirty years in Guatemala (1960-1996). These wars undoubtedly affected the social fabric of these coun￾tries, displacing thousands of citizens from their hometowns and pushing many to migrate to the United States. This legacy of violence and the corresponding erosion of the social fabric have con￾tributed to the weakening of informal social controls, which are important for preventing people from turning to criminal activities and increasing their resiliency in adverse environments. Faced with the violence epidemic, academics and development practitioners have written extensively about its leading causes and the different risk factors at the family, community and societal level. But little consensus exists on the most effective strategies and policies to control and revert the high inci￾dence of crime and violence. COMURES is reluctant to work with USAID and this is the reason why they refused to grant us an interview. The team could not corroborate this information. 3 “Global Study on Homicide, 2013,” United Nations Office on Drugs and Crime, https://www.unodc.org/documents/gsh/pdfs/2014_GLOBAL_HOMICIDE_BOOK_web.pdf . 4 These rates are based on official data from the National Police in each country. 5 Steven Dudley, “How Mano Dura is strengthening gangs,” Insight Crime, November 21, 2010, http://www.insightcrime.org/investigations/how-mano-dura-is-strengthening-gangs. Estimates on the number of gang members range widely from agency to agency and from country to country. Part of the problem is that there are no official records of gang membership and government agencies have no consensus on who can be considered a gang member. There is also little understanding on the international organization and opera￾tion of gangs and the distinctions between different types of engagement with gangs, ranging from sympathiz￾er, collaborator, forced collaborator, and full member. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 5 In El Salvador, Guatemala and Honduras (referred to as the Northern Triangle), policymakers have traditionally given preference for more repressive policies involving massive arrests and longer sen￾tencing periods to address high and increasing levels of violence. The “Mano Dura” approach was first articulated by then-President Francisco Flores in El Salvador in 2003 when the number of hom￾icides attributed to gang violence began to escalate. Honduras and Guatemala followed suit with different variants of the same hardline approach.6 Due to delays in the judicial process, massive numbers of arrests and limited space in prisons, the Mano Dura approach led to prison overcrowd￾ing, which in turn allowed prisons to become schools for gang activity. When the Mano Dura policies failed to substantially contain the violence epidemic, governments in Central America began to support prevention approaches, recognizing that containing the violence epidemic required a more integrated approach that addressed the root causes of crime and vio￾lence. By 2006, the Government of El Salvador created the National Commission of Citizen Securi￾ty and Social Peace (Comisión Nacional de Seguridad Ciudadana y Paz Social) to draft comprehen￾sive short, medium and long-term policy recommendations in the areas of prevention, rehabilitation, reinsertion and law enforcement. 7 In Guatemala, in 2014, the government elaborated its National Crime and Violence Prevention Policy, recognizing that prevention is a more effective, efficient and sustainable strategy for combatting violence and crime.8 Currently, both governments in Guatemala and El Salvador continue to endorse—at least rhetori￾cally—prevention policies and plans, but they have also adopted tough anti-gang approaches. Alt￾hough the law enforcement and prevention approaches are not necessarily mutually exclusive, there is still enormous debate about the effective mix of prevention and repression interventions that are necessary to curb and contain this violent epidemic. USAID has been one of the most important supporters of prevention approaches in the region and has advocated for the need to address the root causes of violence through programmatic interven￾tions as a way to find more lasting and sustainable solutions. In addition to promoting and support￾ing justice sector reform, anti-corruption efforts, and reforms for more professional and effective law enforcement approaches, as well as funding crime and violence prevention programs through￾out the region, USAID has also supported governments in planning and articulating national preven￾tion policies. In Guatemala, USAID worked with the central government in the elaboration of the National Policy on Crime Prevention 2014-2034 (Política Nacional de Prevención del Delito) and lobbied in favor of the creation of the Third Vice-Ministry of Prevention (Tercer Vice-Ministerio de Prevención de la Violencia y el Delito) within the Ministry of Government (Ministerio de Gobernación). This Vice￾Ministry is responsible for leveraging resources, coordinating with other government ministries and supporting local governments in designing and implementing prevention plans. To that end, it estab￾lished a technical unit, the Community Violence Prevention Unit (Unidad para la Prevención Co￾munitaria de la Violencia or UPCV), that sends representatives to municipalities to provide technical assistance to local prevention committees. Although the prevention institutional infrastructure has been established, in interviews with representatives of this Unit, they recognized that they face seri- 6 Claire Ribando Seelke, “Anti-gang efforts in Central America: Moving Beyond Mano Dura?” Center for Hemispheric Policy, University of Miami, April 10, 2007. https://www.justice.gov/sites/default/files/eoir/legacy/2013/11/08/crs_antigang.pdf 7 For an analysis of the evolution of crime prevention policies in El Salvador, see Luis Enrique Amaya, “Siste￾matización de Políticas Públicas de los Ministerios Rectores de la Prevención de la Violencia en el Marco de la Estrategia de Seguridad de Centroamérica (ESCA) en Dos Municipios de El Salvador,” (GIZ manuscript: Sep￾tember, 2015), p. 7. 8Gobierno de Guatemala, “Política Nacional de Prevención del Crimen y el Delito (2014-2034),” p. 41. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 6 ous staff and budgetary restrictions in providing the necessary technical assistance to local govern￾ments.9 Similarly, in El Salvador, USAID collaborated directly with the government of then-President Mauri￾cio Funes to draft the National Justice, Public and Citizen Security Policy (Política Nacional de Justi￾cia, Seguridad Pública y Convivencia) and the National Prevention Strategy in Support of Local Governments (Estrategia Nacional de Prevención de la Violencia en Apoyo a Municipios) in 2010. More recently, USAID supported the elaboration of the new security plan, Plan El Salvador Seguro which prioritizes prevention and assigns 74 percent of the total budget dedicated to citizen security to fund prevention activities.10 The Ministry of Justice and Public Security (Ministerio de Justicia y Seguridad Pública) is responsible for planning and coordinating with other government agencies the implementation of prevention programs and has a specific unit, the General Direction for Social Prevention and a Culture of Peace (Dirección General de Prevención Social de la Violencia y Cultu￾ra de Paz or PREPAZ), that provides training and technical assistance to local governments, and more specifically, to local prevention committees. Although PREPAZ does have permanent repre￾sentatives in each department, in interviews with the DI team, PREPAZ representatives in Cuscatlán and Sonsonate acknowledged they do not have any budget to support local prevention commit￾tees.11 In El Salvador and Guatemala, national citizen security plans and policies acknowledge the im￾portance of crime and violence prevention for mitigating and containing the violence epidemic and explicitly recognize the centrality of municipalities in preventing crime and violence; describe the roles and responsibilities of each level of government; define the inter-governmental organizational structure for coordinating different levels of government and leveraging resources; and identify local institutional mechanisms to plan, design, and implement crime and violence prevention programs.12 The main problem in both countries is that to-date, the governments’ established institutional struc￾tures to support municipalities are underfunded and understaffed. As we elaborate below, AMUPREV’s support to local governments, even if limited, responded to a genuine need. EL SALVADOR AND GUATEMALA According to National Police data, in both El Salvador and Guatemala, homicide rates increased in 2009, with rates of homicide of 71.2 per 100,000 in El Salvador and 46.36 per 100,000 in Guatema￾la. While Guatemala has experienced a consistent, albeit gradual, decline in its homicide rate, in El Salvador the trend has been more erratic; declining sharply in 2012 after the government and the main gangs signed a controversial truce, and then escalating again in 2014 when the truce came to an end.13 9 Interview with the Executive Director of the Unidad para la Prevención Comunitaria de la Violencia (UPVC), Guatemala, April 25, 2016. In the interview, this official recognized that much prevention efforts are funded by the international donor community. 10 “Plan el Salvador Seguro,” Consejo Nacional de Seguridad Ciudadana y Convivencia, January 15, 2015, pp. 20-21, http://www.presidencia.gob.sv/wp-content/uploads/2015/01/El-Salvador-Seguro.pdf. The total budget for the implementation of the Plan el Salvador Seguro is $2.1 billion USD, with $1.5 billion USD assigned to prevention activities, out of which, 52 percent are oriented for education and employment. 11 Interview with PREPAZ representative in Cuscatlán, May 5, 2016; Interview with PREPAZ representative in Sonsonate, May 3, 2016. 12 In El Salvador, these institutional mechanisms are called Comisiones Municipales de Prevención de la Vio￾lencia (CMPV) and in Guatemala, they are called Comisiones Municipales de Prevención (COMUPRE). 13 According to different analysts, the truce had a significant effect in the reduction of the number of homi￾cides but it did not have any effect on the number of extortions, which allegedly went up dramatically. The truce demonstrated that gangs, especially MS13 which is internally more organized, have significant control over their members and can have a direct impact on the levels of violence. Yet, the truce also contributed to PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 7 Source: Guatemala and El Salvador National Civilian Police (PNC). Although compared to El Salvador, with a rate of homicides of 103 per 100,000 inhabitants, the situation in Guatemala appears to be more optimistic, a rate of homicides of 29.54 per 100,000 is still extremely high and considered to be of epidemic proportions.14 Like in the rest of the world, crime and violence are highly concentrated in particular places and involve particular groups of peo￾ple. Although both Guatemala and El Salvador have high incidences of crime and violence, not all municipalities are equally violent and, even in the most violent municipalities, not all neighborhoods are equally dangerous. In Guatemala, the highest incidences of violence and crime are concentrated in the capital and sur￾rounding municipalities, as well as in the municipalities along the border with Mexico, Honduras and El Salvador. In El Salvador, crime and violence are concentrated in the departments of San Salvador, La Libertad, Santa Ana and Sonsonate.15 increasing the gangs’ bargaining power and recognition as stakeholders. For an interesting analysis of the im￾pact of the truce on the levels of violence see Charles M. Katz and Luis Enrique Amaya “La tregua entre las pandillas como una forma de intervención sobre la violencia Implicaciones en políticas y practices.” (San Sal￾vador: Proyecto SolucionES, USAID, 2015). For a robust quantitative analysis using mathematical models to demonstrate the effects of the truce on the level of homicides, see Charles M. Katz, EC Hedberg, and Luis Enrique Amaya, “Gang Truce for Violence Prevention, El Salvador.” Bulletin of the World Health Organiztion, June 2016, http://www.who.int/bulletin/BLT.15.166314.pdf?ua=1. 14 Especially when compared to the global homicide rates, estimated by the United Nations Office of Drugs and Crime (UNODC) to be 6.2 per 100,000 inhabitants in 2012. (http://www.unodc.org/documents/gsh/pdfs/GLOBAL_HOMICIDE_Report_ExSum.pdf) 15 James Bargent, “80% Guatemala Murders Take Place in 25 Municipalities: President,” InSight Crime, April 8, 2013, http://www.insightcrime.org/news-briefs/80-guatemala-murders-take-place-in-25-municipalities￾president; “Corridor of Violence: the Guatemala-Honduras Border,” International Crisis Group, June 4, 2014, http://www.crisisgroup.org/~/media/Files/latin-america/Guatemala/052-corridor-of-violence-the-guatemala￾El Salvador Guatemala 0 10 20 30 40 50 60 70 80 90 100 110 2008 2009 2010 2011 2012 2013 2014 2015 Homicide Rate (deaths per 100,000 people) HOMICIDE RATES: EL SALVADOR AND GUATEMALA 2008-2015 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 8 As the maps and graph below shows, violence in Guatemala has experienced a decline from 2010 to 2014 (the most recent year with available data from all municipalities). Although violence has been persistently concentrated in the same municipalities during this period, in the southwestern part of the country violence has increased. In the northern Peten region, by contrast, the level of violence has declined despite the fact that this region has traditionally been a natural corridor for drug trafficking and other organized crime activities.16 Source: DI Team analysis based on data from PNC. In El Salvador, the levels of violence have intensified significantly from 2010 to 2015. Some munici￾palities have more than doubled their homicide rates during this period, and in few municipalities the homicide rates increased to six times their 2010 rate. The spike in homicide rates came after the end of the truce between the government and the two major gangs.17 honduras-border.pdf; “Atlas de la VIolencia en El Salvador (2009-2012),” Fundaungo, http://fundaungo.org.sv/pdf/2014/Atlas_de_violencia.pdf. 16 Katherine E. Bliss, “Trafficking in the Mesoamerican Corridor: A Threat to Regional and Human Security,” Center for Strategic and International Studies, November, 2009, https://csis-prod.s3.amazonaws.com/s3fs￾public/legacy_files/files/publication/091123_Bliss_Trafficking_Web.pdf. 17 Several analysts argue that the truce allowed gangs to reorganize and re-arm. In exchange for slowing down the pace of homicides, gang leaders were transferred from high security prisons to less secured prisons. Dur￾ing the truce disappearances and extortions continued, but homicides went down. Some argue that the truce allowed gang leaders to strengthen their operations from jail. See Claire Ribando Seelke, “Escalating Violence in El Salvador,” April, 2016, https://www.fas.org/sgp/crs/row/IN10382.pdf and Charles M. Katz and Luis En￾rique Amaya op.cit. Homicide Rates in Guatemala 2010-2014 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 9 Source: DI Team analysis based on data from PNC. The graphs below illustrate the evolution of the homicide rate in Guatemala and El Salvador from 2008 to 2015, both at the national level and in the 11 municipalities where AMUPREV worked in each country. Some of these municipalities were amongst the most dangerous in the country, with homicide rates above the national average. Source: PNC, Guatemala Homicide Rates in El Salvador 2010-2014 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 10 Source: PNC, El Salvador IV. PROGRAM DESCRIPTION: AMUPREV The Municipal Partnerships for Violence Prevention Project (AMUPREV), implemented by the In￾ternational City/County Management Association (ICMA), was launched in 2009. The program orig￾inated as a result of an assessment USAID LAC/RSD/DHR staff conducted when they designed the Central American Regional Security Initiative (CARSI) program. The assessment revealed that there was a funding gap in crime prevention at the municipal level. That year, several mayors from El Sal￾vador approached USAID and requested that they assign funding for crime prevention initiatives at the municipal level. They rightly noted that most international donor funding for crime and violence prevention was being directed either to national level institutions or to community based organiza￾tions, but municipal governments, who ultimately confront the consequences of crime and violence in their communities, were not being directly supported. ICMA, an organization that has supported local governments throughout Latin America, responded to the mayors’ request and drafted a con￾cept paper, requesting funding for a project to provide support to municipal governments through￾out Latin America to design and implement crime and violence prevention interventions. USAID initially agreed to fund this initiative, allocating $1.5 million dollars for a three year project (2009- 2012). The project began work in three municipalities in Panama and two municipalities in El Salvador. Af￾ter 2012, the project received two cost extensions for an additional $600,000 dollars until 2013. AMUPREV ended its work in Panama and increased coverage in El Salvador, assisting three addi￾tional municipalities. In April 2013, AMUPREV received another two-year cost extension of $1.5 million dollars and began to support six municipalities in Guatemala. By the end of 2015, AMUPREV concluded its work in Guatemala and El Salvador and, after receiving another two-year cost exten￾sion of $1.2 million dollars, began to support municipalities in the Dominican Republic and Hondu￾ras.18 18 Information obtained from AMUPREV’s Quarterly Reports. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 11 The overall goal of this regional project is to empower municipal governments and strengthen their capacity to implement effective crime and violence prevention activities in their communities. Since the eruption of the crime and violence epidemic in Central America, national government institu￾tions have taken the lead in designing strategies and implementing programs to control and prevent crime and violence. Despite the implementation of different decentralization reforms seeking to improve accountability, service delivery and citizen participation, local governments in Central America, for the most part, have not been actively engaged in issues relating to citizen security and they have not been proactive in designing and implementing programmatic interventions to address the problem of crime in their communities. 19 Yet, local governments are closer to the community, have more information about local social and political dynamics, and in many cases, are better equipped to identify solutions needed for resolving local conflicts. AMUPREV defined two strategic objectives to achieve its overall goal: 1: Promoting comprehensive municipal-based violence prevention strategies and programs with key Central American stakeholders and supporting the development of regional peer knowledge net￾works to ensure that crime prevention activities become part of the municipal agenda. 2: Supporting comprehensive municipal-based violence and crime prevention programs in select municipalities. To accomplish these objectives, AMUPREV worked with municipal governments to assist them in establishing and/or strengthening municipal institutions responsible for crime and violence preven￾tion. As mentioned above, in Guatemala and El Salvador, these institutions are called municipal committees for violence prevention (CMPV) and are recognized by national crime and violence prevention plans and strategies as the appropriate institutional mechanism at the municipal level to plan and design crime and violence prevention interventions. AMUPREV assisted these committees to conduct a diagnostic of crime and violence in each municipality and to elaborate strategic action plans. AMUPREV also funded a few activities in these municipalities through small grants programs. To foster a regional network of municipal crime and violence prevention practitioners, AMUPREV designed a partnership program based on the CityLinks model with a few cities in the U.S. so that practitioners and government officials could learn from another, share best practices and exchange lessons learned, specifically from the U.S. model of governance and community-oriented policing. AMUPREV invited mayors and members of CMPV of different Central America countries to Stock￾ton and Santa Ana in California; Alexandria, Virginia; Arlington, Texas and Pinellas County, Florida. Police officers, municipal staff, community and private sector representatives from these cities also visited their counterparts in Central America and provided training and technical assistance on how to reduce and mitigate the risk of violence in their communities. The overall goal of these exchang￾es was to share successful experiences, motivate Central American stakeholders and increase their awareness of the need to prioritize crime and violence prevention interventions and work in a co￾ordinated and comprehensive fashion. Finally, to ensure the sustainability of the AMUPREV program and reach out to a larger network of municipal governments in the region, the program established a regional network of municipal asso￾ciations, the RED AMUPREV, to share and disseminate information about crime and violence pre￾vention at the municipal level. A fundamental element of this network was the establishment of a website (www.amuprev.org) to disseminate news on municipal interventions throughout Latin America and the Caribbean region to prevent crime and violence in their communities. The website 19 Clare Ribando Seelke, “Anti-Gang Efforts in Central America: Moving Beyond Mano Dura?” Center for Hemispheric Policy, University of Miami, April 10, 2007, p. 7, https://www.justice.gov/sites/default/files/eoir/legacy/2013/11/08/crs_antigang.pdf. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 12 was launched in 2014 and is maintained by AMUPREV staff. It showcases successful and innovative experiences, provides articles related to municipal crime and violence prevention, and offers practi￾cal tools and guides for municipal stakeholders on how to design, plan and implement crime and violence prevention programs. In addition to the website, AMUPREV also sends a weekly bulletin to its list of more than 2,500 stakeholders throughout the Latin America and the Caribbean region and has a Facebook account where users can exchange or solicit information and access the AMUPREV website. AMUPREV rests on the principle that municipalities need to become more actively engaged in pre￾venting crime and violence if the epidemic of violence is to be mitigated and eventually contained. Despite the many limitations municipal governments face in terms of resources, capacities, and au￾thority, municipal governments have sufficient room to maneuver to leverage resources, design programs that address the needs of their citizens, and reduce the risks of violent behavior. AMUPREV assumes that change is possible provided municipal governments increase their under￾standing of prevention, build their internal capacities and develop greater political will to invest in programs that address the problem of crime and violence. THE ROLE OF MUNICIPAL GOVERNMENTS IN CRIME PREVENTION In Guatemala and El Salvador the role of municipal governments in crime and violence prevention has been explicitly recognized in national and strategic prevention policy documents introduced since 2010,20 yet many mayors still lack the knowledge and the political will to take greater respon￾sibility for directly addressing the risks of crime and violence in their territories. As we discuss in greater detail below, this is for three primary reasons: First, many government officials, including mayors in these countries, understand crime and violence primarily as a law enforcement problem, rather than the result of entrenched social problems relat￾ed to marginalization, exclusion, inequality, disintegration of the family structure, and lack of oppor￾tunities, especially for youth. Given that police forces in Central American countries are centralized, mayors believe that controlling and preventing crime falls within the police’s responsibility, which is outside the purview of municipal administrations.21 Second, most mayors suffer from insufficient financial resources to deliver adequate public services, and consequently, they believe they do not have the resources to invest in prevention activities.22 20 Luis Enrique Amaya, “Sistematización de Políticas Públicas de los Ministerios Rectores de la Prevención de la Violencia en el Marco de la Estrategia de Seguridad de Centroamérica (ESCA) en Dos Municipios de El Salvador,” Op.cit. In El Salvador, the Política Nacional de Justicia, Seguridad Pública y Convivencia, introduced in 2009, explicitely recognized that municipal governments play a fundamental role in crime prevention. In 2010, the government introduced the Estrategia Nacional de Prevención de la Violencia en Apoyo a Muni￾cipios, and established the creation of municipal violence prevention commissions as the institutional mecha￾nism to coordinate between local and central government agencies. In Guatemala, the Política Nacional de Prevención del Delito also recognizes the importance of municipal governments in preventing crime and vio￾lence and identifies the establishment of municipal commissions of crime prevention as institutional mecha￾nisms at the municipal level to design and implement prevention programs. See “Modelo de Abordaje, Uni￾dad para la Prevención Comunitaria de Violencia (UPVC)”, Viceministerio de la Prevención del Delito, Minis￾terio de Gobernación, http://unescoguatemala.org/wp-content/uploads/2015/11/Modelo-deabordaje.pdf. 21 Some municipal administrations have municipal police officers, but they are responsible for protecting public spaces and enforcing local municipal ordinances. They cannot arrest and indict individuals who are suspected or who have committed crimes. 22 Both Guatemala and El Salvador have implemented decentralization policies, but municipalities remain heavily dependent on financial transfers from the central government. Although resources going to municipali￾ties have increased over the years, most mayors complain the central government does not allocate sufficient resources to local governments. Consequently, they often have to use their own resources to pay for services PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 13 The problem is even more pronounced in small and rural municipalities, which typically have fewer sources of locally-generated revenue.23 Although both Guatemala and El Salvador have implement￾ed decentralization policies, municipalities remain heavily dependent on financial transfers from the central government, and these resources are generally insufficient to fulfill their responsibilities. Moreover, mayors often complain that financial transfer amounts are unpredictable and that often they are not delivered on time. Faced with scare resources, mayors lack incentives to invest in pro￾jects that have long-term returns or that cannot generate sufficient electoral support. Crime and violence prevention projects are long-term investments that take time to demonstrate results. Finally, there is still much misunderstanding among municipal stakeholders about what prevention is and what it is not. Mayors often confuse providing basic public services, like education, health, road infrastructure, public lighting, etc., with implementing a prevention strategy. While these services are important and necessary, and do have an overall impact on the incidence of crime and violence, providing these services is a government’s responsibility and a citizen’s right. Prevention, on the con￾trary, entails doing something in addition to providing basic services by identifying the main drivers of crime and violence and addressing them through specific programmatic interventions. Similarly, mayors often confuse organizing occasional sports or other recreational events, which citizens typi￾cally enjoy and support, with prevention interventions. Prevention interventions are strategic; they are designed to mitigate or contain identifiable risk factors in a particular community. In the end, lack of capacity, lack of understanding of prevention interventions, lack of funding and insufficient transfers from the central government, or simply lack of motivation to invest in crime and violence prevention activities remain powerful obstacles to empowering municipal governments to become more active partners in reducing and mitigating the risks of crime and violence in their municipalities. V. FINDINGS 1. EFFECTIVENESS OF THE AMUPREV PROGRAM IN GUATEMALA AND EL SAL￾VADOR A. TECHNICAL ASSISTANCE AMUPREV began to work in El Salvador in 2009 and in Guatemala in 2013. In El Salvador, AMUPREV provided technical assistance to five municipalities: Sonsonate and Nahuizalco in the Department of Sonsonate and San Jose Guayabal, Suchitoto and San Bartolomé Perulapía in the Department of Cuscatlán. In Guatemala, AMUPREV provided technical assistance to six municipali￾ties: Santa Catarina Pinula, Mixco, and Palencia in the Department of Guatemala, and Cobán, Tac Tic, and San Juan Chamelco in the Department of Alta Verapaz. With the exception of the munici￾pality of Nahuizalco, where the program worked for a period of four years, AMUPREV worked two years in each municipality. By December 2015, AMUPREV concluded its activities in Guatemala and El Salvador. AMUPREV provided training and technical assistance to members of the municipal crime and vio￾lence prevention committees. In each country, AMUPREV had a technical director who visited each that should be paid by the central government such as school and hospital supplies, teacher and doctors’ salaries, police cars and ambulances. See Vicente Fretes and Teresa Ter-Minassian, eds., “Decentralizing Revenew in Latin America: Why and How,” Inter American Development Bank, 2015, https://publications.iadb.org/bitstream/handle/11319/6883/Decentralizing_Revenue_in_Latin_America.pdf?sequ ence=. 23 In the case of El Salvador, municipal governments are further restricted due to the absence of property taxes, which in most municipalities around the world, are the main source of locally-generated revenue. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 14 municipality on a weekly basis. AMUPREV’s chief of party visited each municipality at least once a month. In addition to working with municipal officials, the chief of party also offered technical assis￾tance to members of the municipal associations, the Asociación Nacional de Municipalidades (ANAM) and the Asociación Guatemalteca de Alcaldes y Autoridades Indígenas (AGAAI) in Gua￾temala and Corporación de Municipalidades de la República de El Salvador (COMURES). The chief of party advised members on the role and responsibilities of municipalities in designing and imple￾menting prevention programs AMUPREV’s chief of party consulted with USAID in Guatemala and El Salvador to select the munic￾ipalities where AMUPREV was going to provide direct technical assistance. In the case of El Salva￾dor, AMUPREV initially coordinated with USAID on the selection of municipalities where the pro￾gram was going to provide assistance. In the case of two municipalities (San José Guayabal and San Bartolomé Perulapía), however, AMUPREV worked on request from the representative of PREPAZ in the region, with the knowledge of USAID/El Salvador. 24 As mentioned above, AMUPREV’s technical assistance centers on supporting and strengthening municipal entities, recognized in each country’s national plans and policies as institutions responsible for planning, designing, implementing and supervising crime and violence prevention programs and activities at the municipal level. The integration of these committees is similar in both countries: members of the committees include representatives of the municipal office of youth and women services, members of the city council who want to take an active role in the committee, representa￾tives of civil society organizations, representatives of the municipal police force and the national po￾lice, and representatives of national crime prevention entities who work at the department level. The committees are typically directed by the mayor. In El Salvador, in three out of five municipalities where AMUPREV worked—Nahuizalco, San Jose Guayabal, and San Bartolomé Perulapía—the program assisted municipal governments to establish their municipal violence prevention commissions. In these municipalities, AMUPREV worked with the commissions to elaborate crime and violence diagnostics and formulate prevention action plans. It also supported the implementation of a few activities through a small grants program. In Suchitoto and Sonsonate, by contrast, according to AMUPREV’s technical director in El Salvador, AMUPREV was not able to work as effectively with the municipal violence prevention committees because the United Nations Development Programme (UNDP) and Creative Associates were providing similar technical assistance services in these municipalities.25 In these two municipalities, AMUPREV funded activities through small grants and invited municipal stakeholders to travel to U.S. cities in its City￾Link exchange program (see below for further details). In the case of Guatemala, AMUPREV assisted three municipalities (Mixco, Palencia, and Cobán) in establishing their municipal committees of violence prevention. In the other three municipalities, these committees had already been established with support of PREPAZ and/or UNDP. In all cases, however, AMUPREV was able to work directly with members of the COMUPRE, providing tech￾nical assistance and training, supporting them in the elaboration of the diagnostic of violence, and assisting in the design of municipal prevention action plans. In Guatemala, both the relevant USAID/Guatemala official and AMUPREV’s technical director recognized that AMUPREV coordi￾nated effectively with other bilateral programs funded by USAID. 24 Interview with USAID/El Salvador DG official, May 7, 2016; Interview with PREPAZ delegate to Cuscatlan, San José Guayabal, May 5, 2016. 25 Interview with AMUPREV’s technical director, El Salvador, May 2, 2016. In Sonsonate, UNDP had been working in the municipality since 2009 and assisted in the establishment of the CMPV. In Suchitoto, another USAID-funded program implemented by Creative began to provide assistance in 2012 and supported the establishment of the CMPV and its action plan. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 15 Municipal prevention action plans detailed activities and identified entities responsible for their im￾plementation, but for the most part, they lacked indicators to track and analyze results. Moreover, the few action plans that identified indicators only included output level indicators. Virtually all members of the municipal commissions interviewed in El Salvador and Guatemala acknowledged they lack the financial resources to track and monitor performance of their output indicators and also lack the technical capacity to identify and measure higher-level outcome indicators.26 For the most part, they recognized that AMUPREV played a critical role in helping municipal committees draft their action plans, but that the program was too short to give them sufficient time to develop and track performance indicators.27 Through a small grants program, AMUPREV purchased equipment for the municipal committees (including video cameras, projectors, sound equipment, and artistic material for youth outreach cen￾ters, computers, tables, chairs and desks) and was able to fund activities planned by the municipal prevention committee. In some cases, these activities replicated experiences learned in visits to U.S. cities (see below). Some examples include funding tuition for students in mentorship programs, supporting information campaigns, organizing a youth congress, and supporting training activities with parents. Funding for these activities was limited, ranging from $3,000 to a maximum of $14,000 USD in each municipality. In a few cases, however, AMUPREV funded activities that were not clearly related to prevention. For example, in Sonsonate, Suchitoto, and Nahuizalco, AMUPREV purchased surveillance cameras for each municipality. These cameras are typically used by police officers to identify and arrest crim￾inals. As a former director of PREPAZ in El Salvador said, “people like to see repression. They be￾lieve the municipality is doing something concrete to combat crime.”28 Although mayors and police officers like to display cameras throughout the city, they are only useful to the extent that they are used, properly maintained, and supported by consequent police action. In Sonsonate, according to members of the municipal prevention committee, AMUPREV purchased 14 surveillance cameras.29 However, eight of them are broken and of the remaining six, only one works properly.30 Similar stories were heard in other municipalities. Moreover, in the majority of municipalities, police officers complain they lack sufficient patrol cars, equipment and personnel to do their jobs. Similarly, in some municipalities AMUPREV funded activities that are not consistent with the munic￾ipalities’ prevention action plans, were not targeted to at-risk individuals, and/or were not likely to be sustainable because the municipality did not have the resources to continue to fund them. Ex￾amples include painting houses in a neighborhood in Mixco31, purchasing sewing machines for a 26 In Santa Catarina Pinula (Guatemala), members of the municipal prevention committee said they received technical assistance to develop indicators for measuring results of violence prevention activities, but they had not actually used these indicators to monitor or measure results of these activities. 27 This information was heard in interviews in Mixco, Tac Tic, San Juan Chamelco, Santa Catarina Pinula in Guatemala, and in Nahuizalco, San José Guayabal and San Bartolomé Perulapía in El Salvador. In Sonsonate, the DI team was able to review the municipal committee action plan, which was well structured and identified output level indicators for each activity. This plan, however, was developed with assistance of UNDP. 28 Interview with former Director of PREPAZ. San Salvador, May 2, 2016. 29 Members of the municipal prevention committee may not be able to distinguish how many cameras AMUPREV purchased and how many were purchased by other donors, like UNDP. In an interview with AMUPREV’s Chief of Party, May 31, 2016, he stated that AMUPREV only purchased two surveillance cameras. 30 Interview with members of the municipal committee, Sonsonate, May 3, 2016; Interview with the Director of the national police, Sonsonate, May 3, 2016. 31 Interview with member of Mixco’s former municipal prevention committee, Mixco, April 26, 2016. Accord￾ing to the member, AMUPREV organized an activity with children to paint houses in an insecure PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 16 clothing workshop in San José Perulapía, and providing materials for a jewelry workshop in Suchito￾to. According to interviews with mayors in both cities, these activities typically attracted older women.32 AMUPREV designed toolkits and guides that can be used by any municipal administration to estab￾lish municipal prevention committees, regulate their operations and plan their activities. The toolkits and guides are posted on AMUPREV’s website and are publicly available for consultation. In El Sal￾vador and Guatemala, AMUPREV’s technical directors used these guides and toolkits to provide technical assistance and training to members of the municipal prevention committees. In interviews with the DI team, representatives of national crime prevention institutions at the department level in both countries—PREPAZ in El Salvador and UPVC in Guatemala—acknowledged they found these toolkits practical and valuable and said they used them in their work with municipal stake￾holders. In Guatemala, however, while representatives of the UPVC at the national level said they knew about the AMUPREV program, they did not know the program had developed toolkits and guides for municipal committees and that these resources were available on the AMUPREV’s web￾site. When asked what resources they use to assist municipal governments in establishing their committees, they mentioned the guide that was developed with support of UNESCO as part of Guatemala’s National Policy of crime and violence prevention.33 Although AMUPREV was not the first international donor program to emphasize the centrality of municipal governments in crime prevention, it contributed to increasing awareness among municipal stakeholders on the central role municipal governments play in crime and violence prevention. AMUPREV motivated and convinced mayors and members of the municipal prevention committees that they could design and implement activities to reduce the risks of violence in their neighbor￾hoods, even with limited financial resources. Although most municipal stakeholders recognize that municipalities lack sufficient resources to respond to citizens’ needs and demands, they learned that “we can do a lot with little resources.”34 Virtually all beneficiaries of AMUPREV activities interviewed for this evaluation recognized that the program helped them organize and systematize their activities, gain a better understanding of pre￾vention, and gain exposure to innovative ideas that could be introduced or replicated in their mu￾nicipalities. The main complaint, which was consistently articulated in both countries by all inter￾viewees was that the program was too short to make significant inroads and too small to demon￾strate significant and sustainable results. The comments from beneficiaries’ perceptions in El Salva￾dor and Guatemala illustrate these points more eloquently: “AMUPREV strengthened the work we were already doing in the municipality and highly motivated members of the municipal prevention committee. Although we do not have sufficient resources, we have enough resources to invest in prevention. It all is a matter of political will.”35 “AMUPREV provided invaluable technical support for the establishment of the municipal prevention committee and helped to empower its members.”36 neighborhood, but “the houses were never maintained and the neighborhood looks as dirty today as it did before.” 32 Interviews with mayors of Suchitoto and San Jose Perulapia, May 3-4, 2016. 33 Interview with representatives of the Unidad Ejecutora, UPVC, Guatemala City, April 25, 2016. See “Mode￾lo de Abordaje, Unidad para la Prevención Comunitaria de Violencia (UPVC)”, Viceministerio de la Preven￾ción del Delito, Ministerio de Gobernación, http://unescoguatemala.org/wp-content/uploads/2015/11/Modelo￾deabordaje.pdf. This methodological guide for establishing municipal committees of violence prevention and toolkits to design prevention activities in UNESCO is similar to AMUPREV’s Toolkit. 34 Interview with a member of the municipal prevention committee in Nahuizalco, May 4, 2016. 35 Interview with former president of the municipal prevention committee in Coban, April 28, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 17 “Having AMUPREV come to our municipality, which does not receive any donor funding, was a blessing. They taught us how to move forward.”37 “AMUPREV helped us organize many activities that we were already doing, but in a disorderly fash￾ion. It helped us realize that we were already working on prevention but we did not know it.”38 In a region where most mayors lack the political will or understanding to support prevention activi￾ties, “AMUPREV’s most important achievement was political. With limited resources, it convinced mayors to support prevention activities.”39 B. PARTNERSHIPS: CITYLINKS PROGRAM As part of its efforts to increase awareness amongst municipal stakeholders on the importance of supporting prevention activities, AMUPREV designed a program to foster municipal partnerships between U.S. and Central American cities. The main purpose of these partnerships was to share information, disseminate best practices and draw lessons learned on crime and violence prevention at the municipal level. In El Salvador and Guatemala, AMUPREV organized learning exchange trips where mayors, members of municipal violence prevention committees, police officers, businessmen, and officials of municipal associations had an opportunity to interact with their counterparts, visit different sites and learn from one another. Representatives from Nahuizalco and Sonsonate visited Santa Ana, California; representatives from San Bartolomé Perulapía, San José Guayabal and Suchi￾toto visited Alexandria, Virginia. In Guatemala, AMUPREV organized similar visits to Santa Ana and Stockton, California. Representatives from these cities in the United States reciprocated with visits to beneficiary munici￾palities in El Salvador and Guatemala.40 The learning exchanges lasted one week and included visits to several municipal initiatives such as mentoring programs, gang reduction programs, youth outreach centers, and victim-attention cen￾ters. Participants also visited police officers who support community-oriented policing in their mu￾nicipalities and crime control centers, where they receive and process reports on crime and vio￾lence. In interviews with the DI team, all beneficiaries of AMUPREV learning exchanges agreed that this was the single most interesting and inspiring activity of the AMUPREV program. They acknowledged the visits exposed them to innovative experiences, motivated them and encouraged them to do more to reduce the risks of violence and crime in their municipalities. Despite the differences in resources and capacity between the U.S. and Central America, beneficiaries of these exchanges realized that some experiences could be easily adapted to their municipalities. The mayor of San José Guayabal reflected, “AMUPREV helped me to realize it is possible to reduce the risks of crime and violence in my municipality. AMUPREV made me think differently.”41 One experience that clearly impressed many participants was visiting victim attention centers which offer legal, medical, and psychological services to victims of domestic violence. While domestic vio- 36 Interview with former mayor of Palencia, April 27, 2016. 37 Interview with members of the municipal committee in San José Guayabal, May 5, 2016. 38 Interview with former members of the municipal prevention committee in Santa Catarina Pinula, April 27, 2016. 39 Interview with DG official at USAID/Guatemala, April 25, 2016. 40 Salvadorian and Guatemalan officials traveled to the United States once; represnetatives from the United Stsates traveled to their respective beneficiary municipalities in Guatemala and El Salvador twice. The DI team did not interview local government representatives from the United States. 41 Interview with Mayor of San José Guayabal, May 5, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 18 lence is widespread throughout Guatemala and El Salvador, most people interviewed by the team perceived these centers to be too costly and sophisticated, and concluded that therefore, these centers could not be replicable in their municipalities. In these countries, victims of domestic vio￾lence remain largely unattended, even though it has been demonstrated that children who experi￾ence domestic violence in their homes tend to become perpetrators of violence when they grow up. Of the eleven municipalities the participated in AMUPREV in Guatemala and El Salvador, five (Pa￾lencia and Cobán in Guatemala and Nahuizalco, San Bartolomé Perulapía and San José Guayabal in El Salvador) replicated or adapted an activity observed in the U.S. during the exchange program. In Palencia, the visit served to convince and empower the mayor to replicate the mentoring and school reinforcement program he saw in Santa Ana. Through a small grants program, AMUPREV provided equipment for two schools and the mayor implemented a program to assist children who were failing in school. One former Palencia committee member said, “The visit to Santa Ana helped us to open our minds and learn how to work with youth differently.” 42 In the case of Cobán, the visit to Stockton, California encouraged the mayor to create a prevention office with a specific budget inside the municipality. In Nahuizalco, San Bartolomé Perulapía and San José Guayabal, the mayors replicated a mentoring and tutoring program for young students they observed in Santa Ana, California and Alexandria, Virginia, and supported a youth outreach center that offers sports and artistic activities. In the rest of municipalities, the mayors and/or members of the municipal committees stated that the experiences they saw in the United States, while motivating and interesting, were either too costly and/or too complex to be replicable in their municipalities. For example, in Tac Tic, members of the municipal prevention committee who visited Stockton, California stated that in Guatemala, they would never be able to replicate the experiences they saw in Stockton.43 The mayor of Suchi￾toto voiced a similar opinion, stating that they were light years away from being able to replicate what they saw in Alexandria.44 The difference between municipalities that managed to replicate experiences and those that did not can be partly explained by the motivation and engagement of the mayors. The five municipalities that replicated experiences vary in size and resources. Cobán, for example, is a large municipality with more financial and human resources than Palencia or San José Guayabal. While finding the fi￾nancial resources to invest in prevention is important, political will is the single most determinant factor behind municipal commitment to supporting prevention activities. It is unclear whether AMUPREV contributed to increasing the political will of mayors in Nahuizalco, Cobán, San Bar￾tolomé Perulapía, San José Guayabal, and Palencia or whether, because these mayors already had the political will to engage in prevention activities, they were the ones who benefitted most from the AMUPREV program. What is clear is that the exchange program offered AMUPREV a powerful incentive to engage mu￾nicipal stakeholders who might have not been otherwise open or supportive to partner with AMUPREV given the program’s extremely limited financial resources to fund activities, especially when compared to other donor programs in the region. 42 Interview with a member of the former municipal violence prevention committee in Palencia, April 27, 2016. 43 Interview with former members of the municipal prevention committee in Tac Tic, April 29, 2016. 44 Interview with Mayor of Suchitoto, May 6, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 19 C. THE AMUPREV NETWORK [RED AMUPREV] According to AMUPREV’s chief of party, establishing a network of municipal associations for crime prevention—the RED AMUPREV—was an effort to scale up and maximize the results of AMUPREV’s technical assistance and exchange programs, which only benefitted a few municipalities in each country. AMUPREV’s ultimate objective was not necessarily to improve specific municipali￾ties’ capacity to plan and implement prevention programs, but rather to “improve municipal leader￾ship.”45 To that end, the program needed to go beyond a specific set of municipal governments and engage municipal associations in different countries. The RED AMUPREV disseminates news and information about municipal prevention efforts, facili￾tates the exchange of best practices, and promotes greater involvement of municipal governments in prevention activities. To anchor the network in a regional institution, and thus ensure its sustaina￾bility after the end of the program, ICMA approached SICA, the Sistema de Integración Cen￾troamericana, a Central American multinational organization based in El Salvador, to convene mu￾nicipal associations in the region and establish the network. The partnership with SICA, however, did not prosper because not all member countries agree to participate in an initiative funded by USAID. Eventually, ICMA created a virtual network of municipal associations and sought the sup￾port of a newly established confederation of municipal associations of Central America and the Car￾ibbean (CAMCAYCA) to house and maintain it. CAMCAYCA’s presidency rotates among its members and is currently at the ANAM in Guatemala. The RED AMUPREV has a website (www.amuprev.org) which contains news and information on crime prevention, including the toolkits and guides developed by the AMUPREV program, examples of national prevention plans of different countries, municipal crime and violence prevention policies and action plans, success stories and best practices, and results of AMUPREV’s program. The web￾site also advertises upcoming training events or conferences in the region related to crime and vio￾lence prevention. AMUPREV’s communication direction maintains the website on a daily basis from San Salvador. Although municipal associations and governments are encouraged to post news on the website, AMUPREV’s communication director posts most of the information uploaded on the website.46 According to AMUPREV’s quarterlies, the number of visitors to AMUPREV’s website has increased substantially from 1,639 in 2012 to 5,067 in 2015. 75 percent of the visits search news posted on the website. Half of the visitors are from Central American countries.47 45 Interview with AMUPREV’s chief of party, April 15, 2016. 46 When AMUPREV was still active in Guatemala and El Salvador, the program’s technical directors were responsible for posting news and maintaining the website. But after the program closed its activities in these countries, this responsibility has fallen entirely on AMUPREV’s communications director. AMUPREV’s quarter￾lies report that the percentage of postings made directly by municipalities and municipal associations have increased, but this information is not broken down by country and the program does not track in its quarter￾lies or in its indicators the number of postings made by municipalities and municipal associations (that is, inde￾pendent of AMUPREV’s staff). 47 See AMUPREV’s quarterly reports. AMUPREV began to track the number of visitors to the website in 2012. In 2013 the number of visitors dropped to 1,639, but then the number has increased consistently. The quarterlies provide no explanation for this drop. In interviews with AMUPREV’s officials in El Salvador and Guatemala the DI team learned that most of the postings on the website came from AMUPREV’s staff. Once the project left Guatemala, the number of postings decreased significantly. In El Salvador, the number of post￾ings continues because AMUPREV’s communications director is based in El Salvador and he continues to post news from El Salvador. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 20 The majority of beneficiaries of AMUPREV’s program interviewed by the DI team said they were familiar with the website and occasionally read its contents. They thought the most relevant infor￾mation they found on the website was news on crime and violence prevention experiences from other countries. They also acknowledged they receive the weekly bulletin AMUPREV sends every Friday night to more than 2,500 individuals and organizations in its contact list. Only a few people, however, told us they actually read the bulletin. In Guatemala, AMUPREV (in particular, AMUPREV’s chief of party) managed to work well with leaders of the Asociación Nacional de Municipalidades de la República de Guatemala (ANAM) and Asociación Guatemalteca de Alcaldes y Autoridades Indígenas (AGAAI). In both cases, the munici￾pal associations contributed by finding and posting news related to crime prevention on their web￾sites and sharing them with AMUPREV. AMUPREV organized a training workshop to assist online communicators in identifying, writing and posting news on crime prevention on AMUPREV’s web￾site. ANAM hired one of those individuals as its communication director, tasked with finding and posting news on the AMUPREV website. In 2013 and 2014, AMUPREV’s chief of party was invited to be a guest speaker at ANAM’s annual assembly. The situation, however, changed dramatically after ANAM and AGAAI replaced their presidents in 2016.48 Today, there is no evidence that the new leadership of either of the municipal associations in Guatemala is interested or committed to supporting crime prevention initiatives.49 ANAM’s new communication director recognized that the municipal association is mostly interested in increasing the volume of financial resources transferred to municipalities. Despite the severity of the problem of crime and violence in the country, preven￾tion is no longer one of the municipal associations’ top priorities. Although both municipal associations signed memorandums of understanding with AMUPREV to post news and disseminate the AMUPREV website among its members, they have, for the most part, remained passive. ANAM’s website contains virtually no information on crime prevention, and the AMUPREV link on the website is no longer active. ANAM’s new communication director acknowledged she did not know enough about AMUPREV objectives and activities and told us she could not find the information in ANAM’s records and files because “the previous ANAM admin￾istration removed all the information from the association’s files when they left.” She also recog￾nized that she has not been able to send news on Guatemala’s municipal prevention initiatives to AMUPREV’s communications director. She said, “AMUPREV’s communication director is in fact sending me news about prevention efforts in Guatemala’s municipalities. It should be the other way around.”50 The lack of buy-in from ANAM raises serious questions about the sustainability of AMUPREV’s network, which now depends on ANAM’s leadership as the president of CAM￾CAYCA. In El Salvador, the relationship between AMUPREV and the municipal association, the Corporación de Municipalidades de la República de El Salvador (COMURES) was difficult from the beginning. According to AMUPREV’s communication director, COMURES refused to collaborate actively with the program, even when they signed a memorandum of understanding. The DI team was unable to interview representatives from COMURES.51 In theory, municipal associations should be the owners of the AMUPREV network and become its principal stakeholders, users and promoters. This becomes particularly important once AMUPREV 48 Presidents of municipal association serve a period of two years. 49 Over the past months, ANAM’s leadership has change its approach and has demonstrated greater interest in collaborating with AMUPREV’s program. 50 Interview ANAM’s communication director, Guatemala, April 25, 2016. 51 After more than ten calls to COMURES communication specialists, who participated in AMUPREV’s trainings, they refused to schedule an interview with DI. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 21 has concluded its operations in the country. Unfortunately, in Guatemala and El Salvador, no evi￾dence exists that municipal associations have a stake in the maintenance and/or survival of the net￾work.52 As we elaborate in more detail below, virtually none of the online communicators who re￾ceived training from AMUPREV in El Salvador or Guatemala are currently writing and posting news on the AMUPREV’s website. As the former communication director of AGAAI in Guatemala, who remains committed to the AMUPREV program and continues to send information to AMUPREV’s communication director in San Salvador, lamented, “If the communications director leaves the pro￾gram, the AMUPREV network will disintegrate.”53 2. IMPLEMENTATION CHALLENGES Planning and implementing crime and violence prevention programs at the municipal level requires the collaboration of multiple sectors, including education, health, municipal services directed to women and youth, water and sanitation, and public works, among others. Too often, however, there are weak incentives and mechanisms for inter-sectoral coordination, especially at the local level. For this reason, AMUPREV worked directly with local municipal violence prevention commit￾tees, which are officially recognized as the institutional mechanisms for coordination and collabora￾tion for the design and implementation of violence and crime prevention programs at the local lev￾el. AMUPREV confronted several implementation challenges that for analytical purposes can be grouped into three broad categories: (a) administrative, (b) political, and (c) conceptual. A. ADMINISTRATIVE CHALLENGES As noted earlier, AMUPREV made significant progress in the targeted municipalities during their presence in Guatemala and El Salvador. As we mentioned above, AMUPREV was able to do a lot with limited resources, but financial limitations seriously constrained AMUPREV’s ability to accom￾plish its overall goals, especially considering the magnitude of the violence epidemic in these coun￾tries and the presence of other, much larger donor-funded programs. Because of its limited budget, amounting to an average of $500,000 per year,54 AMUPREV was only able to provide technical as￾sistance to a few municipalities and fund very small projects through its small grants program. In small municipalities, particularly those where donors have a limited presence, these grants, albeit small, did contribute to facilitate the implementation of violence prevention activities. The sustaina￾bility of these activities, however, remains in question, given that these municipalities have fewer resources to consolidate prevention programs and continue to fund prevention activities. In the rest of the municipalities, the outcomes of a small program like AMUPREV are difficult to distinguish and tend to be diluted by the presence of other, much larger donor-funded programs.55 Indeed, in many municipalities, beneficiaries of AMUPREV’s activities could not identify which activities were imple￾mented by AMUPREV and which were implemented by RTI and/or Creative. For them, what mat￾tered was that they were all funded by USAID. In some cases, AMUPREV implemented too many “ad-hoc” activities not related to the main objec￾tive of the program and spent resources on equipment that is not related to or conducive to the prevention of crime and violence. For example, in Sonsonate and Suchitoto in El Salvador, AMUPREV purchased video-surveillance cameras. In San Bartolomé Perulapia in El Salvador, 52 It is not surprising that some of the most active members of the AMUPREV network are the municipal associations in Honduras and Dominican Republic, where AMUPREV is currently working. According to AMUPREV’s communication director, the municipal associations in Panama and Costa Rica are also actively engaged in the network. DI could not corroborate this information. 53 Interview with AGAAI former communication director, Guatemala, April 25, 2016. 54 Interview with AMUPREV’s chief of party, April 16, 2016. 55 Interview with DG officer, USAID/El Salvador, May 7, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 22 AMUPREV’s resources helped purchase sewing machines and materials to make costume jewelry (“bisutería”). Given budgetary limitations, resources could have been used more strategically. Funding ad-hoc activities ultimately depleted available resources, impairing the AMUPREV team’s ability to concen￾trate on a set of other short-term, high-impact activities and to spend more time refining municipal violence prevention plans, especially by identifying measurable indicators to track and monitor their implementation. Funding limitations were compounded by insecurity of total funding available for the program. AMUPREV received several cost extensions that, while welcome, were unpredictable. According to AMUPREV’s management team, there was no continuity in the allocation of resources by USAID, which constrained the program and prevented it from becoming more strategic and planning activi￾ties on a longer term basis. This may explain some of the erratic expenses on non-specific violence prevention equipment such as video-surveillance cameras and sewing machines. Another administrative challenge was the limited period of time AMUPREV was able to spend working in each municipality. With the exception of Nahuizalco in El Salvador, where AMUPREV worked for a period of four years, in the rest of municipalities, AMUPREV worked only two years. Promoting improved municipal leadership and supporting comprehensive municipal-based violence prevention strategies and programs requires a long-term approach to consolidate, take root, and demonstrate results. It is not surprising that Nahuizalco is AMUPREV’s most successful case, where there is a consolidated, well-functioning local violence prevention committee; the mayor is fully committed to supporting prevention activities; the municipality has improved its rates and fees to raise additional revenue to fund prevention programs; and the municipality has a supportive com￾munity policing force which has contributed to strengthening violence prevention activities. Virtually everyone interviewed for this evaluation agreed that AMUPREV did not have sufficient time to accomplish its goals and expected results. Interviewees stated that these types of program require longer time frames to consolidate the results and ensure greater sustainability. Even though all mayors welcomed AMUPREV’s work, knowledge and enthusiasm, most recognized that activities AMUPREV supported were too short, and that they are difficult to sustain due to lack of resources and insufficient technical capacity. Another administrative challenge related to fund limitations was the program’s inability to establish a permanent office in each country. AMUPREV’s chief of party “managed the program from the air￾ports.”56 In each country, technical directors worked from their homes and visited municipalities on a weekly basis. Due to the regional nature of this program, decision-making authority was central￾ized in the program’s director. But the centralization of management limited the agility of technical directors to operate, as they did not have authority over spending decisions. According to different project beneficiaries, the centralization of management added another layer of bureaucracy in the day to day management of an already small program because technical directors needed to request approval from AMUPREV’s director for every single purchase of materials and equipment. Finally, USAID officials in Guatemala and El Salvador said that a regional program like AMUPREV should have been managed from the regional office in El Salvador, and not in Washington, D.C. This 56 Interview with DG officer, USAID/Guatemala, April 25, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 23 would have allowed USAID officials in both Guatemala and El Salvador to have closer supervision over program implementation and to ensure better coordination with other bilateral programs.57 B. POLITICAL CHALLENGES One of the most serious challenges for the implementation of municipal crime and violence preven￾tion programs is the weakness or absence of mayors’ political will to support and embrace these programs. Although many mayors say they are interested and committed to violence and crime prevention, a combination of lack of resources, limited mandates, weak technical capacities, high levels of crime and violence in their municipalities, and the pressure to show quick results eventually tends to erode their interest and commitment to violence prevention. While most mayors in the beneficiary municipalities ultimately embraced the importance of violence prevention, many were initially reluctant. In Mixco, Guatemala, one of the most violent municipalities in the country, it is unclear whether the mayor was ever effectively convinced of the importance of prevention, but he did not object to AMUPREV’s support to the local prevention committee58. In the rest of the mu￾nicipalities, AMUPREV was effective in obtaining the mayors’ buy-in, but without AMUPREV’s pres￾ence, it is unclear if this commitment will be sustained. For example, in San Bartolomé Perulapía in El Salvador, the mayor embraced the AMUPREV program, but his commitment to invest in preven￾tion activities is questionable. The mayor stated: “We have a problem of cancer in the municipality, and that is not cured with aspirins. I believe in prevention, but implemented with a firm and heavy hand.”59 In the case of Guatemala, however, where municipal administrations changed after the 2015 elec￾tions, the important political work AMUPREV did in convincing the mayors to support prevention programs has been lost, as new mayors are not interested in continuing to support any of the pro￾grams introduced by the previous administration. None of the new mayors and municipal officials interviewed by the team demonstrated any interest in or knowledge of crime and violence preven￾tion. As the new mayor of Palencia said, “municipal governments do not have the resources to work on citizen security. Decentralization has transferred responsibilities to municipalities, but not resources. Yet, citizens demand greater results, and they want more participation in public affairs. Guatemala, like the rest of Central American countries, is not ready for democracy.”60 Another political challenge affecting the program’s implementation was the lack of buy-in from mu￾nicipal associations. In El Salvador, AMUPREV did not succeed in establishing a collaborative rela￾tionship with COMURES, the municipal association. As AMUPREV’s communications director rec￾ognized, COMURES has been the most hostile and unwilling municipal association in the region to collaborate with AMUPREV. They are not making any effort to integrate more effectively into the network.61 In Guatemala, ANAM and AGAAI were more welcoming and collaborated with AMUPREV more closely. They participated in the AMUPREV network and assisted AMUPREV in approaching the municipalities where it worked. But with the change of administration in both or￾ganizations that commitment has evaporated, especially now that AMUPREV is no longer present in 57 According to officials in USAID/Washington, the program was managed out of Washington as the result of a mutual agreement between Washington and USAID/El Salvador because at the time (2009), El Salvador’s regional platform did not have the capacity to manage any such programs. 58 This mayor was former President Molina’s son and is now under investigation for corruption. According to the USAID/DG officer interviewed in Guatemala, USAID/Guatemala requested from AMUPREV to work in Cobán because this was one of the municipalities where they were already working at the community level (RTI and later Mercy Corps) but had difficulty interacting with the mayor. 59 Interview with Mayor of San Bartolomé Perulapía, May 5, 2016. 60 Interview with Mayor of Palencia, April 27, 2016. 61 Interview with AMUPREV’s communication director, May 2, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 24 the country. As discussed above, today ANAM has other priorities unrelated to crime and violence prevention. Municipal associations have no incentives to collaborate with a regional program like AMUPREV, which generates platforms for knowledge dissemination but offers no significant financial resources to municipal associations or to their members. It is no small achievement that AMUPREV succeed￾ed in establishing the virtual network (RED AMUPREV) and has at least formally convened all mu￾nicipal associations in the region. But this success is more due to AMUPREV’s director’s technical abilities and charismatic personality than to the creation of a feasible institutional infrastructure that will make this network sustainable in the long term. C. CONCEPTUAL CHALLENGES In countries facing high levels of crime and violence, it is often difficult to persuade government offi￾cials to focus on prevention. It has been demonstrated that relying solely on law enforcement poli￾cies is not only ineffective, but can in fact be counterproductive. Despite this, local government offi￾cials who suffer the consequences of crime and violence in their territories often prefer repressive measures to combat the high levels of crime they confront. In the process, the concept of preven￾tion is either confused with law enforcement—perceived as deterrence—or with other activities local governments typically fund in their municipalities, such as building roads, paving streets, in￾stalling street lamps, or organizing sports or musical events. AMUPREV had to work hard to educate mayors and their city councils on the meaning and signifi￾cance of prevention and to ensure that activities the program supports were not used for political or electoral purposes. In Sonsonate, for example, AMUPREV supported an informational campaign to support prevention activities, but realized that the mayor was using the campaign for electoral purposes and subsequently terminated this activity.62 Another related conceptual challenge was AMUPREV’s failure to identify appropriate indicators to measure and evaluate results of crime and violence prevention programs. Although AMUPREV im￾plemented many activities, it is difficult to assess how activities were linked to expected results and to evaluate whether activities were targeted to the right beneficiaries, whether they were appropri￾ate and relevant, and whether they contributed to reducing the risks of crime and violence in the community. A lack of appropriate outcome indicators means that projects can implement activities without a clear strategic purpose. Moreover, the presence of many international donors funding prevention programs in these munic￾ipalities has led municipal stakeholders to embrace prevention as a catch-all phrase. In many munici￾palities the DI team visited, municipal stakeholders told us they were committed (in principle) to prevention, to the point where anything they did was referred to as prevention. In Sonsonate, for example, where UNDP, USAID, and JICA have implemented prevention programs, the mayor had difficulty explaining why some of the activities he supports were related to prevention. He said that as part of his prevention activities he donated a saddle for a horse in a community, and that en￾couraged “family unity,” as the family all came to see the saddle.63 The major risk of not having a clear conceptual understanding of what prevention is and what it is not is that mayors may use do￾nor-funded projects to finance activities that have clientelistic and electoral purposes but are legiti￾mized as prevention interventions. 62 Interview with AMUPREV’s chief of party, April 15, 2016. 63 Interview with Mayor of Sonsonate, May 3, 2016. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 25 3. COORDINATION WITH OTHER BILATERAL PROGRAMS Violence prevention programs and activities are abundant in both Guatemala and El Salvador, given the seriousness of the problem. To date, USAID has financed several programs related to violence prevention and community policing implemented by different contractors. Therefore, coordination between different programs is critical for (1) maximizing outcomes; (2) avoiding duplication of activ￾ities; and (3) avoiding unnecessary financial and human resource costs for both donors and benefi￾ciary municipalities. To this end, the experience of coordination between AMUPREV and other USAID-financed vio￾lence prevention programs was mixed and municipality-specific. In the case of Guatemala, AMUPREV was able to coordinate well with other crime and violence prevention programs funded by USAID and implemented by RTI and Mercy Corps because the objectives of these programs were complementary. Programs managed by RTI and Mercy Corps focused on violence prevention at the community level while AMUPREV worked with the municipal government. According to a USAID/Guatemala official, AMUPREV’s chief of party worked with USAID to select the municipali￾ties where AMUPREV was going to intervene. USAID selected municipalities where mayors were not fully on board with prevention interventions and where, consequently, AMUPREV could be more effective in increasing mayors’ political will.64 USAID/Guatemala saw AMUPREV’s program as an opportunity to engage mayors and other municipal officials directly, educate them on the im￾portance of prevention, and motivate them to support prevention interventions. AMUPREV’s City￾Links program was an effective incentive to accomplish this objective. According to the USAID offi￾cial, AMUPREV was successful in articulating the message of prevention and increasing municipal officials’ support for prevention interventions. In El Salvador, AMUPREV was not able to coordinate as well with USAID as they were able to in Guatemala. Although AMUPREV’s chief of party consulted with USAID/El Salvador on a regular basis, USAID/El Salvador did not select all the municipalities where AMUPREV intervened. Accord￾ing to the official from USAID/El Salvador interviewed, he did not know how AMUPREV selected the five municipalities where it worked. As the project was managed in Washington, D.C. and not from the El Salvador office, he did not have authority over AMUPREV’s decisions. In El Salvador, coordination with other USAID-funded projects depended on the particular mandates of each pro￾gram. While AMUPREV coordinated well with a project managed by RTI in Nahuizalco, it was not able to coordinate as successfully with Creative Associates in Suchitoto.65 The main reason was that RTI, as in Guatemala, supported activities at the community level, while Creative’s project worked with municipal governments. In Suchitoto, AMUPREV would have duplicated efforts, so USAID re￾quested that AMUPREV not work in this municipality until Creative concluded its activities.66 In the case of Sonsonate, where UNDP had a large crime and violence prevention project, AMUPREV also had difficulty supporting the municipal violence prevention committee. Although USAID did not support projects in this municipality, AMUPREV’s program was too small to attract the mayor’s attention. This may be the reason why in Sonsonate, AMUPREV purchased surveillance cameras in an effort to gain traction with the municipal officials. Finally, AMUPREV’s work in San José Guayabal and San Bartolomé Perulapía came in response to a request from the PREPAZ representative in Cuscatlán. USAID/El Salvador did not participate in the selection of these municipalities and USAID does not support other projects in these municipalities. 64 Interview with, USAID/Guatemala DG officer, April 25, 2016. 65 Interview with USAID/El Salvador DG officer, May 7, 2016 66 Creative supported the municipal government to establish its municipal crime prevention committee and drafted its action plan. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 26 In several municipalities in both Guatemala and El Salvador, USAID funded community police pro￾jects implemented by Checchi Consulting. While AMUPREV actively engaged individual police offi￾cials in its CityLinks program and endorsed community policing practices, it did not need to coordi￾nate with Checchi because AMUPREV did not work directly with the police force in the municipali￾ties. In Guatemala and El Salvador, none of the police officers who participated in the CityLinks re￾main in their posts.67 4. SUSTAINABILITY The vast majority of donor-funded programs are vulnerable to problems of sustainability. Unless policymakers and mayors have the financial resources and human capacity to sustain these efforts beyond the life of the program, they are unlikely to survive. Research has demonstrated that sus￾tainability depends on the ability of donors to craft medium to long-term strategies, taking into con￾sideration the organizational and financial needs of beneficiaries.68 Although AMUPREV contributed significantly to promoting a paradigm shift in how to tackle crime and violence at the municipal level, in most municipalities it is unclear if mayors will maintain their level of commitment to prevention activities in the absence of AMUPREV. In Guatemala and El Sal￾vador, the program was too small and too short to develop a well-designed and realistic sustaina￾bility strategy. Many of the activities AMUPREV funded were short-term and driven mainly by the availability of funding. Exacting real change takes time and requires long-term agendas. Building capacity in municipal gov￾ernments that lack sufficient organizational capacity, financial resources, and human resource capaci￾ty is an important challenge for sustainability and should be examined during the design stages of the project. As noted earlier in this evaluation, AMUPREV worked, on average, two years in every municipality, except for Nahuizalco where it worked for four years to provide technical assistance and develop the municipal capacity to work on violence prevention programs. Another factor affecting sustainability of community policing activities, a practice AMUPREV en￾dorsed through its learning exchange program, is the high rotation of police officers from one mu￾nicipality to another in Guatemala and El Salvador, a practice that negatively impacts the entrench￾ment of community policing practices in the region. While there is no “silver bullet” that defines the minimum number of years needed to develop mu￾nicipal capacity, it was clear from the visits to the municipalities and interviews with beneficiaries that two years does not seem to be enough time to develop the basic organizational structure to sustain activities over time. As mentioned before, mayors generally do not have sufficient resources to sustain violence preven￾tion activities in the medium and long-term. Municipal governments do not receive specific budget￾ary transfers from the central government for violence prevention activities. In general, these activi￾ties are financed through donor-funded programs. Of all the municipalities benefitted by AMUPREV, only in one case, Nahuizalco, did the mayor take the initiative to update municipal tariffs and fees to invest additional locally-generated revenue in prevention activities. But even in this case, the addi￾tional resources are not sufficient to sustain the activities AMUPREV funded. 67 The DI team interviewed the director of the national police in every municipality where AMUPREV worked. With the exception of the director of the police in Mixco, who formerly coordinated a police post in the same municipality, police directors had recently arrived to their posts and none of them knew the AMUPREV program. 68 Lincoln Young and Jonathan Hampshire, Promoting Practical Sustainability, Canberra: Quality Assurance Group‐PIA/OPRE, The Australian Agency for International Development, 2000. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 27 Mayors that want to continue supporting AMUPREV activities have had to adjust their expectations to their budgetary realities, meaning they have had to reduce the number of activities, space them out, or eliminate some completely. For example, in Nahuizalco, the municipality is still endorsing the school reinforcement and mentorship program, but it does not have the resources to purchase gifts for the students who complete the program, which constituted an essential incentive of this pro￾gram. Similarly, in San José Guayabal, the youth outreach center continues to operate but the mu￾nicipality lacks sufficient resources to purchase materials needed for their workshops. They are still using materials purchased by AMUPREV, but these will run out in a few months. In San Bartolomé Perulapía, the mayor continues to implement the school scholarship program, but it had to drop the mentorship program for lack of funding. In addition to economic resources, political factors also affect the sustainability of AMUPREV’s pro￾gram. In Guatemala, virtually all municipal administrations were replaced after the 2015 elections.69 This negatively affects the likelihood that activities supported by AMUPREV will be sustained. As the former president of ANAM said, “There is no sustainability of violence prevention programs. If the mayor leaves, the program dies.” While electoral and political realities are outside the control of international donors, these factors should nevertheless be taken into account when planning and implementing programs. By concluding its activities in Guatemala exactly at the time new mayors came to office, AMUPREV increased the risk that the program would not be sustained. In all munic￾ipalities visited by the DI team in Guatemala, the violence prevention commissions (COMUPRE) have been temporarily suspended, awaiting new direction from the mayors’ offices. Furthermore, new mayors interviewed by the DI team stated that they did not know what AMUPREV had ac￾complished in their municipalities and complained they could not find any information on the pro￾gram. While they said they were interested in violence prevention, when asked what activities they were planning to support, they talked about building infrastructure projects, and supporting em￾ployment and education programs. The latter two areas fall outside the authority of municipal gov￾ernments. Another political factor that may affect the sustainability of AMUPREV’s program is a lack of com￾mitment from the municipal associations and the politics associated with these entities. Municipal associations like ANAM in Guatemala and COMURES in El Salvador, which represent all municipali￾ties in their countries and have leverage to influence policy making, could be critical actors for incen￾tivizing municipalities to develop violence prevention programs and activities as well as for promot￾ing a culture of prevention through social media and capacity building. In Guatemala, ANAM changed administrations shortly after AMUPREV completed its technical as￾sistance activities with the beneficiary municipalities. Interviews with the new administration revealed that no manuals and procedures were transferred from the previous administration. The new direc￾tor of communication had no knowledge of how to upload news on violence prevention on their website. Moreover, the current president of ANAM did not include violence prevention as part of ANAM’s agenda of core activities over the next two years.70 In El Salvador, the association of mu￾nicipal governments, COMURES, does not appear to be a committed partner and has limited its participation and interest in AMUPREV’s activities. If municipal associations and organizations are 69 The mayors of Tac-Tic and San Juan Chamelco were reelected, but their city councils were almost entirely replaced by new members. 70 Interview with AMUPREV’s chief of party and AMUPREV’s program director, May 31, 2016. In an interview with AMUPREV’s chief of party after the conclusion of field work, he suggested that AMUPREV continues to work with ANAM and will train its new communications director. While it may be possible to increase the interest of ANAM re-commit them to the AMUPREV program, the team did not see evidence that this is currently the case. ANAM’s agenda for the next two years does not include prevention as one of its priorities. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 28 not supportive of violence prevention programs, it will be more challenging to sustain any donor￾funded activities in the medium and long-term. The sustainability of the AMUPREV network (RED AMUPREV) is also dubious, at least in Guatema￾la and El Salvador. While the network remains active, and the number of visitors to the website has increased substantially, the website is maintained by one individual, AMUPREV’s communication director, who continues to collect news on violence prevention and upload these to the website.71 By disseminating new and innovative interventions or activities in crime and violence prevention at the municipal level, the network could play a critical role in motivating members of the network to continue working in this area, especially after the completion of the AMUPREV program. However, the lack of buy-in from the municipal associations and the over-dependence on one individual casts serious doubts on the network’s sustainability in the long term. V. CONCLUSIONS AND LESSONS LEARNED In countries facing epidemic levels of crime and violence, many citizens continue to endorse repres￾sive measures to contain and mitigate the violence, even when it has been demonstrated that law enforcement approaches are insufficient to curb and contain the violence. In recent years, academ￾ics, journalists and development practitioners have increasingly advocated for the introduction of prevention approaches to address the root causes of crime and violence. In Central America, gov￾ernment officials at the national level are increasingly supporting more balanced approaches com￾bining repressive measures with longer-term prevention interventions, especially in the most dan￾gerous areas of the country. In both Guatemala and El Salvador, the governments have drafted na￾tional crime and violence prevention plans and have established the necessary institutional infra￾structure at both the national and local levels to design and implement prevention programs. The main problems are that local governments lack sufficient funding to fulfill their new responsibilities, and, for the most part, mayors continue to perceive the problem of crime and violence as a law enforcement problem, which is beyond their mandate. The AMUPREV program addressed a real capacity gap and funding need at the municipal level to design and implement crime and violence prevention interventions. Although AMUPREV is not the only donor-funded program supporting municipal governments in the implementation of crime and violence prevention activities, its regional focus and emphasis on knowledge exchange and learning makes this project different from the rest. The AMUPREV program was well-designed, combining good incentives to motivate and empower municipal stakeholders, offering technical assistance and training, funding projects through small grants, and establishing a network of municipal associations to share news on municipal prevention interventions and disseminate lessons learned. The main problem, however, is that the program objectives were too ambitious for the level of funding and time allocated for work in each country. In Guatemala and El Salvador, AMUPREV was successful in implementing a wide variety of activities in a short period of time and with a highly limited budget. It engaged stakeholders from several mu￾nicipalities and municipal associations to reflect on the important role municipal governments can play in preventing crime and violence and to motivate them to take proactive action. It worked di￾rectly in several municipalities, either establishing or strengthening municipal violence prevention committees; it provided training and technical assistance to enhance the capacity of these commit￾tees to plan and implement new crime and violence prevention programs; it exposed municipal stakeholders to new ideas and experiences introduced in several cities in the United States; and it 71 While the numbers of people who visit the website have increased significantly over the years, this number may be misleading as an indicator of the network’s robustness or the level of commitment of municipal associations to maintain the network over the long term. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 29 established a network of municipal associations to share and disseminate news and information on crime prevention for municipal stakeholders. At the regional level, AMUPREV was also successful in contributing to an ongoing debate on the centrality of prevention strategies for reducing the risks of crime and violence, the need to go be￾yond merely repressive measures to contain the violence epidemic, and municipal governments’ roles and responsibilities in designing and implementing crime and violence prevention programs. Gauging the effectiveness of the AMUPREV network and its influence on municipalities not directly benefitted by the AMUPREV program is difficult to assess. The survey was precisely designed to test the effects of the AMUPREV network on non-direct beneficiaries (people who read the AMUPREV website but have not participated in AMUPREV technical assistance or training activities). But only 48 individuals responded the survey, so the results are not statistically significant. Nevertheless, these responses can be used anecdotally, and they corroborate that direct beneficiaries of AMUPREV’s activities perceive they have a much greater capacity to design and implement munici￾pal crime and violence prevention programs than non-direct beneficiaries.72 AMUPREV designed toolkits for planning, designing and implementing prevention programs at the municipal level which, as discussed above, beneficiaries found useful and practical. These tools are publicly available on AMUPREV’s website, but it is unclear how widely these tools have been used by different municipal governments. Judging from anecdotal evidence, most municipal stakeholders interviewed acknowledged not being informed about the contents of the toolkit in AMUPREV’s website. In Guatemala and El Salvador, municipal associations have no visible commitment to continue to support the AMUPREV network. Although the associations have signed memorandums of under￾standing with AMUPREV, it is unclear whether these will be honored, especially since AMUPREV no longer funds activities in these countries. AMUPREV did not design appropriate outcome level indicators to be able to assess beyond anec￾dotal evidence, whether the program has achieved its overall objective and expected results. From the beneficiaries’ perspective, it is clear that AMUPREV was a welcome and successful program, although consensus exists that it was too short and its funding too limited to have more significant and lasting results. Moreover, without strong monitoring and evaluation systems, the AMUPREV program was vulnerable to funding ad-hoc activities, like purchasing surveillance cameras or sewing machines, without clear relationship to its overall objective. Although AMUPREV’s Monitoring and Evaluation Plan does not provide quantitative data compar￾ing results among municipalities, from interviews with program’s beneficiaries in El Salvador and Guatemala, this evaluation concludes that out of the total municipalities where it worked, AMUPREV was most successful in four municipalities: Palencia and Cobán in Guatemala, and Na￾huizalco and San José Guayabal in El Salvador. In all of these municipalities, AMUPREV was effective in strengthening municipal leadership and increasing the technical capacity of the municipal preven￾tion committees to plan, design and implement crime and violence prevention programs. AMUPREV played a decisive role in formulating municipal strategies and action plans for violence prevention. The learning exchanges allowed these municipalities to replicate initiatives that they ob￾served in different U.S. cities. Mayors became motivated by subsequent visits of U.S. delegations to their municipalities. A combination of strong political will on the part of the mayor, good coordina- 72 80% of survey respondents said they were AMUPREV direct beneficiaries. Most of the survey respondents said they thought AMUPREV’s website was good or excellent and considered news about municipal experiences in crime and violence prevention to be the most useful information on the website. See Annex D for a full analysis of survey results. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 30 tion with other bilateral programs funded by USAID, and strong commitment of the members of municipal prevention committees contributed to the success of the program. AMUPREV’s results are more notable in smaller and poorer municipalities, where a small grant can make a big difference in the ability of a municipal committee to implement prevention activities. This is most outstanding in San José Guayabal, where AMUPREV was allegedly the only donor￾funded program ever to work in this municipality. Paradoxically, it is in these small and poor munici￾palities where the sustainability of AMUPREV-supported activities is more at risk because municipal governments, even if highly committed, have no alternative sources of funding. In bigger and richer municipalities, and especially those where other donors are funding prevention activities, AMUPREV’s results are more difficult to ascertain. Typically, other donor-funded programs working in these municipalities have much larger budgets, which dilutes the results of AMUPREV’s activities. Indeed, in these municipalities, many beneficiaries interviewed for this evaluation had diffi￾culty identifying which activities were specifically funded by AMUPREV, aside from the CityLinks exchange programs. A good exception is the case of Nahuizalco, the only municipality where AMUPREV worked for four consecutive years and where the program effectively coordinated with a large USAID-funded crime prevention program implemented by RTI. The sustainability of the programs’ results is questionable, especially in Guatemala, where virtually all municipalities where the program worked have new mayors who are not committed to supporting any of the programs or activities introduced by previous municipal administrations, including pro￾grams related to violence prevention. The conclusion of AMUPREV’s activities in 2015, precisely as new mayors came to office, only exacerbated sustainability challenges. In general, however, insufficient funding, continued weak technical capacities on the part of munici￾pal officials and lack of commitment of municipal associations threaten the sustainability of the AMUPREV program. AMUPREV’s overall objectives of improving municipal leadership and empowering mayors to design and implement crime and violence prevention programs are highly ambitious, especially considering its time and funding constraints and the scale of the violence epidemic in the region. Both USAID officials in Guatemala and El Salvador agreed that a regional program like AMUPREV requires a much larger budget, a longer time frame, a permanent presence in each country, and direct supervi￾sion and management from USAID’s regional office to achieve such ambitious objectives. VI. RECOMMENDATIONS Based on the evaluation’s findings, DI has developed the following recommendations to improve the program and to replicate some of the program’s most effective components in other USAID￾funded municipal programs addressing violence prevention. I. Donor funded municipal crime and violence prevention programs should be designed with longer time frames and adequate, secure, levels of funding. Mayors, even when initially commit￾ted to supporting violence prevention programs, tend to prefer investing in projects that have quick or visible results and that have electoral payoffs. Prevention interventions, which take time to demonstrate results and affect behavioral and/or attitudinal changes, are not politically expe￾dient in the short term and are disrupted by electoral cycles. For this reason, funding of crime and violence prevention activities should be sufficient to allow programs to plan strategically and for the longer term. II. In addition to designing appropriate training and technical assistance interventions, donor￾funded municipal crime and violence prevention programs like AMUPREV need to introduce the right incentives to motivate mayors to engage and remain committed to prevention. The CityLinks program was one of AMUPREV’s greatest accomplishments, as it effectively motivated PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 31 and encouraged municipal stakeholders to support prevention interventions. This type of incen￾tives should be replicated in other municipal crime and violence prevention programs. III. To increase local buy-in and improve program sustainability, regional donor-funded crime and violence prevention programs should maintain a strong presence at the municipal level to sup￾port and mentor mayors and their administrations and to engage with civil society organiza￾tions. Programs should also invest in socializing crime and violence prevention initiatives with the population at large to increase social support for these programs, generating greater social demand for these types of initiatives, and thereby increasing the political payoffs for mayors to invest in crime and violence prevention programs. IV. To increase the sustainability of municipal association networks, leaders of municipal associa￾tions need to be fully committed to the establishment and maintenance of these networks. These types of networks depend on the ability of leaders to demonstrate concrete benefits to their participants. A program that seeks to establish such networks needs to maintain a longer presence in each country, engage directly with a larger number of municipal governments, and implement a strong communications campaign to disseminate information about the network and educate potential participants about its benefits. V. Political factors such as elections can significantly affect program sustainability and should be taken into account during program design. While electoral cycles are beyond the control of the international donor community, donor-funded programs can include measures to mitigate these risks. For example, in Guatemala AMUPREV could have decided to stay longer or executed a more gradual phase-out to prevent losing momentum on the work it had achieved with the municipal prevention committees and to allow it to work with the newly elected mayors, edu￾cating them on previous accomplishments and motivating them to continue supporting the program. VI. Small and rural municipalities pose greater challenges to the sustainability of donor-funded pro￾grams because they have limited financial and human resources and weak institutional capaci￾ties. In these municipalities where programs like AMUPREV can become more apparent and appreciated by local stakeholders, it is even more critical to design measures to mitigate the risks of donor dependency. This can be accomplished by investing more strategically, ensuring that activities are related to the program’s overall goal, and enhancing municipal government of￾ficials’ capacity to generate local sources of income to fund prevention initiatives. VII. A regional municipal program working on crime and violence prevention should have a decen￾tralized decision making structure, granting technical directors in each country sufficient spend￾ing authority to respond more expeditiously to local needs and demands. VIII. Crime and violence prevention programs should develop explicit theories of change and identi￾fy better performance indicators. Robust monitoring and evaluation frameworks increase pro￾grams’ accountability and help implementers maintain a more strategic focus on the activities they support. To reach program indicator targets, activities have to be clearly linked to the re￾sults the program is seeking to achieve. Unfortunately, lack of appropriate indicators is a short￾coming prevalent among many donor-funded crime and violence prevention programs. In a re￾gion where there is poor understanding of crime and violence prevention, where the political will of government officials to support prevention interventions is uncertain, and where citizens understandably demand from their government quick and visible results, crime and violence prevention programs often support a host of ad-hoc activities without much strategic planning, focus or direction. Examples are numerous, but some examples include supporting one-time events like music concerts and sports tournaments, purchasing surveillance cameras for police officers, and financing arts and craft workshops not appropriately targeted to the right benefi￾ciaries, i.e. youth. IX. Donor-funded violence prevention programs should pay greater attention to the development of outcome, not only output indicators to assess whether they are accomplishing their overall objectives. Aside from increasing a program’s accountability, outcome indicators are a critical PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT 32 learning tool to identify what works and what does not and to assess the changes that need to be made to improve program effectiveness. X. USAID should consider whether regional programs could be managed from regional offices. USAID officials in Guatemala and El Salvador agreed that a regional program of municipal crime prevention should have been managed from the El Salvador regional office, not from Washing￾ton, DC. Although officials agree that regional programs are innovative and can be effective, lack of direct management and supervision from the regional office limits the effectiveness of these programs to adapt to particular conditions, respond more expediently to country-specific demands, and coordinate more effectively with other bilateral donor funded programs. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-1 ANNEX A: SCOPE OF WORK FOR AMUPREV EVALUATION I. Introduction The U.S. Agency for International Development (USAID), represented by USAID’s Bureau for Latin America and the Caribbean, Office of Regional Sustainable Development (LAC/RSD) seeks the support of a short-term technical team to design and implement a performance evaluation of LAC/RSD’s program Municipal Partnerships for Violence Prevention (AMUPREV) in Central Ameri￾ca and the Dominican Republic. This program, funded by RSD’s funds as well as the Central Ameri￾ca Regional Security Initiative (CARSI) and the Caribbean Basin Security Initiative (CBSI), and im￾plemented by the International City-County Management Association (ICMA) focuses on two key objectives: (1) to promote comprehensive municipal-based violence prevention strategies and pro￾grams with key Central American stakeholders and foster development of regional peer knowledge networks; and (2) to provide training and technical assistance to local governments and community groups, in coordination with national police efforts and other municipal-based programs. The AMUPREV program also facilitates learning exchanges between the State and City of Rio de Janeiro and Central America, but this component will not be considered under this evaluation. This evaluation will be conducted by Democracy International under Task Order #AID-OAA-TO￾13-00047, held by USAID’s Bureau for Latin America and the Caribbean, and will be overseen by the Bureau’s Office for Regional Sustainable Development, Democracy and Human Rights Team (LAC/RSD/DHR), with assistance from the Contracting Officer’s Representative in the Bureau’s Of￾fice for Central America and Mexico. II. Performance Evaluation Rationale A) Evaluation Purpose AMUPREV is in its seventh year of implementation and USAID wishes to better understand what the program has accomplished so far toward its desired outcomes of improving municipal actors’ capacity to design and implement crime and violence prevention programs, as well as whether the program has achieved any unexpected outcomes. By funding this evaluation USAID seeks evidence that will help it determine specifically whether and/or how to: ● Improve the program, including through expansion or contraction of specific pro￾gram components; and ● Replicate successful high-impact components, if any, in other USAID-funded munic￾ipal programs addressing violence prevention. B) Audience and Intended Uses Audience ● USAID staff working on citizen security initiatives in Central America and other re￾gions; ● Local and national government officials in Central America and the Dominican Re￾public; ● Regional governmental organizations, e.g. the Central American Integration System; ● The U.S. Department of State, the National Security Council, Congressional stake￾holders, and other USG agencies; ● Municipal associations in each implementation country; PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-2 ● The Confederation of National Municipal Associations in Central America and the Caribbean (CAMCAYCA); and ● Civil society and academic organizations interested in citizen security issues. Intended uses ● To guide USAID program management decision making, including the scope and focus of ongoing and future security programs; ● To disseminate lessons learned and promising practices for effective implementa￾tion, including those for addressing implementation challenges; ● To help USAID identify programmatic and/or geographic gaps in the program that should be addressed by USAID or other actors; ● To help USAID and the implementing partner improve the program; and ● To identify any high-impact components of the program that should be expanded or replicated in other USAID-funded municipal crime and violence prevention programs. III. Background Information In 2009, when the first CARSI programs were under development, LAC/RSD decided to design a program focused on building municipal-level actors’ capacity to prevent crime and violence in a con￾text where citizen security had traditionally been viewed as the sole responsibility of national gov￾ernments. The program was designed to focus on two key objectives: (1) to promote comprehen￾sive municipal-based violence prevention strategies and programs with key Central American stake￾holders and foster development of regional peer knowledge networks; and (2) to provide training and technical assistance to local governments and community groups, in coordination with national police efforts and other municipal-based programs. To accomplish Objective 1, AMUPREV has: developed a toolkit for municipalities to use to pro￾mote the creation of municipal-led mechanisms to provide leadership in bringing together key stakeholders to design and implement crime and violence prevention programs; facilitated the par￾ticipation of experts in national, regional and international conferences; conducted workshops for municipal associations and municipal stakeholders; developed a website (www.amuprev.org) which, along with social media accounts, showcases daily examples of municipalities throughout Latin America that develop promising programs to build the foundation for crime prevention in their communities; and documented and shared via video the experiences of ten municipalities in Central America that have demonstrated a commitment to crime prevention through the leadership of their elected officials, the dedication of their police forces and the collaboration of representatives from all sectors of civil society, as well as national government agencies. To accomplish Objective 2, AMUPREV has engaged in a series of learning exchanges with cities in the U.S. to share the model of community-oriented policing and governance with municipalities in Panama, El Salvador, and Guatemala - the learning exchanges will expand to the Dominican Repub￾lic and Honduras in the 2016-2017 phase. Through these partnerships, police officers, municipal staff, community and private sector representatives from Santa Ana, California; Arlington, Texas Pinellas County, Florida; Alexandria, Virginia; and Stockton, California have provided information, training and technical assistance to their counterparts to bring about a greater awareness of the need to prioritize and implement violence and crime prevention programs in a coordinated and comprehensive fashion. The lessons learned and knowledge gained from the municipal partnerships has been shared with the broader network through the AMUPREV website. In September 2012, USAID issued a six-month extension to ICMA through March 29, 2013. Sub￾sequently, USAID issued a no-cost extension to ICMA through April 30, 2013 and another cost extension through September 30, 2013. During the period of October 2012 through September PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-3 2013 ICMA continued to support both objectives. ICMA applied the tools and methodologies test￾ed throughout the first Phase of this Cooperative Agreement to new Work Plan – October 2015 to September 2016 Municipal Partnerships for Violence Prevention in Central America 4 municipali￾ties in Guatemala and El Salvador; provided continuing support to the Municipal Crime and Vio￾lence Prevention Commissions (MCVPCs) of Palencia and Santa Catarina Pinula, Guatemala; pro￾vided continued strategic support to Nahuizalco and Sonsonate, El Salvador; continued to share information and practices with its established network of violence prevention and other interested practitioners in the Region; and developed a sustainability plan to maintain the network after project end. In October 2013, USAID issued a two-year cost extension to ICMA to continue supporting the creation and strengthening of Municipal Crime and Violence Prevention Committees in Guatemala and El Salvador through September 30, 2015, including expansion of the work in three municipali￾ties in Alta Verapaz, Guatemala (Cobán, Tactic and San Juan Chamelco) as well as Mixco and three municipalities in Cuscatlán, El Salvador (Suchitoto, San José Guayabal and San Bartolomé Perulapía). Two additional U.S. cities – Alexandria, Virginia and Stockton, California – committed to CityLinks Partnerships with the new municipalities. During the extension ICMA continued to strengthen the Municipal Association Network for Violence Prevention in Central America and the Dominican Re￾public (the “Red”) formed in August 2013 to enable it to sustain the work of AMUPREV in the Re￾gion. In September of 2015, USAID issued another two-year cost extension to expand the CityLinks Partnerships into four additional target municipalities, two in Honduras and two in the Dominican Republic and build on the existing AMUPREV network of security practitioners. III. Scope of Work for Performance Evaluation of AMUPREV Program A) Evaluation Design The team’s goal will be to design and implement an evaluation of the two key objectives of the AMUPREV program with input and participation from USAID, including LAC/RSD, LAC/Strategy and Program Office, and other relevant offices and/or bureaus. This evaluation should seek to an￾swer the following key questions, in order of priority: 1. To what degree has creation of a municipal-level crime and violence prevention network in the region facilitated design and implementation of new and/or improved crime and violence prevention programs by key stakeholders, namely national-level municipal as￾sociations and individual municipalities. 2. To what degree has municipalities’ participation in AMUPREV learning exchanges contributed to design and implementation of new and/or improved crime and violence prevention programs? 3. To what degree is AMUPREV complementary to existing bilateral Mission pro￾grams? To what degree is AMUPREV overlapping or working at cross-purposes with bilat￾eral Mission programs? 4. What implementation challenges, if any, have negatively affected program imple￾mentation? This could include internal factors, such as challenges coordinating with several Missions, or external factors such as host countries’ legal frameworks, political will, capacity of local partners, availability of information, etc. The contractor’s conceptual approach to answer these questions will start with project monitoring data and will be complemented with third-party data sources and information obtained during field work, such as key informant interviews, perception surveys, observations from site visits, focus PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-4 groups, consultations with relevant stakeholders and other data gathering methods as proposed by the performance evaluation team in its performance evaluation inception report. The contractor’s consultants must work in conjunction with other team members to plan and im￾plement the proposed performance evaluation. USAID Washington-based staff and Missions in the Dominican Republic, El Salvador, Guatemala, and Honduras will be involved to varying degrees with design, planning, and logistics, but the performance evaluation team must provide significant and overall leadership and direction, as well as full responsibility for the performance evaluation duties and deliverables. B) Data Collection and Analysis Methods The evaluation team will start by reviewing program information, including monitoring data, and identifying information gaps in an inception report to guide additional data collection required as part of the performance evaluation. Data requirements, collection methods, and required analyses will be explicitly defined in the inception report along with proposed sites and schedule for field work and draft questionnaires to be used in interviews, focus groups, etc. Written approval of the inception report from LAC/RSD and the COR is a prerequisite for field work and, as with all deliverables, the first drafts may receive comments and requests for improve￾ment before approval is provided. The inception report will be prepared and discussed and a final version approved by LAC/RSD and the COR before field work will be approved. Interviews with project staff cannot be the sole source of information collected by the performance evaluation team to complement monitoring data and answer the performance evaluation questions. Documentary evidence and discussions with beneficiaries will be important, among other sources the performance evaluation team may propose. Consistent with ADS 203.3.1.6 guidance on performance evaluation methodologies, to the extent possible, a combination of quantitative and qualitative methods in data collection and analysis should be employed by the performance evaluation team in the process. A triangulation method should be applied by the performance evaluation team to increase the level of validity in data collection and processing of results. USAID encourages the performance evaluation team to use a multiple-perspective view that trian￾gulates data sources. Even if a performance evaluation question can be answered using only one method, often it is preferable to combine multiple methods to answer the same question in order to gain a more complete understanding of the issue and more confidence in the findings. The performance evaluation team must ensure that its methods are gender-sensitive, capturing quantitative and qualitative data on whether and how AMUPREV addresses issues specific to wom￾en and girls, boys and men, and lesbian, gay, bisexual, transgender/gender non-conforming, and in￾tersex (LGBTI) persons. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-5 Sample Template: Inception Report for Pre-Field Visit Data Needs and Analytical Guide Evaluation Questions Relevant Ex￾isting Project Monitoring Data Recommended New Data to be Collected during evalua￾tion New Data Col￾lection Meth￾od(s) Sampling or Selection Crite￾ria of New Da￾ta Outstanding Data Needs that Cannot be Met (Data Limi￾tations) (1) 1. 2. 3. … n. n = number of existing project mon￾itoring data indicators relevant to this question 1. 2. … m. m = number of proposed new data indicators, to be collected by the evalua￾tion team dur￾ing field work or from 3rd par￾ty sources, rele￾vant to this question 1. 2. … m. m = number of proposed new data indicators, to be collected by the evaluation team during field work or from 3rd party sources, relevant to this question 1. 2. … m. m = number of proposed new data indicators, to be collected by the evalua￾tion team dur￾ing field work or from 3rd par￾ty sources, rele￾vant to this question 1. 2. … p. p = number of indicators that would be valuable if they existed, but which cannot be obtained by the evaluation team during field work or from project data, relevant to this question C) Site Visits and Data Collection The AMUPREV Program currently works with national municipal associations in Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, and Panama, as well as with specific munic￾ipalities in Honduras and the Dominican Republic; in the past it has also worked with municipalities in El Salvador, Guatemala, and Panama. The evaluation team will need to visit, at a minimum, El Sal￾vador and Guatemala in order to conduct site visits and interviews with participants in learning ex￾changes, municipal association officials, ICMA staff, and USAID Mission staff. The team may also choose to visit Panama, where the first AMUPREV-managed learning exchanges took place, as well as Honduras and/or the Dominican Republic. For those countries that the team does not visit in person, the team will conduct interviews via phone. Democracy International will propose a field visit schedule for discussion; the final schedule will require approval from LAC/RSD and the COR. D) Period of performance The period should start on or around February 1, 2016. A final evaluation should be completed and delivered to the Mission no later than June 27, 2016. E) Work plan Within one week of receiving this support scope of work, Democracy International should produce a work plan that details its understanding of the assignment, timeline, and budget. At that time, Democracy International should also submit a proposed list of key personnel for the design effort, along with their resumes or CVs. At the end of the fieldwork, the design team will present its draft findings to LAC/RSD and the COR. The design team should submit a draft final evaluation to LAC/RSD (via the Task Order’s Contracting Officer’s Representative in LAC/CAM) within 14 days of completing data gathering and PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-6 field work. USAID will provide comments to the design team within one week of the submission of the draft evaluation. Depending on the nature of LAC/RSD’s comments, there may need to be more than one round of revisions necessary to finalize the scope of work. The design team will submit the final evaluation to USAID within one week of receiving USAID’s comments. Upon final￾izing the evaluation, the evaluation team will present its findings to LAC/RSD and LAC/CAM. IV. Budget Democracy International should submit a proposed budget that matches the proposed design ob￾jectives, staffing plan, timeline, and fieldwork. V. Field Work The AMUPREV Program currently works with national municipal associations in Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, and Panama, as well as with specific munic￾ipalities in Honduras and the Dominican Republic; in the past it has also worked with municipalities in El Salvador, Guatemala, and Panama. The evaluation team will need to visit, at a minimum, El Sal￾vador and Guatemala in order to conduct site visits and interviews with participants in learning ex￾changes, municipal association officials, ICMA staff, and USAID Mission staff. The team may also choose to visit Panama, where the first AMUPREV-managed learning exchanges took place, as well as Honduras and/or the Dominican Republic. For those countries that the team does not visit in person, the team will conduct interviews via phone. Democracy International will propose a field visit schedule for discussion; the final schedule will require approval from LAC/RSD and the COR. The contractor is responsible for arranging travel logistics (e.g., plane tickets, vehicle rental and driv￾ers, hotel accommodations, meeting spaces) as needed for site visits and fieldwork. USAID staff in LAC/RSD, LAC/CAM, relevant USAID Missions, and ICMA will be available to the evaluation team for consultations regarding logistics related to fieldwork but will not be responsible for the team’s scheduling, hotel reservations, transportation etc. The team will meet with in person or virtually with LAC/RSD at the beginning, middle, and end of the evaluation period to go over plans, expectations and findings. VI. Special Provisions Unless in possession of an active USG security clearance, the design team will not have access to any USG classified material. A six-day work week is allowed. VII. Key personnel/design team Given the complexity and importance of this design, the evaluation team must include a combina￾tion of expertise from practitioners, researchers, and policymakers, and must include local experts and USAID staff. USAID encourages the participation of team members with expertise related di￾rectly to the region. USAID suggests the following team composition; however, Democracy Interna￾tional is encouraged to present a composition that it feels will best allow it to undertake the pro￾gram design. ● Team leader / senior monitoring and evaluation expert with at least 10 years of relevant experience working directly on the design and management of develop￾ment projects related to municipal capacity-building and/or crime and violence prevention PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-7 and a graduate degree in a related field. He/she must have experience using quantitative and qualitative methods to assess or manage programs. He/she must have experience working in a developing country context, preferably in Latin America. English fluency (speaking and writing) and Spanish fluency (speaking and writing) are required. ● Senior social scientist / researcher / crime and violence prevention expert with at least 10 years of relevant experience and advanced graduate training in social sci￾ence research methods and/or evaluation methods, and with experience in areas related to the crime prevention methodologies being evaluated (criminology, youth development, so￾ciology, community development, juvenile justice, etc.). He/she must have extensive experi￾ence using both quantitative and qualitative methods to conduct assessments or evalua￾tions. He/she must have experience working in a developing country context, preferably in Latin America. English fluency (speaking and writing) is required, and Spanish fluency (speaking) is strongly preferred. The Team Leader, responsible for overall guidance of the program design and for producing the deliverables. S/He will take the lead in the design, develop schedules, timelines and reporting re￾sponsibilities for the team; receive and integrate inputs of each team member into the draft report; and produce final deliverables. S/he will serve as the primary liaison with USAID and ICMA. ● At least one local expert with demonstrated knowledge of and expertise in munic￾ipal capacity-building, crime and violence prevention, and/or monitoring and evaluation of the same. Very good English and excellent Spanish language skills are required. ● Administrative support coordinator who supports the evaluation’s implementation and production of deliverables. ● USAID will contribute, depending on availability, at least one staff member to the evaluation team, who will support from Washington and may join the performance evalua￾tion team in the field. Level Of Effort The maximum level of effort for a single member of the team is 34 days. A six-day work week dur￾ing field work is authorized. Task Expatriate Consultants Local Consultant(s) Desk-Top Review and Initial Meetings Washington: 4 days Capital city/cities: 2 days per city Field Work 18 days 18 days Finalizing Reports 12 days 5 days VIII. DESIGN DELIVERABLES All deliverables should be submitted in electronic form to the Contracting Officer’s Representative for the Task Order in LAC/CAM and to the LAC/RSD point of contact. The evaluation team will provide the following: ● An evaluation design outlining the evaluation methodology; proposed team com￾position; proposed timeline; and work plan. ● An inception report including proposed sites and schedule for field work and draft questionnaires to be used in interviews, focus groups, etc. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT A-8 ● A final report that addresses the evaluation questions and includes recommenda￾tions for short-term course corrections as well as whether USAID should seek to further extend the AMUPREV program and/or consider designing a new program focused on mu￾nicipal capacity-building for citizen security once the program has ended. These recom￾mendations should identify ways to: improve the program, including through expansion or contraction of specific program components; and replicate successful high-impact compo￾nents, if any, in other USAID-funded municipal programs addressing violence prevention. ● A verbal presentation with accompanying slide deck and brief (under two pages) fact sheet summarizing the report’s main findings to LAC/RSD. Gender Integration It is the policy of USAID to integrate gender in all projects and programs across sectors. As such, evaluation efforts should take into account how the AMUPREV program considers, affects, or is affected by gender. For further information on gender integration and analysis see http://www.usaid.gov/policy/ads/200/201sab.pdf. V. Key Documents LAC/RSD, ICMA, and relevant USAID Missions will provide access to relevant information, including the program description, quarterly and annual progress reports, monitoring and evaluation data, and other documents as needed. LAC/RSD will help the evaluation team to identify and schedule meet￾ings with stakeholders in Washington and the field. The evaluation team should meet with Wash￾ington-based USAID staff, ICMA staff, and other key partners before departing for field work. VI. Evaluation Management and Points of Contact This performance evaluation will be conducted as part of a Task Order managed by a Contracting Officer’s Representative within LAC/CAM. LAC/RSD staff will oversee the technical aspects of the performance evaluation and the field work. Points of contact for this performance evaluation are: USAID/LAC/RSD, E. Brennan Dorn, ebdorn@usaid.gov and USAID/LAC/CAM, Jeremy Biddle (COR), jbiddle@usaid.gov. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT B-1 ANNEX B: LIST OF PEOPLE INTER￾VIEWED Name Title City María Elena Morales Vice-Minister, III Vice Ministerio de Prevención de la Violencia y el Delito, Ministerio de Gobernación Guatemala City Carlos Gómez Unidad Ejecutiva, Unidad para la Prevención Comu￾nitaria de la Violencia Guatemala City Carlos Rosales USAID/DG/ Guatemala Guatemala City Betsabé Méndez Communications Director, ANAM Guatemala City Isaías Martínez Mayor of San Diego Zacapa, former President of ANAM Guatemala City Leticia Sontay Former AGAAI communications specialist Guatemala City Karin Peña AMUPREV, Former Technical and Logisitcs Advisor for Guatemala Guatemala City Lic. Isauro Gómez López Director of Security, Municipality of Mixco Mixco, Guatemala Verónica Pérez Municipal Social Services, Mixco Mixco, Guatemala Ana Ligia Castillo Former Municipal Security Director Assistant, Mixco Mixco, Guatemala Ana Luisa Montufar Gómez National Police Director, Mixco Police Station Mixco, Guatemala Francisco Rodas Community Leader, Mixco Mixco, Guatemala Carlos Guárquez Executive Director, AGAAI Guatemala City Gerson López Rodas Former Executive Director, ANAM Guatemala City Silvia Vásquez Former Vice-Minister, III Vice Ministerio de Preven￾ción de la Violencia y el Delito, Ministerio de Gober￾nación Guatemala, City Lorena Guerra Former Vice-Minister, III Vice Ministerio de Preven￾ción de la Violencia y el Delito, Ministerio de Gober￾nación Guatemala, City Víctor Alvarizae Mayor, Santa Catarina Pinula Santa Catarina Pinula, Gua￾temala Pedro Peña City Councilor, Santa Catarina Pinula Santa Catarina Pinula, Gua￾temala Irma Yolanda Ochoa Gon￾zález Community Leader, Santa Catarina Pinula. Member of the Municipal Committee of Violence Prevention Santa Catarina Pinula, Gua￾temala Nery Edmundo García Community Leader, Santa Catarina Pinula. Member of the Municipal Committee of Violence Prevention Santa Catarina Pinula, Gua￾temala Heidy Moran Director, Outreach Center. Member of the Munici￾pal Committee of Violence Prevention Santa Catarina Pinula, Gua￾temala Pastor William Quiñonez Pastor, Member of the Municipal Committee of Vio￾lence Prevention Santa Catarina Pinula, Gua￾temala Profe. Clarita School Director, Member of the Municipal Commit￾tee of Violence Prevention Santa Catarina Pinula, Gua￾temala Artemio Uscam Deputy Police Director, National Police Station in Santa Catarina Pinula Santa Catarina Pinula, Gua￾temala Lilian Vino Director, Municipal Police Santa Catarina Pinula, Gua- PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT B-2 temala Guadalupe Alberto Reyes Aguilar Mayor, Palencia Palencia, Guatemala Juan De Dios Pineda Former Director, Municipal Social Services Palencia, Guatemala Ramiro Pérez Former Mayor, Palencia Palencia, Guatemala Marco Tulio Alvizurez Former City Council Member Palencia, Guatemala José Alejandro Cambara National Police Official, Palencia Station Palencia, Guatemala Vicente Cortez Hernández National Police Official, Palencia Station Palencia, Guatemala Renzo Rosales Political Analyst Guatemala City Cocky Córdoba Mayor, Cobán Cobán, Guatemala Marlon Dubón City Manager, Cobán Cobán, Guatemala Carmen Cacao Former Delegate of the National Police, Cobán Cobán, Guatemala Amílcar Salvador Former Council member, Cobán Cobán, Guatemala Byron Randolfo López Mu￾ñoz Official, National Police, Cobán Station Cobán, Guatemala Aura Rey Director of Municipal Court services, Former Munic￾ipal Secretary Cobán, Guatemala Mario Hernández Chenal Regional Technical Advisor, Unidad para la Preven￾ción Comunitaria de la Violencia, Cobán Cobán, Guatemala Marco Tulio Sierra Lemus Council Member Cobán, Guatemala Edin Guerrero Miliam Mayor, Tac Tic Cobán, Guatemala Carlos Giovani López Deputy Mayor, Tac Tic Tac Tic, Guatemala Ottoniel Coy Director, Municipal Youth Office Tac Tic, Guatemala Rony Oswaldo Caal Vice-President, Municipal Committee of Violence Prevention (COMUPRE) Tac Tic, Guatemala Marvin Arnoldo Quej Director, Municipal Office for the Protection of Chil￾dren and Adolescents Tac Tic, Guatemala Marvin Isem Director, Municipal Office for Women Tac Tic, Guatemala Glendi Cho Deputy Director, Municipal Office for Women Tac Tic, Guatemala Julio Alberto López Moran Inspector, National Police, Tac Tic Station Tac Tic, Guatemala Sergio Dubón City Councilor Tac Tic, Guatemala Ervin Tut Quim Mayor, San Juan Chamelco San Juan Chamelco, Gua￾temala Carlos Cuc City Council, San Juan Chamelco San Juan Chamelco, Gua￾temala Francisco Astacio AMUPREV, Communications Director San Salvador, El Salvador Marisa Fortín de Miranda AMUPREV, Former Technical Director, El Salvador San Salvador, El Salvador Santiago Flores Former Director, PREPAZ (2010-2013) San Salvador Jose Roberto Aquino Mayor, Sonsonate Sonsonate, El Salvador Alvaro Díaz Director National Police, Sonsonate Delegation Sonsonate, El Salvador Rosario Abarca Regional Representative, Prepaz Sonsonate, El Salvador Janira Díaz Delegate of the Human Rights Ombudsman’s Office, Gender Coordinator, Municipal Committee of Vio￾lence Prevention Sonsonate, El Salvador Nelson Villalta Technical Secretary, Municipal Committee of Vio￾lence Prevention Sonsonate, El Salvador Carlos Eduardo Bernal Doctor, Municipal Committee of Violence Preven￾tion Sonsonate, El Salvador Maira Manuel Municipal Gender Coordinator, Municipal Commit- Sonsonate, El Salvador PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT B-3 tee of Violence Prevention Julian Cruz Former Technical Secretary of the Municipal Com￾mittee of Violence Prevention Sonsonate, El Salvador Carlos Arevalo Municipal Committee of Violence Prevention Sonsonate, El Salvador Julio César Monterroso Municipal Police, Municipal Committee of Violence Prevention Sonsonate, El Salvador Jorge Willer Patríz Cas￾taneda Mayor, Nahuizalco Nahuizalco, El Salvador Carmen Rauda de Valdez Municipal Secretary, Nahuizalco Nahuizalco, El Salvador Lizbeth Zúniga Municipal Committee of Violence Prevention Nahuizalco, El Salvador Danilo Candel Executive Secretary, Municipal Committee of Vio￾lence Prevention Nahuizalco, El Salvador Flor Alicia de Elios Treasurer, Municipal Committee of Violence Preven￾tion Nahuizalco, El Salvador Lidia del Carmen de Valdéz Deputy Secretary, Municipal Committee of Violence Prevention Nahuizalco, El Salvador Mauricio Vilanova Mayor, San José Guayabal San José Guayabal, El Sal￾vador Blanca Celia Melendez Pre￾za City Councilor, Member of the Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Etelvina Interiano Aguilar Regional Delegate, PREPAZ San José Guayabal, El Sal￾vador Maria Lucila Martínez City Councilor, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador José Santos Hernández City Councilor, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Carlos Guillermo López Director, Youth House, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Raul Alexander Mensivar Guzman Youth House, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador María de los Ángeles Alfaro Ortega Director, Municipal Women’s Office, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Ana Rosmery Joachín Rivas Municipal Social Services Manager, Member of Mu￾nicipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Enrique Antonio Cantor Rodríguez Municipal Official, Member of Municipal Committee of Violence Prevention San José Guayabal, El Sal￾vador Vicente Quintanilla Pérez Chief Police Officer, National Police post in San José Guayabal San José Guayabal, El Sal￾vador Cristian Marvin Sermeño Arias Mayor, San Bartolomé Perulapía San Bartolomé Perulapía, El Salvador Keysi Lisbeth Henríquez Municipal Women’s Office, Member of the Municipal Committee of Violence Prevention San Bartolomé Perulapía, El Salvador Ulises Alfredo López Cruz Member of the Municipal Committee of Violence Prevention San Bartolomé Perulapía, El Salvador Saul Lopez Official at the National Police, San Bartolomé Peru￾lapía Station San Bartolomé Perulapía, El Salvador Kaisy de Orellana Secretary of the Municipal Committee of Prevention San Bartolomé Perulapía, El Salvador Rogelio Pons Municipal Government, Member of the Municipal Committee of Prevention San Bartolomé Perulapía, El Salvador PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT B-4 German Perez Member of the Municipal Committee of Prevention San Bartolomé Perulapía, El Salvador Tomas Echevarria Member of the Municipal Committee of Prevention San Bartolomé Perulapía, El Salvador Carlos Molina Martined Director Health Center and Member of the Munici￾pal Committee of Prevention San Bartolomé Perulapía, El Salvador Pedrina Rivera Hernández Mayor, Suchitoto Suchitoto, El Salvador Flor Bergeli Rivera de Pa￾nameño Municipal Official, Member of the Municipal Commit￾tee of Violence Prevention Suchitoto, El Salvador Walter Candelario Rodri￾guez Municipal Official, Member of the Municipal Commit￾tee of Violence Prevention Suchitoto, El Salvador Torivio Emilio Rivera Municipal Official, Member of the Municipal Commit￾tee of Violence Prevention Suchitoto, El Salvador Conchi Serrano Municipal Official, Member of the Municipal Commit￾tee of Violence Prevention Suchitoto, El Salvador Salvador Augusto Reyes Police Official, National Police, Suchitoto Post Suchitoto, El Salvador Mauricio Herrera USAID/DG/El Salvador San Salvador PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-1 ANNEX C: EVALUATION INSTRU￾MENTS LIST OF QUESTIONS FOR IMPLEMENTER (ICMA) 1. En el diseño del programa AMUPREV, cual fue el objetivo central que se pretendía alcanzar al nivel más general? Qué problema se pretendía resolver o atender con el programa? 2. De donde surgió la idea del programa?  USAID?  ICMA?  Autoridades locales? 3. Según los quarterly reports, existen dos componentes fundamentales (los quarterly reports lla￾man a estos dos componentes “objetivos”; nosotros los llamamos resultados intermedios para no confundirnos con el objetivo central o el “overall goal”) del programa:  Generar una red de autoridades locales para promover estrategias municipales de pre￾vención de violencia y diseminar información relevante—aquí entra el establecimiento de la RED, la creación del sitio internet y el desarrollo de la caja de herramientas (tool￾kit) para autoridades municipales.  Fortalecer a autoridades municipales (6 municipios en Guatemala y 5 en El Salvador) para fortalecer sus capacidades en el diseño e implementación de programas de pre￾vención de violencia en sus municipios—aquí entra el entrenamiento y asistencia técni￾ca a las autoridades municipales, el establecimiento del CMPV, las donaciones peque￾ñas en los municipios y los intercambios entre ciudades Citylinks. Confirmar que entendemos bien los componentes del programa.  Qué tipo de asistencia técnica y/o entrenamiento le ofrecieron a las autoridades municipa￾les?  Cuantos programas de donaciones pequeñas implementaron en los municipios? Cuáles fue￾ron los objetivos de estos programas?  Cómo se formó la RED y qué tan útil ha sido para los alcaldes? 4. En El Salvador y en Guatemala, dos países con estructuras centralizadas de gobierno, cuál es el rol de los municipios en la prevención de violencia? Cuáles son los desafíos más importantes? 5. Cómo seleccionaron a los municipios en los que trabajaron? A los participantes de los inter￾cambios? Qué criterios utilizaron para seleccionar a los municipios/participantes individuales? 6. Después de 7 años de implementación, ha cambiado el objetivo central? PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-2 7. Cuáles han sido los logros más sobresalientes? Y cuales han sido los desafíos más importantes? 8. Cuáles son las posibilidades de que los programas de prevención se mantengan en los munici￾pios en los que trabajaron? 9. Que posibilidades existen de que la RED AMUPREV se sostenga sin el apoyo del programa? 10. Que cambios harían al programa? Lecciones aprendidas? LIST OF QUESTIONS FOR PROGRAM BENEFICIARIES 1. Qué tan grave es el problema de violencia en su municipio? 2. Cuál es el problema más grave? 3. Qué tanto ha cambiado la situación de violencia en los últimos cinco/ tres años? 4. Han realizado un diagnóstico sobre la situación de la violencia en su municipio? Cuándo? a. Cuáles fueron los resultados de dicho diagnóstico? Se utilizaron los resultados para di￾señar la estrategia de prevención en el municipio? Si no, que otra información utilizaron para diseñar sus estrategias de prevención? b. Esta estrategia de prevención se diseñó con un enfoque multisectorial? Es decir, partici￾paron otros sectores y la sociedad civil? 5. Qué tareas le corresponden al municipio en la prevención de violencia? Qué puede hacer el gobierno municipal? 6. Cómo se coordina el gobierno municipal con el gobierno central? Qué tipo de actividades se hacen de forma conjunta? Reciben transferencias del Gobierno Central para temas de preven￾ción de la violencia? 7. Se coordina con otros sectores para actividades de prevención de violencia? Con cuáles? 8. Qué tantos recursos se destinan al tema de seguridad ciudadana? Qué porcentaje proviene de fondos municipales y que porcentaje de fondos del gobierno central (eso si es que reciben transferencias del gobierno central) 9. Qué hicieron con AMUPREV? 10. Qué le resultó más útil del programa? Y lo menos útil? 11. Que tan satisfecho está con el programa? 12. Qué tan sostenibles son las iniciativas de prevención que implementaron a raíz del programa AMUPREV? PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-3 13. Qué cambios le haría al programa? 14. Algunas lecciones aprendidas? LIST OF SURVEY QUESTIONS Q1 Que tan grave es el problema de la violencia en su municipio?  Grave (1)  No es grave (0) Answer If Existe un problema de violencia en si municipio? Si Is Selected Q2 Cuál es el problema principal?  Homicidios (1)  Extorsiones (2)  Robos (3)  Otro (4) Q3 En los últimos tres años, la violencia ha empeorado, ha mejorado o sigue igual?  Ha empeorado (1)  Sigue Igual (2)  Ha disminuido (3)  No sabe (88) Q4 Quien es el responsable de la prevención de la violencia?  gobierno municipal (1)  gobierno central (2)  los dos son responsables (3)  No sabe (88) Q5 Que puede hacer el gobierno municipal para la prevención de la violencia? PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-4 Q6 En los últimos tres años, qué cambios ha hecho su gobierno municipal para mejorar la preven￾ción de la violencia? Q7 Existe alguna instancia municipal de prevención de violencia en su municipio?  Si (1)  No (2)  No sabe (88) Q8 Conoce el programa AMUPREV?  Si (1)  No (0) Q9 De donde conoce el programa AMUPREV? (puede responder a varias respuestas)  Recibo su boletín (1)  Participé en sus actividades sin ser beneficiario directo (2)  Participé en sus actividades como beneficiario directo (3)  Soy beneficiario (4)  Otro (5)  No sabe (88) Q10 Ha visitado el portal web de AMUPREV?  Si (1)  No (0)  No sabe (88) Answer If Ha visitado el sitio web de AMUPREV? Si Is Selected Q11 Del 1 al 5, que tan útil le resulta el sitio?  Excelente (1)  Bueno (2)  Normal (3)  Regular (4)  Muy malo (5) Answer If Ha visitado el portal web de AMUPREV? Si Is Selected Q12 Que es lo que considera más útil del portal web de AMUPREV?  La caja de herramientas (1)  Noticias (2)  Regulaciones (3)  Experiencias en prevención (4) PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-5 Answer If Ha visitado el portal web de AMUPREV? Si Is Selected Q13 Que información considera que debería incluirse? Q14 Participó en las actividades del programa AMUPREV?  Si (1)  No (0) Q15 Cual es la razón por la cual participó en el programa de AMUPREV? (puede responder a va￾rias respuestas)  Para mejorar mis conocimientos sobre prevención de la violencia (1)  Para interactuar con otros practicantes/expertos en prevención de la violencia (2)  Aprender de experiencias de prevención de la violencia en otros países (3)  Mi trabajo requirió/sugirió que participara (4)  Aprender sobre herramientas prácticas de prevención de la violencia (5)  Para crear/participar en redes de prevención de la violencia (6)  Enseñar a otros sobre prevención de la violencia (7)  Otro (8) Q16 En qué actividades participó mayormente del programa de AMUPREV?  Conferencias (1)  Talleres (2)  Reuniones (3)  Visitas a proyectos (4)  Intercambios (6)  Otro (5) Answer If En que tipo de actividades participó? Otro Is Selected Q17 Que otras actividades? Por favor, especificar Q18 Que tan satisfecho estuvo con las actividades de AMUPREV?  Muy satisfecho (1)  Algo satisfecho (2)  Poco satisfecho (3)  Nada satisfecho (4)  No sabe (88) Q19 En cuantas actividades ha participado?  1 a 5 (1)  6 a 10 (2)  11 a 15 (3)  16 a 20 (4)  más de 20 (5) PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-6 Q20 Cree Ud. que continuará participando en actividades de intercambio en prevención de la vio￾lencia con otros municipios a través de una comunidad de practicantes u otra plataforma?  Si (1)  No (2)  No sabe (88) Q21 Cuanto cree Ud. que está capacitado para diseñar programas de prevención de la violencia en su municipio?  Muy capacitado (1)  Capacitado (2)  Poco capacitado (3)  Nada capacitado (4)  No sabe (88) Answer If Conoce el programa AMUPREV? No Is Selected Q22 Qué cambios ha realizado en su municipio para prevenir violencia? Q23 Qué tan viable es que estos cambios se sostengan en el futuro?  Muy viable (1)  Viable (2)  Es difícil saber (3)  Poco viable (4)  No es viable (5)  No sabe (88) Q24 Que tanto participa la sociedad civil en las actividades de seguridad ciudadana en su municipio?  Mucho (1)  Poco (2)  No es relevante en las tareas de prevención (3)  No sabe (88) Q25 Que tanta confianza tiene a la policía en su municipio?  Mucha confianza (1)  Algo de confianza (2)  Poca confianza (3)  No confía (4)  No sabe (88) Q26 En qué país reside?  El Salvador (1)  Guatemala (2)  Otro (3) Answer If En qué país reside? Otro Is Selected Q27 Que país?  Panamá (1)  Honduras (2)  Costa Rica (3) PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-7  Nicaragua (4)  México (5)  Estados Unidos (6) Q28 Cual es su sexo?  Hombre (1)  Mujer (0) Q29 En que año nació? Q30 Cual es su estado civil?  Casado (1)  Divorciado (2)  Soltero (3)  Viudo (4)  Conviviendo en pareja (5)  Otro (6)  No sabe (88) Q31 Cual es su lugar de residencia?  Urbano (1)  Suburbano (2)  Rural (3)  No Sabe (88) Q32 Cual es su mayor nivel de educación?  No completó escuela primaria (1)  Completó escuela primaria (2)  No completó escuela secundaria (3)  Completó secundaria (4)  Carrera técnica (5)  2 años de grado universitario (6)  Grado Universitario (7)  Post-grado (Master) (8)  Doctorado (9)  No Sabe (88) Q33 Dónde trabaja?  Oficina del Presidente del Gobierno (1)  Oficina del Alcalde (2)  Ministro (3)  Parlamentario (4)  Alcalde (5)  Secretario de Seguridad/prevención de la violencia (6)  Secretario de otras secretarias (7)  Concejal (8)  Policía/cuerpos de seguridad (9)  Líder comunitario (10) PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT C-8  Iglesia/grupos de fe (11)  Academia/think tank (12)  Otro (13)  No Sabe (88) Q34 ¿Usted se considera una persona ladina, indígena u otra?  Ladina (1)  Indígena (2)  Otra (3)  No sabe (88) Q35 Fue víctima del delito en el último año?  Si (1)  No (0)  No Sabe (88) Q36 Hablando del lugar o el barrio/la colonia donde usted vive y pensando en la posibilidad de ser víctima de un asalto o robo, ¿usted se siente:  Muy seguro (1)  Algo seguro (2)  Algo inseguro (3)  Muy inseguro (4)  No Sabe (88) Q37 Cual es su nivel de ingreso?  Alto (1)  Medio (2)  Bajo (3)  No sabe (88) PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-1 ANNEX D: RESULTS OF SURVEY RESPONDENTS Democracy International designed a survey instrument to collect information on the AMUPREV program and on people’s perceptions on municipal governments’ role for designing and introducing crime and violence prevention programs from a wider sample of visitors of AMUPREV’s website. The purpose of this survey was to respond to the questions in this evaluation and to broaden data collection beyond El Salvador and Guatemala, where the DI traveled to conduct field work. The survey also was designed to triangulate anecdotal evidence collected through personal interviews with AMUPREV’s beneficiaries in El Salvador and Guatemala and to assess whether those individuals who were direct beneficiaries of AMUPREV’s activities perceived they had more capacity to design and implement crime and violence prevention programs at the municipal level than those individuals who visit AMUPREV ‘s page but have not been direct beneficiaries of AMUPREV ‘s activities. DI used a propensity score matching methodology to assess the impact of AMUPREV on program beneficiaries. 73 The final questionnaire consisted of 37 questions and included both closed-ended and open-ended questions. The questionnaire was sent online to all AMUPREV’s contact list of more than 2,500 beneficiaries and non-beneficiaries to try to capture enough information and to attempt to create treatment and control groups for our analysis. The length of time required to complete the whole survey was 15 minutes if the respondent responded to the whole survey74. The survey was sent out in April 2016 by AMUPREV’s communications director and included a brief note from AMUPREV’s chief of party requesting individuals to respond the survey, and the team sent out two reminders to survey respondents to garner as many responses as possible. Unfortunately, only 48 individuals re￾sponded to the survey. There are a few limitations worth noting about the survey instrument. The instrument was pilot tested with DI’s and ICMA’s team and adjustments were made based on inputs from respondents. The instrument, however, was not pilot tested with a random group of beneficiaries for various rea￾sons. First, the survey was administered online and therefore it was very difficult to find a random group of potential respondents to pilot test the survey because of its online nature. Second, DI’s team had to administer the survey via AMUPREV’s team which sent the link to the survey in their weekly bulletin or as a direct email to respondents. AMUPREV sent only two direct emails about the survey to encourage respondents to take the survey. Third and most notably, DI’s team encouraged AMUPREV and ICMA to send more reminders in light of the surge in responses after respondants received a direct email. However, according to AMUPREV and ICMA, this was not possible and, as a result, the survey had to be deactivated. 73 Shenyang Y. Guo and Mark W. Fraser, Propensity Score Analysis: Statistical Methods and Applications, Vol. II (Thousand Oaks: Sage Publications, 2015). Matching methods allows us to find a credible counterfactual by matching similar individuals based on observable characteristics and then making one-to-one comparisons between participants and their matched non-participants according to different algorithms. In other words, if the matching is conducted correctly, the differences between the treatment and control groups would disappear and, therefore, any differences in the outcome are attributable to the program. 74 If the respondent was not a direct beneficiary of AMUPREV, s/he did not have to respond the section about AMUPREV. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-2 The above may have affected the overall low response rate. Surveys that are administered online generally require weekly reminders to ensure the research team obtains an adequate sample to conduct analyses. The following paragraphs present some of the main descriptive findings of the survey. It is important to caution the reader that the sample is small (only 48 individuals responded to the survey) and, therefore, not representative of the opinions of participants in AMUPREV’s network. Most of the respondents (80%) to the survey think that there is a serious problem of violence in their municipality, compared to less than 30% who think that violence and insecurity is not a serious problem. Figure D-1. Whether violence is a problem in your Municipality Source: DI’s analysis based on the results of the AMUPREV survey However, about 50% of respondents think that violence has increased in the past five years, com￾pared to less than 30% who think that violence has either decreased or stayed the same. This con￾firms official statistics as well as the information collected during the interviews in the beneficiary municipalities. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-3 Figure D-2. Evolution of Violence in the Municipality Source: DI’s analysis based on the results of the AMUPREV survey Respondents were asked about who should be responsible for violence prevention in their munici￾pality. The vast majority of respondents think that both the central and municipal government should be responsible for the prevention of violence. Interestingly, however, 20% of respondents think the central government should be mostly responsible. The remaining 80% said the municipality should be mostly responsible for violence prevention. This is an interesting finding, as municipalities are underfunded to effectively design and implement violence prevention programs. Figure D-3. Whether Central or Local Government are Responsible for Violence Prevention Source: DI’s analysis based on the results of the AMUPREV survey Respondents acknowledged there is an entity in their municipality responsible for violence preven￾tion. The large percentage (80%) in the responses probably indicates that there is a municipal vio￾lence prevention committee. Most of these committees do not have sufficient resources to imple￾ment prevention activities. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-4 Figure D-4. Existence of Violence Prevention Instances in your Municipality Source: DI’s analysis based on the results of the AMUPREV survey When respondents were asked about whether they knew AMUPREV, 75% knew about AMUPREV. Out of these respondents (not shown here) about 80% were direct beneficiaries of AMUPREV’s activities and 21% were not. In addition, the vast majority of respondents (75%) who know of AMUPREV were males ages 38 to 39 years old. Figure D-5. Whether respondent knows about AMUPREV Source: DI’s analysis based on the results of the AMUPREV survey The fact that respondents are not direct beneficiaries of AMUPREV’s activities does not mean they are not active members of the network. For example, as the figure below illustrates, on average, about 80% of respondents visit AMUPREV’s portal. If we disaggregate further, the percentage of respondents that know AMUPREV and visit the portal increases to 85% (not shown here). Figure D-6. Whether respondent visits AMUPREV’s portal PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-5 Source: DI’s analysis based on the results of the AMUPREV survey Most visitors to the AMUPREV website said they think the website is very good. While somewhat expected, most respondents see value in the experiences shared in AMUPREV’s website. The wealth of municipal experiences disseminated every week on AMUPREV’s website provides infor￾mation and serves as an incentive for municipal leaders and practitioners to try to implement similar prevention activities in their municipalities. Ultimately, it is the role of the network to feed policy￾makers and practitioners about promising experiences on violence prevention and these results somewhat confirm that the network is effective. Moreover, there are no significant differences be￾tween direct beneficiaries of AMUPREV’s activities and those who just know of it or only receive the weekly bulletin. Figure D-7. Respondents Opinion of AMUPREV’s Utility Source: DI’s analysis based on the results of the AMUPREV survey The results for those respondents who are direct beneficiaries of the program show that more than two thirds have participated in 1-5 program activities. Interestingly, however, the percentage of re￾spondents who have participated in 6-10 activities declines sharply, only to increase again with those respondents who have participated in 16 or more activities. The primary activity in which PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-6 beneficiaries participated in was meetings, followed by workshops and exchanges, which are the most common activities AMUPREV implemented in the municipalities they work on. Figure D-8. Total number of participations in AMUPREV’s activities Source: DI’s analysis based on the results of the AMUPREV survey Figure D-9. Types of AMUPREV’s activates in which respondents participated Source: DI’s analysis based on the results of the AMUPREV survey PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-7 Finally, most beneficiaries are very satisfied with AMUPREV. This confirms our findings in the qualita￾tive part of this evaluation. The overwhelming majority of AMUPREV’s direct beneficiaries were very happy with the activities implemented and the program’s efforts to strengthen their capacity to implement violence prevention programs in spite of limitations in terms of timing, capacity and re￾sources. Figure D-10. Overall satisfaction with AMUPREV Source: DI’s analysis based on the results of the AMUPREV survey The descriptive findings of this survey suggest that AMUPREV had a positive influence on both ben￾eficiaries and non-beneficiaries of the program. It is important, however, to measure whether AMUPREV had an actual impact on those individuals that were direct beneficiaries of the program. To examine the impact of AMUPREV’s program, we hypothesized that participation in AMUPREV activities increases participants capacity to develop violence prevention programs, since one of AMUPREV’s main objectives was to strengthen the technical capacity of municipalities and, in par￾ticular, of the members of the violence prevention committees to design and implement violence prevention programs. More specifically, this section examines the following hypothesis: AMUPREV’s participants have more capacity to develop and implement crime and violence preven￾tion programs than non-participants. Using a propensity matching methodology, we obtained a similar treatment and control group based on the observable characteristics (see figure below). Thus, any differences between the treatment and control groups can be attributed to the impact of the program. PSM is a robust methodology to conduct impact analysis of a policy or a program when working with observational data and in the absence of random assignment75. Like other quasi-experimental methods, it has advantages and disadvantages. A chief advantage of PSM over random assignment techniques is that it can avoid the ethical issues associated with receiving a potentially benefitting treatment and being randomly assigned out of the 75 This is known as the golden standard in impact evaluation. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-8 treatment. In the case of violence prevention activities, because they are public goods, everyone should be able to receive such benefit. Also, experimental methods, like random assignment, are very costly and take a significant amount of time. PSM, by contrast, can be done after the project has terminated and the associated costs are significantly lower than with, for example, random assignment. Some of the key disadvantages of PSM are related to the fact that the methodology can only ac￾count for observable data and therefore, the matching exercise requires as many observable char￾acteristics as possible. Moreover, after the matching is done, it is possible that hidden bias exists between the treatment and control groups, which could bias the estimates. This, however, can be tested using different techniques76 The evolution of PSM method in recent years has helped mitigate some of the main disadvantages associated with this methodology. Figure D-11. Distribution of Treatment and Control Groups before and after Matching Source: DI’s analysis based on the results of the AMUPREV survey The figure below illustrates the estimated program impact on the outcome variable based on the matched sample. The results revealed that the AMUPREV program had a positive effect on increas￾ing beneficiaries’ perceived capacity to develop and implement violence prevention programs. The lines in the graph depict the impact of the control group (left side) and treatment group (right side). The difference between the figures on each end of the line is the actual program impact. All the models illustrated show an increasing line which indicates a positive difference in favor of the treat￾ment group. In other words, all the models suggest a positive impact of AMUPREV’s program on the outcome employed for the analyses. 76 Some of the techniques used are the Rosenbaum Bounds test for hidden bias. PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT D-9 More specifically, the estimates suggest that for the restricted (basic) model (blue line in the graph￾PSM), participation in AMUPREV increases beneficiaries’ capacity by 0.5077 compared to the control group. This finding is statistically significant at the 10% level, which indicates that there is a 10% probability that the results were due to pure random change. Once we employ other matching algorithms, however, we still find positive impacts ranging from 0.08 to 0.26 depending on the model, but they lose statistical significance at the 5% or 10% level so essentially there is no differ￾ence and no further inferences can be made from this result. Figure D-3. AMUPREV Program Impacts (Treatment and Control Groups) Source: DI’s analysis based on the results of the AMUPREV survey The different methodologies employed in the analysis provide useful information that AMUPREV substantially improved participants’ perceptions of their capacity to develop and implement crime and violence prevention programs compared to the control group. 77 Program impact is measured as the difference between the treatment and control groups. In this case, the value for the treatment group is 0.75 and the value for the control group is 0.25. The difference, which equals 0.5, is the actual program impact. 0.25 0.75 0.66 0.49 0.5 0.66 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Control Group Treatment Group PSM PSM-Mahalanobis PSM-NN PSM-Caliper PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT E-1 ANNEX E: BIBLIOGRAPHICAL RE￾FERENCES Amaya, Luis Enrique. (2015). “Sistematización de Políticas Públicas de los Ministerios Rectores de la Prevención de la Violencia en el Marco de la Estrategia de Seguridad de Centroamérica (ESCA) en Dos Municipios de El Salvador,” In print, GIZ manuscript. AMUPREV Quarterly Reports 2009-2015. Bliss, K. E. (2009). Trafficking in the Mesoamerican Corridor: A Threat to Regional and Human Security. Center for Strategic and International Studies. Dudley, Steven, (2010). “How Mano Dura is Strengthening Gangs.” Insight Crime. November 21, 2010. http://www.insightcrime.org/investigations/how-mano-dura-is-strengthening-gangs Fretes Vicente and Teresa Ter-Minassian, eds. (2015). “Decentralizing Revenue in Latin America. Why and How.” Inter-American Development Bank. https://publications.iadb.org/bitstream/handle/11319/6883/Decentralizing_Revenue_in_Latin_Americ a.pdf?sequence=1 Fundaungo, “Atlas de la Violencia en El Salvador (2009-2012).” http://fundaungo.org.sv/pdf/2014/Atlas_de_violencia.pdf Gobierno de Guatemala, “Política Nacional de Prevención del Crimen y el Delito (2014-2034).” Gobierno de Guatemala, Unidad para la Prevención Comunitaria de Violencia (UPVC), Viceministe￾rio de la Prevención del Delito, Ministerio de Gobernación, “Modelo de Abordaje.” http://unescoguatemala.org/wp-content/uploads/2015/11/Modelo-deabordaje.pdf Gobierno de El Salvador, “Plan El Salvador Seguro.” http://www.presidencia.gob.sv/wp￾content/uploads/2015/01/El-Salvador-Seguro.pdf International Crisis Group, “Corridor of Violence: the Guatemala-Honduras Border.” June 4, 2014. http://www.crisisgroup.org/~/media/Files/latin-america/Guatemala/052-corridor-of-violence-the￾guatemala-honduras-border.pdf Insight Crime, Investigations. http://www.insightcrime.org/latest-investigation Katz, Charles M. and Luis Enrique Amaya (2015). “La tregua entre las pandillas como una forma de intervención sobre la violencia Implicaciones en políticas y practices.” San Salvador, Proyecto Solu￾cionES, USAID, 2015. Katz, Charles M. EC Hedberg, and Luis Enrique Amaya, (2016). “Gang Truce for Violence Preven￾tion, El Salvador.” Bulletin of the World Health Organization. June 2016. http://www.who.int/bulletin/BLT.15.166314.pdf?ua=1 Seelke, C. R. (2007). Anti-gang efforts in Central America: moving beyond mano dura? Miami: Cen￾ter for Hemispheric Policy. Seelke, C.R. (2007). “Anti-Gang Efforts in Central America. Moving Beyond Mano Dura.” Center for Hemispheric Policy, University of Miami,. P. 7. https://www.justice.gov/sites/default/files/eoir/legacy/2013/11/08/crs_antigang.pdf PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT E-2 Seelke, C.R. (2016). “Escalating Violence in El Salvador.” April, 2016. https://www.fas.org/sgp/crs/row/IN10382.pdf United Nations Office on Drugs and Crime (UNODC,. 2013). “Global Study on Homicide, 2013.” “https://www.unodc.org/documents/gsh/pdfs/2014_GLOBAL_HOMICIDE_BOOK_web.pdf World Bank, (2011). “Crime and Violence in Latin America. A development challenge.” Sustainable Development Department and Poverty Reduction and Economic Management Unit Latin America and the Caribbean Region. https://siteresources.worldbank.org/INTLAC/Resources/FINAL_VOLUME_I_ENGLISH_CrimeAndVi olence.pdf Young, L. and Hampshire, J. (2000), Promoting Practical Sustainability, Quality Assurance Group‐ PIA/OPRE, The Australian Agency for International Development. available at: www.ausaid.gov.au/publications/pdf/sustainability.pdf#search=promoting%2 practical%20sustainability’ PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT F-1 ANNEX F: DISCLOSURE OF CON￾FLICT OF INTEREST PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT G-2 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT G-3 PERFORMANCE EVALUATION OF AMUPREV – FINAL REPORT G-4 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov