DHARTE External Evaluation Report DHARTE Developing Abilities and Learning Responsibilities to Transform the Environment Award: AID-523-A-15-00004 USAID México FY14-18 Development Objective 1: Crime and violence prevention models replicated by local stakeholders Prepared by: David Guzman Matadamas, Evelyn Aguado, y Miguel Cabañas October 1, 2018 1 TABLE OF CONTENTS ACRONYMS ............................................................................................................................................................ 3 EXECUTIVE SUMMARY ........................................................................................................................................... 4 I. PURPOSE OF THE EVALUATION .................................................................................................................... 7 IMAGE NO. 1: EVALUATION KEY ASPECTS........................................................................................................................ 7 II. PROJECT BACKGROUND............................................................................................................................... 8 III. EVALUATION METHODS ............................................................................................................................. 11 A. MIXED METHODS SUMMARY..............................................................................................................................11 TABLE NO. 1. INFORMATION SOURCES AND EVALUATION QUESTIONS .................................................................................11 B. DESCRIPTION OF INFORMATION PROCESSING .......................................................................................................13 TABLE NO. 2. ELEMENTS AND CLASSIFICATION OF QUALITATIVE INFORMATION. ...................................................................14 IMAGE NO. 2: LIST OF VARIABLES ................................................................................................................................15 TABLE NO. 3 PROCESSING THE DATABASE.....................................................................................................................17 TABLE NO. 4 STRATIFICATION OF THE MAIN VARIABLES OF INTEREST. ................................................................................18 C. EVALUATION LIMITATIONS ..................................................................................................................................19 IV. FINDINGS AND ANALYSIS .......................................................................................................................... 20 TABLE NO. 5. EVALUATION QUESTIONS .........................................................................................................................20 TABLE NO. 6 CHARACTERISTICS OF THE TOTAL POPULATION OF THE STUDY DURING 2015 TO 2018. ...................................21 A. EFFECT ON BENEFICIARIES ................................................................................................................................22 IMAGE NO. 3: ERI QUESTIONNAIRE SCORE RESULTS, PERIOD 2015 TO 2018.................................................................23 TABLE NO. 7: ESTIMATE OF THE CRITICAL/SIGNIFICANT AND SIGNIFICANT FACTORS DURING THE YEARS OF THE EVALUATION'S DURATION (2015 TO 2018) IN EACH OF THE SCHOOLS..................................................................................................24 V. DISCUSSION .............................................................................................................................................. 26 IMAGE NO. 4: ERI RISK SCORE 2015 TO 2018 ...........................................................................................................27 A. INTERVENTION DESIGN MODIFICATION AND REPLICATION.......................................................................................29 B. INTERVENTION EFFECT ......................................................................................................................................30 TABLE NO. 8: LIFE SKILLS AND ACQUISITION OF EMOTIONAL INTELLIGENCE SKILLS AMONG BENEFICIARIES. ............................35 TABLE NO. 9: HIGH RISK ESTIMATION IN MULTIPLE LINEAR REGRESSION MODELS FOR THE 4 YEARS OF THE PROGRAM'S INTERVENTION. ..........................................................................................................................................................37 VI. CONCLUSIONS AND RECOMMENDATIONS ................................................................................................ 38 A. CONCLUSIONS .................................................................................................................................................38 B. RECOMMENDATIONS.........................................................................................................................................39 2 Acronyms AHM Alianza Heartland México CDCS Country Development Cooperation Strategy FY2014-FY2018 DHARTE Desarrollando Habilidades Aprendiendo Responsabilidades para Transformar el Entorno DO Development Objective ERI Individual Risk Evaluation EC Espacio Creativo (Creative Space) ECF Espacio Creativo Familia (Family Creative Space) IR Intermediate Result INJUVE Instituto Nacional de la Juventud RS Relaciones Saludables (Healthy Relationships) SDQ Strengths & Difficulties Questionnaires SAVRY Structured Evaluation of Violence Risk in Youth USAID United States Agency for International Development 3 EXECUTIVE SUMMARY INTRODUCTION This document contains research carried out between June 4 and August 29, 2018, for the purpose of answering a series of questions linked to the DHARTE intervention and the effects of its actions among students of primary and secondary education levels of the municipalities of Escobedo, Guadalupe, Juárez, and Monterrey; which are all located in the state of Nuevo León. The DHARTE Program contributes to the achievement of Development Objective 1 (DO11) of the USAID strategy, within the Pillar IV framework of the Merida Initiative2 , defined as the “Crime and Violence Prevention Models Replicated by Local Stakeholders,” as a result it relates to Intermediate Result 3 (IR3): “Increase at risk youth’s capacity to play a productive role in their communities.” The aforementioned is part of the USAID/Mexico CDCS. Intermediate Result 1, related to DO1 of the USAID strategy, focuses on increasing at ‐ risk youth capacity to play a productive role in their communities. The DHARTE Program is based on the following change theory: “if at ‐youth in vulnerable communities reach positive results risk regarding the strengthening of life skills, and their participation in crime and violence is avoided; this will increase their capacity to play a productive role in their communities.” DHARTE is based on a comprehensive multiple intervention model that includes primary and secondary prevention activities. Alianza Heartland Mexico (AHM) defines the risk factors as the accumulation of characteristics that increase the probability of an individual becoming a victim or a perpetrator of crime and violence. The work plan and methodology focus on the following evaluation questions: • Was the intervention successful in reducing both the exposure as well as the perpetration of crime and violence among the beneficiaries? • What were the key factors facilitating or preventing the decrease in violence? • Do the results related to violence prevention differ according to location or gender? • To what extent is the replication model efficient and effective? • How do the results of the activities of the intervention replication compare to those of the complete project intervention? • Is the replication of the model viable? • Are there practices or experiences from emerging projects that would be worth highlighting as best practices or lessons learned that were not captured in the replication model? • What progress was made by the project in implementing the midterm evaluation recommendations? • Did the project’s key actors value the recommendations as important elements for improving the effects on its beneficiaries? • What effect did the midterm recommendations have over the effectivity and impact of the project’s interventions? • At community, organizational, and participant levels, what are the project’s key achievements that will remain long after the end of life and context of the project? METHODOLOGY 1 See https://www.usaid.gov/sites/default/files/documents/1869/Mexico%20CDCS%202014%202018.pdf 2 The Merida Initiative is a bilateral agreement to improve security in communities in Mexico, promoting the building of strong and resilient communities that can withstand the effects of crime and violence 4 The evaluation’s focus was on analyzing the effects of the intervention carried out with the youth, the progress of activities, and the results of the replication strategy; as well as the sustainability of results of some of the components of the DHARTE Program on the participants. AHM defined evaluation questions that were discussed among the organization’s personnel and the external evaluation team for designing the relevant methodology to uncover the findings that are described in the corresponding sections. Given the characteristics of the information necessary to answer the evaluation questions, two kinds of processes were carried out focused on the types of analysis. In the case of qualitative information, besides the bibliographic and in-house review regarding the program implementation, the DHARTE model guides, as well as the quarterly and annual reports, were analyzed; and a consultation with the Program staff was carried out to get a detailed knowledge of the operation of the workshops and replication strategy. Regarding the quantitative information, in regards with the DHARTE Program’s Monitoring and evaluation mechanism, a secondary analysis of the databases that were provided by the Heartland Alliance International Specialist was carried out. The data process involved defining the operationalization of the predictive and the scripted analysis, focusing on the processes, as follows 1) Approach to the coding language; 2) Processing of databases; and 3) Identifying variables of interest. FINDINGS The quantitative evidence showed that the intervention has a positive result on the beneficiary population. A marginal decrease in the exposure to violence throughout the life of the Program was observed. The results on measuring risks show a tendency towards risk reduction in those cases classified as “high risk.” Regarding the protection factors, that is, those related to decreasing violence, the youth mentioned, to a greater degree, that those that allow them to face situations of violence are related to family, their neighborhood community, and their schoolmates. Specifically, the factors linked to family are related to conversations with family members, confidence in the youth’s capacities, support in decision-making and reaching agreements. Regarding factors related to the community (neighborhoods), those that facilitate a decrease in violence in the context in which youth live, are the mediation among neighbors or “neighborhood justice3” to solve problems, the neighborhood councils, as well as reporting to authorities when there is any crime and notifying adults when they have a problem. Regarding the factors related to school, the youth mentioned that communicating with their peers and coexisting with their schoolmates to solve problems among themselves, mutual help and respect, as well as confidence in establishing a dialogue with an aggressor schoolmate in case of bullying were helpful. Regarding risk triggers, that is, factors related to the increase of violence, youngsters mention other familiar aspects, such as obstacles to face violent situations, including lack of communication— absent father or mother, lack of attention from parents, and/or problems of addiction in the family. It is important to highlight that the students mentioned on several occasions the violent situations they go through, such as fights; disagreements and discussions; verbal, physical violence, and the ones caused under the influence of alcohol. The community, to a lesser degree, can be an obstacle—and a trigger—when there are thefts and drug trafficking, lack of surveillance, and insecurity in general. Likewise, youth identify bullying, beatings among schoolmates, and discrimination as elements in common that hinder the decrease in violence surrounding them. The lack of teachers’ attention in 3 The figure consists of an inhabitant of the Community serving as a local representative who facilitates conflict mediation in the community http://portal.monterrey.gob.mx/pdf/reglamentos/1/22%20Reglamento%20de%20%20Jueces%20Auxiliares%20del%20Municipio%20de %20Monterrey.pdf 5 the face of aggressive events against students can result in lack of confidence and communication as factors that do not contribute to offer a solution to their problems with their peers and doesn’t help them face violent situations. The interviewed adults felt that risky situations and potential violence affect men and women equally. Both sexes are considered aggressive; young women are found to be involved in gangs, and while involvement is more frequent in young men, respondents felt the difference has been decreasing since both sexes have similar emotional and family problems. Respondents also felt that differences exist by sex because males are more “ill tempered” or have a greater exposure to situations of violence, while women are more vulnerable to becoming victims of violence or crime. Regarding replicating the model systemized under the DHARTE Program intervention, the INJUVE, and Monstritos organizations considered that the workshop for “Replication Transfer” has allowed them to identify important aspects and needs for implementing the DHARTE Model. When asked about the sites and implementation plan, however, the organizations responded that they are still in the planning phase and didn’t mention having chosen intervention sites or having an operational team prepared for the implementation. Regarding the matter of financial resources, there were no comments supporting an intervention of the Model replication in the near future. The quantitative evidence shows the effect of the intervention. At a community level, the DHARTE effect cannot be assumed to permeate to peer teenagers residing in nearby populations. During interviews with school personnel, there was no testimonial regarding activities focused on assisting the neighboring communities/populations residing near the schools. The intervention’s effect, due to its focus, does not permeate to the other school grades in the case of primary schools; in other words, the results obtained in the 5th and 6th grades of primary school do not allow inferencing that the positive effects have permeated to the other parts of the school community. In the case of the secondary schools, because they will not continue with active workshops in the near future, the effect obtained in decreasing the risks among the beneficiaries will not continue helping the school community. The evidence shows that the contents included in the workshops, as a medium to acquire life skills, will foster the permanence of self-awareness, self-control, communication, confidence, and empathy skills among the beneficiaries of the DHARTE Program. Also, it is important to highlight that the most important achievement among participants is the decrease in RISK associated with “high-risk” triggers regarding self-destructive behaviors. 6 I. PURPOSE OF THE EVALUATION The purpose of the evaluation is to measure the intervention’s effect, given the relevant findings among the beneficiaries, as well as the group of individuals involved, also considering the potential replication of the model. The contents of this document include the advances concerning the data analysis (content analysis– semi-structured interviews and participation instruments;—parametric, bivariate analysis; multiple regression models—ERI questionnaire). The DHARTE program external evaluation team carried out the information gathering with key informants, making phone calls and collecting data on site between June 27 and July 20, 2018; see Attachment 1 Work Plan and Methodology for more information. The activities that appear in the schedule are grouped as follows: a) Reviewing documentation of relevant information related to crime and violence in Mexico. i. Reading of the Program’s relevant documents. Results: adaptation of questionnaires and proposed methodology4. ii. Program data review (quantitative information gathered ERI, SDQ). Results: preparation of the List of Variables and Analysis Plan. b) Qualitative data gathering and visiting with the implementation team; which considered two modalities; the remote data gathering (via phone calls), between June 27 and July 12, 2018, and the in-person data gathering between July 16 and July 20. i. Processing qualitative data from the key-informant’s interviews. This activity was carried out between June 27 and July 20, 2018. c) Analysis of qualitative and quantitative information, related to the evaluation questions, breaking down information according to territory location (school/type), and sex. i. During the preparation of the Methodology and Work Plan, the vision included breaking down data according to age and grade. However, during the (univariate and bivariate) exploratory analysis, the focus was on presenting the data in tables classified into Schools and Sex categories. Following the research questions, the study used a mixed-method approach, via qualitative and quantitative data analysis, to answer the questions. Image No. 1: Evaluation Key Aspects 4 Product 1, delivered on July 3, 2018. (See Annex A) Reviewing documentation (project documents) Data gathering tools (mixed methods) Analysis using a quantitative and qualitative approach Beneficiary Effect Model Replication Sustainability 7 II. Project Background AHM started to work on initiatives for the prevention of violence in Mexico, by way of the EJEMPLAR project, financed with the USAID’s generous support. The DHARTE Project is an initiative to continue the work carried out with teenagers, by promoting activities based on art therapy. The initiative used a comprehensive approach for social prevention of violence, and the actors involved were students, fathers/mothers, and school personnel; all of whom participated in workshops and activities in the period between 2015 and 2018.5 The DHARTE model includes art therapy as a way for promoting healthy, interpersonal and family relationships using artistic expression. Likewise, the intervention includes building life skills, such as problem solving and leadership among youth, and entrepreneurship. Youth in Nuevo León must withstand the effects of crime and violence as part of their everyday context; therefore, the goal of this intervention focusing on primary and secondary schools is to help the youth of Nuevo León to develop skills. According to the figures of reported victims—per 100,000 inhabitants—concerning high impact crimes reported by Observatorio Nacional Ciudadano de Seguridad for 20176, the state of Nuevo León’s current context is in the 21st place of 32 states regarding intentional homicide by location with a 1.55% in accordance with the rate per 100,000 inhabitants. Regarding manslaughter, Nuevo León is in the 11th place with 3.36%. The figures regarding kidnappings place the state in the 13th place with 2.86%. It is worth mentioning that this state ranks second nationwide in the number of victims of extortion crime with a rate of 10.96%. On violence against women, rape counts as .78% of investigation files (dockets in process) for rape per 100,000 inhabitants, which indicates the state is in the 12th place for such crimes. Regarding robberies, the rates per 100,000 inhabitants in relation to the investigation files for robbery with violence, rank the state of Nuevo León in the 27th place among the 32 states with a rate of 5.18%. Likewise, car-theft rates represent 4.65% for the state and places it in the 25th Pl. of the country. The rates for investigation files regarding home robberies occupy the 19th place for Nuevo León with 3.84%. Regarding business robberies, Nuevo León’s rate for investigation files is 4.63%. Robbing people passing by represents 2.4% and the 9th place for the state in comparison with the rest of the country. According to the organization Semáforo Delictivo7, the Municipality of General Escobedo’s figures are above the municipal historical average in extortion, business robberies, injuries, rape, and domestic violence crimes. The Municipality of Juárez shows figures above the historical average for business robberies, and the Municipality of Guadalupe’s figures increased above the historical average for domestic violence and homicide crimes. Lastly, the municipality of Monterrey shows an increase in the historical average for crimes of violence and domestic violence. In summary, the municipalities in which the DHARTE Program intervenes, with the exception of Juárez, show increases related to domestic violence. The information gathered from key informants—mothers, school staff, DHARTE staff—that interact with the external evaluation team, reflects the current situation that youth face in their locations in the State of Nuevo León. 5 Taken from file “Características intervención” 2018 p.39 [2018 Intervention characteristics] 6 For more information please read the complete report found at the website http://onc.org.mx/wp-content/uploads/2018/04/Enero￾18.pdf 7 Semáforo Delictivo’s webpage performs a monthly historical measurement since 2015 using the data provided by Secretariado Ejecutivo Nacional de Seguridad Pública (National Executive Secretariat of Public Security) establishing a 25% reduction goal. Their methodology includes identifying the municipalities in red: figures above the historical average, in green: under the goal, and in yellow: between the average and the goal. The crimes measured are homicides, kidnappings, extortions, auto thefts, household robberies, business robberies, injuries, rapes, and domestic violence. For more information, see http://nl.semaforo.com.mx/ viewed on August 8, 2018 8 The beneficiary population in the metropolitan zone of Monterrey—in particular in the Municipalities of Juárez and Escobedo—mention the presence of organized crime, gangs, family disintegration, and insecurity. They also mentioned the exposure of youth to risk situations such as teenage pregnancies, the need to work at a young age, and living in an environment in which public services are scarce or non-existent. The crimes related to drug use and drug trafficking, prostitution, and assaults occur in the youth’s everyday life. The family mothers mention that alcohol smuggling (“alcoholeo”) starts to become part of the teenagers’ everyday life, which initiates them into organized crime. This becomes relevant because students are recruited by other youngsters (linked to crimes) outside the school. In these last months, the drug-use and drug-trafficking situation resulted in some schools carrying out backpack checks because of the neighboring sites distributing addictive substances in the schools’ peripheral areas. Finally, violence against women has begun in some nearby communities in the form of femicides and kidnapping. Gangs and violence have evident consequences for the youth, for example, on occasions youngsters cannot move around areas near the school because of the fights among gangs; other times, the gangs damage school facilities, which generates a feeling of insecurity among the school community. Even more so, the lack of police surveillance, and the distrust in the police forces and other authorities further worsens the violence situation and weaken the rule of law in these areas. “There are lots of gang activities, the school is on a mountain, those who need to go up cannot do so and vice versa because of the fights. The youth cannot participate in school work because they can’t move around. They don’t feel safe there, they live nearby” interview with School Official “The situation of insecurity is so critical that some coordinator positions and an assistant principal position are empty because nobody wants to go work there” interview with School Official “In the specific case of Juárez, it is complicated because there is a small police force; it seems as if there is no law in Juárez” interview with AHM Staff The recent factors causing vulnerabilities in the youth are as follows: teenage pregnancies, as well as the students’ early labor insertion—particularly in the areas of cleaning services and construction work. Likewise, the lack of access to public services and to recreational spaces in the communities near the schools becomes an important matter, because this causes the youth’s recreational time to be compromised by the risk factors to which they are exposed. “They don’t even have a movie theater or town square with trees in Juárez; they have to take up to two trucks and travel for more than an hour to reach any recreation place.” interview with AHM Staff The youth’s socialization with their teachers and students is an important factor that contributes to decreasing violent or risky behaviors. The interviewed adults mentioned that violence in the school community—fights and bullying among schoolmates, fights between parents and teachers, and teachers towards students, or lack of supervision—result in an unhealthy environment. Likewise, the lack of personal skills that the students mention as obstacles in decreasing the violence surrounding them, and the pressure to commit violent actions among them, or towards themselves, generates important risks for the students’ well-being. “There is peer pressure, girls have issues with craving acceptance, and with pressure to self-injuries. The boys have issues concerning physical fights among themselves and threats with objects against someone else. The girls express their violence against themselves, and the boys show off violence.” interview with AHM Staff “There is a lot of aggressiveness among students, three cases of attempted suicide have been detected among students in 6th grade.” interview with School Official “There is bullying by students and teachers. I reported it to the social worker, and the teachers and principal didn't do anything until the situation escalated to beatings. They had many nicknames (for 9 my son), they picked on him because his mother took him to school, or they say obscene things to his sister in front of the student.” interview with Mother of a family. Finally, the youth’s center of socialization in adopting cultural patterns resides in the family. In this respect, the school personnel and the DHARTE Program staff mentioned that the youngsters are witnesses of the violence and the drug use, as well as of family disintegration that happens in their homes, even viewing it as a “normal” part of reality. “Seeing the parents and other people using drugs.” AHM Staff “This situation is difficult because the children's family is immersed in this context; they already see it as a job, a way of life, and part of their culture; they see no difference." interview with AHM Staff The lack of parental supervision due to work issues, family disintegration, or parenting styles, are relevant topics, because young people adopt risky behaviors based on a socialization in which limits and corrective disciplining play an important role in avoiding their participation in situations of violence. “Another risk is poor upbringing; there is no supervision because both parents are working; there are cases in which siblings take care of each other, and in few cases there are caretakers.” interview with AHM Staff The family mothers also consider that these factors directly impact on the youth and their involvement in situations of violence and crime. In particular, mothers mention that youngsters engage in risky behaviors because they are in families in which the parents have long work shifts that foster cohabitation problems with their children, and/or lack of cohabitation; also, because they have no limits and have a negative scale of values. “Long workdays resulted in cohabitation problems. Children need someone listening to their problems; the parents are very stressed, and they take it out on their children. Or the parents work and don’t notice what’s happening.” interview with a Mother of a family Finally, one of the causes of youth involved in violence or crime is related to the stage in which children explore and define their identity, which is influenced by observing adults or peers who they admire, resulting in a motivation to imitate the behaviors with which young people are familiar. “Organized crime, admiring drug dealers, structures and preparing parents, inter-generational learning. A climate of children feeling isolated at school; they succumb if there are no feelings of attachment towards the school, nor any safe place at home, and identifying with crime.” interview with AHM Staff 10 III. Evaluation Methods A. Mixed Methods Summary DHARTE’s final evaluation was designed considering analyzing the outcome of the intervention carried out with the youth, as well as progress of activities, and results of the replication strategy. Also, a written analysis of the contents of the Program’s supporting documents and questionnaires was carried out for the Heartland staff to identify the inclusion of the evaluation’s mid-term recommendations of the DHARTE Program’s scheduled activities in the third year of operation. Additionally, a special analysis regarding the sustainability of results of some of the DHARTE Program’s components regarding students was performed. AHM defined evaluation questions that were discussed among Heartland’s personnel and the external evaluation team to design the relevant methodology in order to discover the findings that are detailed in the corresponding sections herein. Table No. 1. Information sources and evaluation questions Information Source8 Evaluation Questions ERI: risk measurement scale Was the intervention successful in reducing both the exposure as well as the perpetration of crime and violence among the beneficiaries? AHM Personnel Interviews Teenager Workshop What were the key factors facilitating or preventing the decrease in violence? Interviews (AHM Personnel, fathers and mothers, school personnel) ERI Survey Teenager Workshop Do the results related to violence prevention differ according to location or gender? Combination of the Monstritos and INJUVE interviews and documentation review To what extent is the replication model efficient and effective? How do the results of the activities of the intervention replication compare to those of the complete project intervention? Is the replication of the model viable? Are there practices or experiences from emerging projects that would be worth highlighting as best practices or lessons learned that were not captured in the replication model? Reviewing documentation: Annual work plans. Comparing operating plan and testimonials from the implementation team Semi-structured interview What progress was made by the project in implementing the midterm Evaluation recommendations? Semi-structured interviews Did the project’s key actors value the recommendations as important elements for improving the effects on its beneficiaries? In-house analysis, reviewing implementation of recommendations What effect did the midterm recommendations have over the effectivity and impact of the project’s interventions? Interviews (AHM Personnel, fathers and mothers, school personnel) At community, organizational, and participant levels, what are the project’s key achievements that will remain long after the end of life and context of the project? The work plan developed and approved by AHM is in line with the research questions; which required the use of mixed methods to answer with the frequencies and arguments in line with the “nonexperimental” research design. 9 8 To obtain more information regarding the interviews carried out with key informants, see Annex B. Regarding the evaluation workshops with young people (students), see Annex C. 11 1. Qualitative Evaluation Methods The qualitative analysis began with an in-house and bibliographic review regarding the program implementation. Afterwards, the DHARTE Model guides, quarterly reports, annual report were analyzed, and a consultation with the Program staff was carried out to obtain details regarding the operation of the workshops and replication strategy. The qualitative research instruments took the instruments used in the midterm evaluation of the program as a reference, for the purpose of answering the evaluation questions. The primary information sources were collected via semi-structured interviews carried out with key informants such as school officials, DHARTE Program staff, family mothers, and personnel of organizations selected to carry out the program replication. Participative methods of evaluation were considered relevant for this analysis due to the prior interaction or socialization of the students with group expression and introspection activities. Therefore, this evaluation applied/included? a four-hour workshop with teenagers participating in the camp activity carried out by Heartland in Monterrey on July 16 to 17, 2018, to identify the young people’s general context and the effects it had on reducing violence and risky behaviors. The information concerning the progress of the replication activities was collected during semi-structured interviews to implementing organizations, such as Monstritos A.C. and INJUVE Nuevo León, at their offices located in the Municipality of Monterey. Finally, an analysis of the administrative documents and a gathering of information based on questionnaires sent by electronic mail was carried out with AHM staff, to identify the inclusion of the midterm evaluation recommendations of the DHARTE Program. 2. Quantitative Evaluation Methods The research instruments used correspond to a combination of mixed methods. Given the characteristics of the evaluation questions, the information sources were identified, and in accordance with the “Work Plan and Methodology”10, the levels of the variables that will present the descriptive statistics consider four categories as seen in Table No. 2. Given the processes associated with cleaning, coding, and statistical analysis, the three steps carried out by this evaluation are described in the section “Information Processing.” The aforementioned steps are, as follows: 1) Approach to the coding language. 2) Database processing. 3) Identifying variables of interest, that illustrate with specific detail the effort carried out. 9 For further information see Annex A 10 See Annex A. 12 B. Description of Information Processing The two types of methods were processed differently according to the characteristics of the information. In the following paragraphs, there is a detailed description of the processes developed for each one of the methods. As previously mentioned, in the case of qualitative data, the evidence gathering process consisted of carrying out semi-structured interviews by phone calls with several informants—such as family mothers, school officials, and the DHARTE Program’s staff—for the purpose of identifying the situation they’re currently experiencing related to violence and risk, their participation, perception of the program, as well as the effects it had on students to reduce violence. Additionally, an information gathering was carried out on-site during the evaluation team’s field visit. In this regard, a participative evaluation workshop was carried out with the DHARTE Program’s young participants to identify the support mechanisms and triggers to reduce violence and identify the effects it had on them during this intervention. Lastly, the officials of the organizations replicating the model were interviewed—Monstritos and INJUVE—to gather information related to the replication model. The gathered evidence, as well as the answers given by each of the interviewed, was detailed in a MS Excel11 file organized by question and type of informant. The observations and material gathered from the workshops were transcribed and classified in the MS Excel file. The same procedure was carried out to systematize the data gathered in the interviews with the replication organizations. The data were later systematized, the responses were grouped and recorded based on their similarity; and as a result of said association, analysis categories were established. Using a Nick Duncan inductive process, in which as a result of the coding, grouping, and frequency count of the contributions given by the key informants, analysis categories were defined to align and relate the findings discovered with the evaluation questions. The result of the classification was the preparation of organized matrices, which were the groundwork for analyzing associations among data, in order to answer the questions in the report’s findings section. The general categories can be observed in the elements to be described according to each one of the items to identify the findings corresponding to this evaluation. The effects on young participants were classified mainly according to the personal intelligences summarized by Salovey, cited by Daniel Goleman12, based on the personal intelligences proposed by Horward Gardner on Multiple Intelligences. The five main competences are (Goleman, pp 44) as follows: 11 See Annex E named “Quantitative and Qualitative Analysis Input” 12 Daniel Goleman is a professor in psychology at Harvard University, author of the book titled Inteligencia emocional [Emotional Intelligence]. His fundamental principles are based on the five skills that each human being needs to be successful in the Western world. The referenced skills are divided into two. Personal skills are those such as emotional self-awareness, or self-knowledge as the skills we need to recognize our emotions and their consequences; the skill of emotional self-control or self-management as the capacity of recognizing and controlling the emotions; self-motivation as the capacity to direct the emotions towards an objective for the purpose of maintaining attention on goals instead of obstacles. The social skills are empathy understood as the capacity to recognize the emotions of others; and the social skills such as capacity to recognize the feelings of others and produce emotions in them, are mainly characterized by the ability to negotiate, work in teams, display leadership, among others. Goleman Daniel (1995) Inteligencia Emocional. Editorial Kairos. New York, United States of America. 13 Aware of one's own emotions (self-awareness): this competence is related to your inner knowledge and to the capacity of recognizing your own feelings when they appear. This skill is the ability to understand one’s inner feelings and moods, and it aids in identifying these emotions to better lead other individuals’ lives when making decisions. The findings of the evaluation are described jointly as self-awareness and self-control, in the sense that the youth increased their ability to identify and control the emotions towards displaying tolerance inside and outside the school. The capacity to control emotions (self-control): that is, the consciousness of one’s inner self as the basic ability that allows human beings to control their own feelings and adjust these to the relevant moment. This skill refers to one’s own capacity to remain calm by managing anxiety, sadness, and anger. Table No. 2. Elements and classification of qualitative information. ITEM ELEMENTS INFO SOURCE FACTORS Sense of self= “aspirational” belonging Economic needs= insufficient family income Crime= incidence/context (organized crime, theft/robberies, drug trafficking and use). Teenage pregnancy Gangs= street fights and vandalism (destroying public property) Child labor Family problems= violence, lack of supervision, addictions (alcoholism), divorce, family values, styles of upbringing (limits, discipline, etc.) Insecurity: related to the local context Lack of access to public services Violence against women Intra-school violence Interviews parents, school officials, and DHARTE Program staff team members. Workshops carried out with the students FACILITATORS13 Family, community, school/schoolmates, school/teachers, social/personal skills Workshops carried out with the students OBSTACLES 14 Family, community, school/schoolmates, school/teachers, social/personal skills EFFECTS/RESULTS Confidence=self-confidence Self-Awareness= Teamwork= ”collaboration among peers” Creativity= forms of expression Communication= interacting with peers, teachers, parents. Assertive intent when transmitting a message. Learning cohabitation/social skills Interviews parents, school officials, and DHARTE Program staff team members. Workshops carried out with the students DESIGN Clarity of the replication model In-house analysis REPLICATION Adaptation/customization Transmit the model (replication process design) Sustainability Interviews with the replication organizations In-house analysis 13 Are defined as Facilitators, because the caused effect is considered positive. 14 They are defined as Obstacles because their effect is considered negative. 14 The capacity to motivate oneself (self-motivation): control of one's emotions and moods and their subordination to an objective is essential in achieving attention, motivation, and creativity. This capacity allows delaying gratification and controlling impulsiveness when seeking to achieve goals. The findings on self-motivation are aligned to this conceptual category in regards to the capacity the youth have in staying away from “bad influences” and getting involved in risky situations, as well as seeking decisions different from those which the youth know. The capacity to recognize emotions in others (empathy): this is based on emotional self-awareness, and it is the fundamental ability to engage socially with others, as well as altruism. Empathetic persons are normally tuned-in to the subtotal social signals that indicate what others need or want. The empathy described in this report corresponds to this conceptual foundation, in particular to the capacity to respect and resolve conflictive situations based on valuing other people's feelings. The capacity to control relationships (social skills): this is based on the capacity to interpret and be able to establish relationships with other people, and it includes the ability to control relationships with others through popularity, leadership, and interpersonal efficacy. The findings in this document regarding social skills refer to the youth’s teamwork with their schoolmates, using assertive communication with their teachers, parents, and faculty. In addition to the findings about the program’s effects in the emotional sphere, an additional category was identified (unrelated to emotional intelligence) which describes the cognitive learnings regarding raising awareness and action strategies related to reducing violence. Regarding the processing of quantitative information, the evaluation team focused on the research questions oriented towards the external evaluation analysis. The processing and analysis included the following three steps: 1. Approximation to the coding language Quantitative analysis requires systemizing the raw databases (original data). An Excel15 file was handed over as part of the Work Plan to facilitate data processing. This file contained five tabs Image No. 2) containing data gathered by various tools between 2015 and 2018. The following paragraphs describe the structure of the Excel file and each tab according to its elements: Image No. 2: List of Variables 15 See Annex E named “Quantitative and Qualitative Analysis Input” 15 Raw Database: this is the information data editor, and it is shown as a marker located at time ZERO of the information analysis and processing. ERI: shows the order of the data naming categories that correspond to the operationalization of this instrument’s measurements, described as follows: PHASE= year {2015, 2016, 2017, and 2018} VARIABLE= variable coding. Example: N22015 LABEL= description of the variable Example: N22015 “2. Was part of a group or organization_ERI2015” CLASSIFICATION OF THE VARIABLE= definition of the type of variable according to its characteristics (numerical or qualitative). The options in definitions are: Numerical and Qualitative (dichotomous, nominal, ordinal). INSTRUMENT OPERATIONALIZATION= considers the type of response generated by each type of variables. For example: open question, dichotomous (Yes, No), category (always, sometimes, never). DATABASE OPERATIONALIZATION= considers the type of response corresponding to each variable when it was included in the database. For example: a continuous interval with response options from 0 to 9, category from 0 to 2, dichotomous (sex= F, M). DOMAIN= corresponds to the classification of the values according to a theme axis. For example: support network, family, individuals, etc. OBSERVATIONS= corresponds to the identification of situations related to data processing. For example: coding to be discussed with Heartland, the questionnaire’s number of question categories in the database and Specify variables/labels don’t match SDQ: this tab shows three components describing how measurements were operationalized via the tool. PHASE= year {2017, and 2018} VARIABLE= variable coding. Example: N2ScoreHiperSDQ2017 LABEL= description of the variable Example: N2ScoreHiperSDQ2017 “N2_I am restless, hyperactive_SDQ2017”. EspCre_RelaSal: this tab shows three components describing how measurements were operationalized via the tool. PHASE= year {1, 2 y 3}. VARIABLE= variable coding. Example: Escuela_ECY2. LABEL= description of the variable Example: School_ECY2 “School, Creative Space Y2”. Risk Evaluation Factors: the scales of significant and critical factors are defined. See Annex E, “Quantitative and Qualitative Analysis Input,” for more details on the process used to modify the databases which is linked to the Do file. File “Analisis de datos.do”. 16 2. Database processing16 The processes for quantitative data analysis are described below using examples. The purpose of this section is to provide transparency to the cleaning and labeling of variables, which is shown in the following Table No. 3: Table No. 3 Processing the database Databases Associated Processes Observations DHARTE_Clean_ Master_Data_ ForFinalEval.dta Original database: was not compatible with the Stata 13 software. The external evaluation team made the request to the HAI database administrator for the data version in a file compatible with the software Stata 13. DHARTE_Clean_ Master_Data_ ForFinalEval_ST13.dta ERI2018Data_ CleanStata13.dta SDQ2018_T2 CleanStata13.dta Database second version in stata 13, the exploratory analysis was carried out. For observations gathered in 2017 and 2018, this information was integrated for its consideration in the overall analysis because it contains ERI and SDQ information in its last stage of the intervention. General revision for data cleaning: recategorizing variables; relabeling (databases, variables, and response categories); imputation (process to discard data without affecting the information contents and the possible statistical inferences); after the exploratory analysis, variables were eliminated because of “missing data”, and those values without response categories were eliminated from variables of interest. Example variable “N26ThatWas2015” Stata Data examples: .encode School2015, gen(School_2_2015) .codebook School_2_2015 .tab School_2_2015 .recode School_2_2015 (1=1 "Arcadio Lozano")(5=2 "Niños Heroes")(6=3 "Rafael Ramirez")(7=4 "Sec.2 Forum Universal")(8=5 "Sec. 3 Amado Nervo")(10 15=6 "Sec. 6 Historians")(9 14=7 "Sec. 43 Melchor Ocampo")(12 17=8 "Sec. 71 Leandro Valle")(13 18=9 "Sec. 88 Prof. Maria Lavalle")(2 3 4 11 16=.), gen (School_category_2015) tab School_category_2015 Note: see Do file “limpieza_retiquetado_imputacion.do”; the process was carried out for all the observations reported under the information sources ERI (2015, 2016, 2017, and 2018), SDQ (2017 and 2018), to consolidate a database. DHARTE_ERI_3eros201 7_Stata13. dta Merge_base_completa. dta Final database with all the information required for the complete analysis The outcome of the work with the aforementioned databases was the consolidation of a General Database “Merge_base_completa.dta” for the descriptive and inferential analysis named the analysis program found in the Do file “Evaluacion_final.do”. 16 Taken from the document: Procesamiento de Bases de Datos de David M. Kroenke. (Fundamentos, Diseño e implementación). Pearson Educación, Mexico 2003. Cap.II, Page 43–46. 17 3. STEP 3: Identifying variables of interest17. The quantitative data collection stages corresponded to the lifetime of the Cooperation Agreement between AHM and USAID. Measurement was performed via the ERI and SDQ questionnaires, which are described in the following paragraphs18: • ERI - applied both in primary and secondary schools during school years 2015-2016, 2016- 2017, and 2017-2018. This instrument was adapted from the Structured Evaluation of Violence Risk in Youth (SAVRY) to assess risk of violence risk, including instructions on how to use the instrument, its scale, application process, administration, and data capture19. • SDQ - applied both in primary and secondary schools during 2017 and 2018. The specific objective of this instrument is to assess life skills20. Because of its recent use in relation to the lifetime of the DHARTE Program, the analysis presented in the evaluation report was not included. Once the evaluation list of variables was structured, the main variables were identified to help answer the evaluation questions. Below, the stratification of the main variables of interest is shown. Table No. 4 Stratification of the main variables of interest. Stratum Description Schools, intervention units Intervention Units= {Schools distributed among the municipalities of: Escobedo (Secundaria No. 3 Amado Nervo), Monterrey (Primaria Prof. Arcadio Lozano Martínez, Secundaria No. 43 Melchor Ocampo, Primaria Rafael Ramírez), Guadalupe (Primaria Niños Héroes), Juárez (Secundaria No. 2 Fórum Universal de la Culturas 2007, Secundaria Técnica No. 71 Leandro Valle, Secundaria No. 6 Historiadores de Nuevo León, Secundaria Técnica No. 88 Prof. María Lavalle Urbina)} Gender21 Sex {male, female} Education Level Grade= {5th primary, 6th primary, 1st secondary, 2nd secondary, and 3rd secondary} Program School hours= {morning classes, evening classes, extended school hours} 17 Daniel, Cauas (2015) Definición de las variables, enfoque y tipo de investigación. Editorial Universidad Nacional de Colombia. Page 77. 18 See Annex E, Quantitative and Qualitative Analysis Input: List of Variables. 19 For more information see the document “Inputting Data Guide of ERI_May 2017” 20 Among which the instrument was customized to obtain information related to the following: a) Communication skills/healthy relationships; b) Decision-making and problem solving; c) Building empathy; d) Cooperation, teamwork; and e) Emotional control skills. 21 Given that biological sex is an anatomical condition, the category “gender” used to define a stratum of interest, is conceptually equivalent to the identities of man and woman, without considering the existing social construction in regards to the “gender.” 18 C. Evaluation Limitations This evaluation process occurred concurrently with the end of the school year in Nuevo León. Therefore, data on program design and implementation could not be triangulated on-site because there was insufficient time to carry out a field visit to the program schools. Consequently, the triangulation of key informant interview data was not performed, but checked using observation of program activities. The evaluation team understands that this did not affect the key informant interviewee selection from the population of school staff and family, and that key informant interviewees constitute typical or representative cases. The program staff enabled the interaction with key informants (list of school personnel, parents, and DHARTE staff), and allowed the facilitation of the workshop with youngsters at a summer camp (offered by AHM during the week of July 16 to 20, 2018), where participative evaluation instruments were used22. Another limitation refers to applying the ERI questionnaire. In particular, students of the participating schools may not have been included in data ERI data collection because of family situations which were not under the control of the programming team. Therefore, it may be inferenced that the samples each year vary because: - Students changed of residence. For example, because the parents of the family changed jobs. - Lack of interest in participating in the program. Such as, for example, the parents’ lack of support to continue participating in the program's activities. The participation of the teenagers also decreased in some schools because of the particular circumstances which resulted in the interruption of the AHM staff work, for example in Escuela Secundaria No. 69. 22 See Annex D, Field Visit Report. 19 IV. FINDINGS AND ANALYSIS The sections of this chapter outline the evaluation findings in relation to the evaluation questions and information sources, as can be seen in Table No. 5: Table No. 5. Evaluation questions Findings Section Evaluation Questions Information Source Effect on Beneficiaries Was the intervention successful in reducing both the exposure as well as the perpetration of crime and violence among the beneficiaries? ERI: risk measurement scale What were the key factors facilitating or preventing the decrease in violence? AHM Personnel Interviews Teenager Workshop Do the results related to violence prevention differ according to location or gender? Interviews (AHM Personnel, fathers and mothers, school personnel) ERI Survey Teenager Workshop Replication To what extent is the replication model efficient and effective? Combination of the Monstritos and INJUVE interviews and documentation review How do the results of the activities of the intervention replication compare to those of the complete project intervention? Is the replication of the model viable? Are there practices or experiences from emerging projects that would be worth highlighting as best practices or lessons learned that were not captured in the replication model? Intervention Design Modification What progress was made by the project in implementing the midterm evaluation recommendations? Reviewing documentation: Annual work plans. Comparing operating plan and testimonials from the implementation team Semi-structured interview Did the project’s key actors value the recommendations as important elements for improving the effects on its beneficiaries? Semi-structured interviews What effect did the midterm recommendations have over the effectivity and impact of the project’s interventions? In-house analysis, reviewing implementation of recommendations Intervention Effect At community, organizational, and participant levels, what are the project’s key achievements that will remain long after the end of life and context of the project? Interviews (AHM Personnel, fathers and mothers, school personnel) What behavior changes occurred (intermediate effects) in the beneficiaries? Workshop (participative evaluation) What final effects will remain at a community level? Workshop (participative evaluation) and, if possible, recover something via information from the parents. 20 Characterization of the student population is based in the strata of interest (primary–secondary schools and sex) as mentioned in section “Description of Information Processing.” The quantitative analysis was performed using the initial categories of the ERI questionnaire. The descriptive analysis allowed for observation of study population characteristics in each of the years of the intervention. During the years of the program's duration, there was information from the nine schools included in the evaluation: baseline in September 2015 (n=2992), end of Year 1 in June 2016 (n=2821), end of Year 2 in June 2017 (n=1680), and end of Year 3 in June 2018 (n=828). In years 2015 and 2016 the average age was 13 years old and the response rate was higher for the students in the three grades of secondary school and the morning classes. Below, Table No. 6 shows important aspects regarding the characteristics of the students who participated in the study. Table No. 6 Characteristics of the total population of the study during 2015 to 2018. Baseline 2015+ (n=2992) 2016+ (n=2821) 2017+ (n=1680) 2018+ (n=828) P Value *** School, % Percentage 0.047 Primaria Prof. Arcadio Lozano Martínez 0.90 1.67 1.47 2.90 Primaria Niños Héroes 0.77 1.24 0.98 1.57 Primaria Rafael Ramírez 1.24 1.70 0.80 2.05 Secundaria No. 2 Fórum Universal de la Culturas 2007 24.20 24.35 28.83 24.76 Secundaria No. 3 Amado Nervo 10.49 10.24 6.44 6.52 Secundaria No. 6 Historiadores de Nuevo León 11.90 11.77 11.53 9.18 Secundaria No. 43 Melchor Ocampo 9.59 11.20 10.67 16.67 Secundaria Técnica No. 71 Leandro Valle 24.97 24.35 26.32 26.09 Secundaria Técnica No. 88 Prof. María Lavalle Urbina 15.94 13.47 12.94 10.27 Gender (years of age) % Women 51.47 51.29 53.25 51.79 0.016 Men 48.53 48.71 46.75 48.21 Age (years of age) % 𝑥𝑥̅= 13.02 SD 1.13)* 𝑥𝑥̅= 13.52 SD 1.14* 𝑥𝑥̅=12.02 SD 0.46* 𝑥𝑥̅= 14.04 SD 1.1* 9 0.02 - 11 3.68 3.93 1.67 3.02 12–14 78.31 71.66 72.02 60.51 15 or more 18.01 23.41 26.31 36.47 Program, % Morning Classes 0.01 56.28 57.47 56.10 38.16 Evening Classes 40.84 37.92 40.68 54.25 Extended School Hours 2.90 4.60 3.22 7.59 School Grade, % 0.01 Fifth and sixth grade of primary 2.91 4.61 3.19 6.99 First of secondary 30.48 30.68 3.73 8.38 Second of secondary 36 35.03 42.04 6.03 Third of secondary 30.61 29.68 50.96 78.60 ERI Questionnaire Scores ** 𝑥𝑥̅= 16.14 SD 6.16)* 𝑥𝑥̅= 10.03 SD 7.06* 𝑥𝑥̅=17.81 SD 6.69* x� = 20.20 SD 6.2* Low (0 to 14) 0.023 30.31 27.87 28.39 16.43 Medium (15 to 38) 37.87 48.30 53.63 67.39 High (39 and over) 31.82 23.83 17.98 16.18 NOTE: Each category shows the percentages for each of the variables, which is 100% when adding up all the categories.. * Corresponds to the mean 𝑥𝑥̅ and standard deviation (SE). **Information extracted from the 2015 ERI questionnaire. *** Considers the statistical significance of the value P by comparing groups using the Chi-Square test. + Data collection: 2015 September, 2016 June, 2017 June, and 2018 June. Regarding the risk evaluation, for the Year 1 school year, the ERI questionnaire score in relation with the baseline evidence (i.e., September 2015) shows that the average of the total risk score was 21 16.14 (SD 6.16) which places it at an intermediate risk level and that 31.82% of the participants are placed in the “high-risk” category. At the end of the school year (i.e., June 2016), the mean of the total risk score was 10.03 (SD 7.06), placing it at a “low-risk” level and 23.83% of the participants for this year are placed in the “high-risk” category. The DHARTE project was designed to follow a cohort of youth through graduation, therefore each year the sample size decreased in size as students graduated. At the end of the Year 2 school year (i.e., June 2017), the mean total risk score was 17.81 (SD 6.69), which places them in an “intermediate risk” level, with an important decrease in the “high risk” category (17.98%). In Year 3 (i.e., June 2018), there was a considerable increase in the total average score of the risk in relation to previous years (Average 20.20, SD 6.2) and 16.18% of this year's participants scored in the “high risk” category, reporting the lowest percentage in this category of all the years of the program's duration. A. Effect on Beneficiaries The first section of the findings23 corresponds to the questioning of the aspects related to the beneficiaries of the Program. The effects of the intervention as well as the proper management that would allow involving or exposing the students to risky situations are described from the testimonials made by the youth and school personnel. In this respect, the contextual factors are identified, such as facilitators or obstacles, that influence the effects of the Program and their differentiation in the students’ sex. 1. Was the intervention successful in reducing both the exposure as well as the perpetration of crime and violence among the beneficiaries? The quantitative evidence shows there is a positive effect of the intervention, attributed to the workshops carried out by DHARTE. With this, a marginal achievement to maintain the reduction in the exposure to violence throughout all of the intervention is observed (because this positive effect is only seen in some schools, that is, the extrapolation to all is not possible). The results in measuring risk show a tendency to reduce que amount of cases diagnosed as “high risk”, which tendency through time shows that among the students of the schools that were involved with DHARTE during the intervention, it decreased. 23 For more details concerning the qualitative methodology implementation, see Annex D. 22 Image No. 3: ERI Questionnaire Score Results, period 2015 to 2018 The risk tendency has been controlled at the trigger level as can be observed; in the specific case of “high risk”, it decreased when compared to the information between 2015 and 2018 among the participants of the nine schools, as shown in Image No. 3. 2. What were the key factors facilitating or preventing the decrease in violence? Considering as a main information source to measure the frequency among teenagers assessed at their risk level, a bivariate analysis24 was carried out with the intention of identifying those variables that, by association, would allow to show the factors that influenced, and consequently could be considered, in the inferential analysis, to demonstrate the effect on beneficiaries. The analysis includes each one of the schools and considers the information gathered between 2015 and 2018, which are in the DHARTE implementation duration scope in Nuevo León. Given the ERI tool response rate, the variables considered in the models presented in Table No. 7 are based on the statistical significance between significant and critical factors. The following variables presented the correlation coefficient with the highest risk (obtained from the Multiple Linear Regression models) in all the years of the Program’s duration (regression models; see Table No. 9): a. I feel comfortable talking about my problems with my family, friends, or partner (communicating problems to the support network). b. My parents support me or motivate me to do what I like (parent/guardian support). c. In my family, there is yelling, insults, or beatings (intra-family violence). 24 The description of the findings by school can be found in Annexs G and H 16.14 10.03 17.81 20.2 31.82 23.83 17.98 16.28 0 5 10 15 20 25 30 35 2015 2016 2017 2018 Average total risk High risk 23 Table No. 7: Estimate of the critical/significant and significant factors during the years of the evaluation's duration (2015 to 2018) in each of the schools. Communicating Problems to the Support Network Parent/guardian Support Parent/guardian supervision: Free time Fist Fights Threatening/Assailin g with objects Intra-Family Violence Substance Abuse (type and frequencies) Problems Associated with Substance Abuse Sexual Initiation Sexual Activities: Methods Self-Inflicted Injuries Death Plan Primaria Prof. Arcadio Lozano * . ☒ ☒ . . ☒ . . . . ☒ . Primaria Niños Héroes ** . ☒ . . ☒ ☒ . . . . ☒ . Primaria Rafael Ramírez*** . ☒ . . ☒ . . . . . . Secundaria No. 2 Fórum Universal de la Culturas 2007* ☒ ☒ ☒ ☒ ☒ ☒ ☒ ☒ ☒ ☒ Secundaria No. 3 Amado Nervo*** ☒ ☒ ☐ ☒ ☒ ☒ . ☒ ☒ ☒ ☒ ☒ Secundaria No. 6 Historiadores de Nuevo León* ☒ ☒ ☒ . ☒ ☒ ☒ ☒ . . ☒ ☒ Secundaria No. 43 Melchor Ocampo*** ☒ ☒ ☒ ☒ ☒ ☒ ☒ . . . ☒ ☒ Secundaria Técnica No. 71 Leandro Valle * ☒ ☒ ☒ ☒ ☒ ☒ ☒ ☐ ☒ . ☒ ☒ Secundaria Técnica No. 88 Prof. María Lavalle Urbina* ☒ ☒ ☒ . ☒ ☒ ☒ ☒ ☒ . ☒ ☐ Note: according to the data availability of the ERI questionnaire, among the variables of interest (critical and significant factors): Four years*, From three to four years** Three years*** It must be noted that Table No. 7 only includes the analysis of the twelve factors considered as significant/critical and significant of the ERI questionnaire, because we consider them of greater relevance in the risk and for being highly correlated with other items of the questionnaire, (Example: Item 3—when I have a problem, I can count on friends to hear me out and it can help—is highly correlated to item 4, one of the significant factors: I feel comfortable talking about my problems with my family, friends, or boyfriend. The selection of the factors considered in the final models, as described in the section “Intervention Effects: Quantitative Evidence (Beneficiaries)” was based on assessing the p value (<0.05; therefore, establishing statistical significance) and that the coefficient was contained in the confidence intervals, but also did not include 1. For example: coefficient=1.04, CI 95%=0.99–1.09 and value of p=<0.001; as can be observed in the coefficient “the value of P” is statistically significant, the coefficient is contained in the interval and it also includes 1. In conclusion, the coefficients and the interpretation of the estimators in the tables must consider the null value; we can infer that it is not a reliable estimator and accordingly it wouldn’t be statistically significant; for more information see Attachment G). Among the variables of interest: Parents/guardian support, parents/guardians supervision: free time, assault/threats with instruments, intra-family violence, self-injury and plans to die; these were considered as important factors to be included in the final models because they were the ones that fulfilled the aforementioned criteria for the majority of the schools (including primary and secondary schools). 24 Participative evaluation workshop with youth Facilitating factors for reducing violence: The youth mentioned, to a greater degree, that those that allow them to face situations of violence are related to family, their neighborhood community, and their schoolmates. Specifically, the family factors are related to the dialogue among family members, confidence in the youth’s capacities, support in decision-making and reaching agreements. The neighborhood factors that facilitate the decrease in violence in the context in which youth live are the mediation among neighbors or “neighborhood justice” to solve problems, the neighborhood councils, as well as reporting to authorities when there is any crime, and notifying adults when they have a problem. In school, youngsters mention that it is helpful to maintain communication with their peers and coexistence with their schoolmates to solve problems among themselves, count on mutual help and respect, as well as the confidence of establishing the dialogue with an offending schoolmate in case of bullying. To a lesser degree, the students see the teachers as a resource that facilitates the decrease of violence; in this respect, they indicate they maintain a relationship of respect, support, and communication. Finally, the teachers mentioned their personal and social skills as input to face such situations, including their self-confidence and self-esteem, respect for their peers, their capacity to dialogue, and motivation to reach their goals. “If you have self-esteem issues, but you have your parents’ support, you can get out of the problem.” “The teachers are supporting our ideas because they know the potential we have, and we can do the work...” “In my community there is a neighborhood justice; every time something happens that involves having the police come, there is a neighborhood justice proposing the solutions to reach an agreement for those who agree with that statement...” “What helps us is to have a good communication among ones schoolmates, talk, an example is when I have the support of my schoolmates when facing a situation” Risk factors that trigger violence: the youth mention other family aspects as impediments to face violent situations, including the lack of communication, absence of a parent, lack of attention toward the youth, and/or addictions problems in the family. It is important to highlight that the students mentioned on several occasions the violent situations they go through such as fights; disagreements and discussions; verbal, physical violence, and the one caused under the influence of alcohol. The community, to a lesser degree, can be an obstacle, especially when there are thefts and drug trafficking, lack of surveillance and participation of neighbors in community activities, and insecurity in general. Likewise, the youngsters identify bullying and beatings among schoolmates and discrimination as elements in common that impede the reduction of the violence surrounding them. The lack of the teachers’ attention in the face of aggressive events against students can result in the lack of confidence and communication as factors that do not contribute to offer a solution to their problems with their peers and doesn’t help them face violent situations. Lastly, the youth mentioned the negative attitude, low self-esteem, and the non-assertive communication as additional obstacles. “Not trusting your teachers... if you don't trust them, you're not going to tell them; but if you trust your teachers, you tell them that someone's going to beat you up, and then they help...! Teenager workshop “Alcoholism is an obstacle related to the family, (is related to) communication with the father because that's the way one speaks with the father...” Teenager workshop “In the school there is verbal bullying, then the schoolmates would say that to solve it one had to speak to adults telling them that one was being picked on...” Teenager workshop “In the community or in the family, family violence, the father beats the children and the mother just because, then to stop beating one needs to talk” Teenager workshop 25 V. Discussion The risk typified with the quantitative evidence shows its presence in all the schools, independently of the school and sex. Risk triggers among youngsters, in interacting with violence, are the aggression–threat with instruments, intra-family violence, and self-injury. The statistical data show a marked tendency among youngsters of a lack of parental or guardian supervision, which was referenced in relation to the last six months (respective of each application moment of the ERI survey), related to a lack of orientation regarding their dreams and wishes. In addition, the lack of supervision is evident, because youngsters don't report their whereabouts in their free time, being accordingly more exposed to situations of violence. The aspects that young people registered positively to face risky situations are the communication with the support network, support from parents/guardians, as well as parent/guardian supervision in their free time. The qualitative evidence shows that dialogue and support in decision-making, jointly with family members, helps in reducing violent behaviors. The conflict solving by the “neighborhood justice” in the neighborhood councils are community factors that facilitate the reduction of violence. On the other hand, the teenagers’ peers in the school benefit the decrease in violence in relation to their communication, conflict solving, mutual help, and support network inside the school. It is important to mention that dialogue was identified as a helpful tool to face cases of bullying. According to the interviewed adults, the obstacles the youngsters face in the family are lack of communication and attention to their children, as well as the parents with addictions—an aspect with which the youngsters agree,—as well as the presence of domestic violence. In conclusion, that quantitative and qualitative data both suggest that the lack of parental supervision influences in the exposure to “high-risk” situations of violence. 1. Do the results related to violence prevention differ according to location or gender? Among the results of the participating schools (from Table No. 7), jurisdictions (municipalities) Escobedo (Secundaria No. 3 Amado Nervo), Guadalupe (Primaria Niños Héroes), Juárez (Secundaria No. 2 Fórum Universal de la Culturas 2007, Secundaria No. 6 Historiadores de Nuevo León, Secundaria Técnica No. 71 Leandro Valle, and Secundaria Técnica No. 88 Prof. María Lavalle Urbina), Monterrey (Primaria Prof. Arcadio Lozano, Primaria Rafael Ramírez, and Secundaria No. 43 Melchor Ocampo), the effects achieved by the intervention were found in 2018, that are shown by the multiple regression models, considering there is significance in the value of p and in the IC95%. • Secundaria No. 3 Amado Nervo, 2018 Model, coefficient=2.88, IC95%=1.02–4.74, and the value of p=0.002 (See Table No. 9). School: example from Secundaria 71 Leandro Valle in the 2018 Model. Coefficient=2.8, IC95% 1.17–4.53 and the value of p=0.001 The students of secondary school Secundaria 71 Leandro Valle have on average 2.83 times more probability of having a high total risk (ERI Scale), in comparison with other schools that were included in the analysis for the same year, with a confidence interval of 1.04 to 4.70 in which the coefficient is contained and doesn’t include 1. The value of p= <0.001 indicates that this association is statistically significant and was adjusted by other variables of interest, discarding confusion and interaction among the model’s variables. See Table No. 9 High-risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. 26 • Secundaria Técnica No. 71 Leandro Valle, 2018 Model, coefficient=2.8, IC95%=1.17–4.53, and the value of p=0.001 (See Table No. 9). As the results show, these two schools have a significant value up to 2018 for violence prevention, while the other schools have a marginal significance. Image No. 4: ERI risk score 2015 to 2018 15.48 17.46 17.73 19.99 16.73 18.55 18.41 20.38 0 5 10 15 20 25 2015 2016 2017 2018 Average total risk (boys) Average total risk (girls) 27 2. Risky Behavior between Sexes: The majority of the interviewed adults reported that both sexes are equally likely to engage in risky behavior. Young women are found to be involved in gangs, and while involvement is more frequently witnessed in young men, respondents felt that this difference has been decreasing due to both sexes facing similar emotional and family issues. Respondents also felt that boys were more “ill tempered” or have a greater exposure to violent situations, while women are more vulnerable to becoming victims of violence or crime. Sex: example of sex in the 2015 Model Coefficient=0.42, CI 95% 0.07–0.76 and the value of p=0.016 The men have on average 0.42 times more probability of having a high total risk (ERI Scale), in comparison with the women for this same year, with a confidence interval of 0.07 to 0.76 in which the coefficient is contained and doesn’t include 1. The value of p=<0.016 indicates that this association is statistically significant and was adjusted by other variables of interest, discarding confusion and interaction among the model’s variables. See Table No. 9 High-risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. 28 A. Intervention Design Modification and Replication This section addresses the progress made by the DHARTE Program in relation to the recommendations of the midterm evaluation. The description of the process is made based on the information sent by the members of AHM and the in-house analysis of the Annual Plan for year three. This was compared to the information obtained from the interviews with key informants, according to the evaluation questions. The use of participative methodologies contributed to document the work carried out during the implementation with a focus on strategic planning among the organizations INJUVE-Nuevo León and Monstritos A.C. Regarding the question Is replicating the model feasible? The INJUVE, and Monstritos organizations considered that the workshop for “Replication Transfer” has allowed them to identify important aspects and needs for the implementation of the DHARTE Model. When asked about the sites and implementation plan, however, the organizations responded that they are still in the planning phase and didn’t mention having chosen intervention sites or having an operational team prepared for the implementation. Regarding the availability of financial resources for implementing the model, the organizations made no comments supporting the operation of the replication in the near future. To which extent is the replication model efficient and effective? Both organizations believe the workshop was of great help. However, they don't have the intervention guide of the Model and given the planning stage in which they are, there is no evidence allowing an answer to this regard, in relation to the terms “efficient and effective.” What progress was made by the project in implementing the midterm evaluation recommendations? What effect did the midterm recommendations have over the effectivity and impact of the project’s interventions? The recommendations of the midterm evaluation of the DHARTE Project are related to reentering the printed surveys of the ERI instrument via the database to decrease error rates; strengthening the project monitoring via the systematic use by supervisors to facilitators; requesting that the schools provide school data about students (grades, dropouts, detentions, quarrels, and suspensions); involving a gender expert to evaluate workshop contents; increasing parents’ commitment in workshops; creating a policy for the participation of faculty in the workshops; carrying out additional sessions and meetings with teachers to define a better sustainability plan; operationalizing a learning system based on feedback from facilitators to report updates in real time; and developing a security plan for the Heartland personnel. Regarding parent participation in the Program, the evaluation team identified in their findings the midterm recommendations, some of the factors that impact such involvement, as well as actions implemented to increase it. For example, during the interviews carried out by the evaluation team, Heartland staff members made references to specific strategies carried out to facilitate their attendance, including “sending communications, phone calls, food package raffles, workshop offers, and links to other organizations to offer more services, adjusting schedules according to the needs of mothers (including on Saturdays), adjusting the workshop to the time/hour of the attendance, calling meetings jointly between school and AHM team, and managing WhatsApp groups.” The aforementioned actions helped in forming groups with parents participating in scheduled sessions. The PRIME25 document includes activities that promote the aforementioned participation of the parents: 25 The PRIME document is a tool for internal use by the Heartland Alliance International subsidiaries for the purpose of systemizing and informing the implementation of their programs. This document contains the DHARTE Program plan for assessing and monitoring, as well as the matrix of indicators. 29 • Activity 1.1.3, regarding the evaluation of the participating family profiles with medium or high risk for informing future activities, has a delayed status. • The activity identified as sub-objective 1.3, regarding the active participation of the families to promote healthy relationships with the youth for the purpose of reducing the risk of getting involved in crime and violence, was evaluated with the percentage of the families that feel that their relationship with their son or daughter improved The goal of 58% was reached as of June 18, 2018. • Activity 1.3.1, regarding the design of workplans for each family, has a delayed status. The recommendation to include other instruments that measure the at-risk population was the recommendation that got more coverage, both in the third-year Annual Plan as well as information gathered by the evaluation team during the interviews. It must be mentioned that according to the quantitative analysis, the student population surveyed by the ERI instrument decreased significantly in the last two years of the Program. The recommendations to hold additional sessions and meetings with the teachers to define a better sustainability plan are included in the PRIME document in activity 1.1.4, which proposes carrying out focal groups for school personnel, with the purpose of assessing the work environment and the main risks the youth face. Said activity is marked as delayed. Likewise, the implementation of the recommendation regarding the use of the schools’ statistical information was not possible because of difficulty in having the schools gather the information and sharing it with the Program. The AHM personnel referenced this difficulty as follows: “We sometimes depend on the schools to obtain information; for example, we would like to gather information of grades and attendance in the schools, but it was very difficult to obtain the information; we have some data but they aren’t complete.” Finally, according to the Heartland staff, the recommendation “regarding the hiring of a specialist in mental health and gender to review the guidelines, it was somewhat delayed because such review was performed up to the third quarter.” B. Intervention Effect 1. Changes the program generated in the youth The quantitative evidence shows the intervention’s effect. Testimonials helped respond to evaluation questions were used for this purpose. At a community level, the DHARTE effect cannot be assumed to permeate to peer teenagers residing in nearby populations. During interviews with school personnel, there was no testimonials regarding activities focused on assisting the neighboring communities/populations residing near the schools (primary and secondary schools). The intervention’s effect, due to its focus, for example, in the 5th and 6th grades of primary school does not allow concluding that the positive effects have permeated to the other parts of the school community in primary schools. The secondary schools will not continue with active workshops. Therefore, the effect will not continue benefiting the school community at this time. 2. What behavior changes occurred (intermediate effects) in the beneficiaries? Self-Awareness and Self-Control: These categories are grouped in the same chapter because the interviewed individuals mentioned these results in a joint manner. 30 Repeatedly, the interviewed adults participating in the program mention that the students changed their behavior towards identifying and controlling emotions, especially regarding tolerance towards their schoolmates and family. They also refer to said process as emotional maturity. In addition, some of the interviewed adult individuals mention that the changes are reflected in the decrease of school bullying, drug use, and the awareness raised around this topic, as a different option to get along together transcending violence. “Drug use decreased, they observed this by checking backpacks; it is no longer an everyday activity.” Interview with School Official Among the testimonials by youngsters participating in the evaluation workshops, there is little mention of self￾knowledge and controlling their emotions, as follows: “(It helped us to) know how to control ourselves, when I felt bad, calming down helped me to decrease the violence.” Workshop with teenagers Self-Confidence: On the other hand, the students’ confidence increased—normally mentioned by the adults who participated in the DHARTE Program—as self-esteem and security in their expression, as well as questioning about their context. The youth that participated in the workshops mentioned (third place) that the matter of having confidence in themselves and in others was a result of their participation in the Program. The students mentioned the confidence related to their potential and strength, as well as an increase in their own qualities and authenticity, and having the confidence to work towards their dreams, becoming more positive individuals. “It brought out the potential in me, and taught me how to be stronger than I already am” Workshop with teenagers “Be myself, not let myself be influenced by bad comments” Workshop with teenagers Self-Motivation: According to the interviewed adults, the contents of the workshop regarding the life project (among others facilitated by the Program) generated motivation among the youth to prepare themselves academically seeking to progress. Likewise, the students identified different life perspectives than the current situation of violence that they encounter around them. For example, it was mentioned that one of the changes is found in evading “bad influences” and drug use, as well as in improvemed life skills to identify environmental risks, projecting themselves toward the future, and making better decisions. “When the youth graduated, they did it with a much more specific life project because they then had the dream of going to college to continue their studies.” interview with School Official “The life project helped them question the worldview and re-focus the participants. Since there are different stereotypes, there are alternatives to not copy them. For example, the girls wanted to have children.” interview with Heartland Staff “Spend time with healthy teenagers, there is no need to take risks, one can live without the need of getting high on drugs, to be somebody and have dreams.” interview with School Official Critical/significant factors: example: “I've considered and tried to carry out a plan to harm myself and die”, the comparison among models shows an effect of the treatment when comparing the results after the modification of the intervention resulting from the Midterm Evaluation 2017 Model, coefficient=9.2, IC95%=7.62–10.77, and the value of p=<0.001 2018 Model, coefficient=5.6, IC95%=3.83–7.38, and the value of p=<0.001 See Table No. 9 High-risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. 31 The school personnel mentioned that the Program generated a reduction in student dropout and school absenteeism, which could be resulting from the increase in the youth’s self-motivation. Assertive Communication: The school personnel and DHARTE Program staff members commented on the adoption of assertive communication among the youth as a result of the intervention. In particular, the informants mention the capacity of the youthto listen, express their emotions and themselves, establish limits, as well as the assertiveness to question the school and family context in a constructive manner. The young people who participated in the workshops mentioned (fourth place) that communication was a result of their participation in the workshops. In particular, they mentioned an increase in communication with teachers and schoolmates, being more assertive, and improving the ways they express themselves. In fifth place, the youth identified that their creative capacity increased as a result of the Program’s intervention in their schools. “I learned to express myself much more and in different ways.” Workshop with teenagers “Maintaining a better communication with my teachers and schoolmates.” Workshop with teenagers “It taught me to be more assertive.” Workshop with teenagers Empathy: The school personnel and DHARTE Program staff members mentioned that the young people make decisions pacifically when facing conflicts with their schoolmates. That is, the empathy generated with their schoolmates helped them seek solutions based on respect and tolerance among them, which promotes the reduction of aggressions in the intervened schools and in the schools’ nearby areas (street fights). “They give them examples, they explain how to solve problems. They see and compare in case a situation such as fights between schoolmates happens to them, it's easier to see the way out for decision-making. Before the intervention, they would watch and film them; they now notify their teachers, they get very worried about their schoolmates, their consequences, and feelings,” interview with School Official “The hostile environment has changed in 10%26 of the 600 students” interview with School Official “The youth have always respected the school; the school was able to work with the parents and the police also. No more fights were seen in an area of about 2 or 3 blocks” interview with School official Among the testimonials (fifth place), the youngsters that participated in the workshops stated that the program helped to increase the respect towards their schoolmates. Social Skills: To a lesser degree, the interviewed adults mentioned that the young people learn to work in teamsThat is, the youth acquired greater confidence and tolerance in socializing. In this regard, the aforementioned increase in personal development (self-awareness, self-control, motivation, and empathy) promoted a more effective integration of the students in work teams at school. 26 This testimonial points to the fact that the reduction in violence is positive, and although the estimated proportion is not significant, it is included because of the emphasis the informant made on it. 32 In the workshops carried out with the youth, it is more frequently mentioned that the effects generated in them are related to teamwork. On the other hand (second place), the youth mentioned that as a result of their participation in the program, they learned to support and coexist with their schoolmates, promoting friendship and socialization. “My way of getting along with my friends and family members got better, and I made new friends”; Workshop with teenagers What behavior changes occurred (intermediate effects) in the beneficiaries? Below, behavior change generated by the intervention are described (cognitive learnings): Raising Awareness: The DHARTE Program generated additional effects other than the development of personal and social skills. At a cognitive level, the school personnel and DHARTE Program staff consider that the students show changes in relation to a greater awareness of the situations surrounding them, particularly regarding school violence and addictions. “Having them learn the risks regarding violence, addictions, and informing them, sensitized them, and showed them an alternative.” interview with Heartland Staff Action strategies: the program contents helps the students implement security plans and establish support networks, in which the students identified their parents and teachers as support elements that can provide help in facing the experiences the teenagers go through. In summary, the most important achievements of the DHARTE Program in reducing risky behaviors were self￾awareness and self-control, assertive communication, having confidence, and empathy. At a community, organizational, and participant level, what are the project’s key achievements that will remain long after the life and context of the project? For the purpose of answering this question, the evidence from the testimonials gathered in the interviews with key informants and the teenagers participating in the workshops show some of the effects that will remain after DHARTE ends. Sustainability: This concept is understood as the permanence effect of the Program, especially regarding the students’ personality and behavior changes that continue once the DHARTE Program has ended in the schools. In some cases, interviewed adults mentioned that if the DHARTE Program’s intervention were not to continue, there was a possibility of losing the learning related to the acquired self-esteem, confidence, personal expression through art, and eradicating bullying, because of the context circumstances and family situations. That is even though they consider that certain life skills have permeated in the students’ personalities. Significant/Critical Factors: example of cutting or self-inflicting injuries in 2016 Model. Coefficient=4.41, IC95% 3.73–5.09 and the value of p=<0.001 Youngsters who mention that at some point in their life they've considered or tried to harm themselves (cutting or self-injury) have on average 4.41 times more probability of having a high total risk (ERI Scale), in comparison with those youth who haven't considered it or have done it this same year, with a confidence interval of 3.73 to 5.09 in which the coefficient is contained and doesn’t include 1. The value of p= <0.001 indicates that this association is statistically significant and was adjusted by other variables of interest, discarding confusion and interaction among the model’s variables. See Table No. 9 High-risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. 33 Overall, the adults stated that there are changes in the youth that will remain in time. In this way, according to the interviewed individuals, the youth’s motivation to prosper, progress, and have confidence in themselves, the increase in tolerance and self-esteem, the capacity to act assertively, as well as manage emotions, and work in teams will prevail in them. Below, the changes that the interviewed adults mentioned as sustainable over time are explained in detail: • Self-Awareness and Self-Control: The most common testimonials made by the adults are related to self￾awareness and self-control. In particular, they mention that identifying and controlling their emotions, as well as interest in the arts, are factors that may continue to reduce violence among the youth. • Communication: Hand-in-hand with the students’ self-control of their emotions, the adults who participated in the program stated that the youth learned to express their emotions and to solve conflicts via dialogs—the latter is one of the effects mentioned with the most emphasis by the interviewed adults. Along with communication, confidence, self-esteem, empathy, and positive decision-making (far from risky situations) are important aspects that were mentioned as durable effects. In particular, it is mentioned that the students will have more communication and socialization with their parents about their emotions. • Confidence and Empathy: Likewise, there is indication that the students increased their self￾esteem when facing their problems and motivation to increase their quality of life. Thanks to exploring their artistic skills, the youth took on leading role, which allowed them to express their ideas and talents by creating artistic products—such as short films and mural painting. In this sense, the students acquired the confidence to work in teams with their schoolmates and family members. 3. Regarding the Effect of DHARTE throughout Time Significant factors: the example of in my family there is yelling, insulting, or hitting in the 2017 Model. Coefficient=7.11, IC95% 6.19–8.03 and the value of p=<0.001 The youth who mention that in the past 6 months yelling, insulting, or beatings are common in their family have on average 7.11 times more probability of having a high total risk (ERI Scale), in comparison with those youth who mention that that same situation rarely or never occurs, with a confidence interval of 6.19 to 8.03 in which the coefficient is contained and doesn’t include 1. The value of p= <0.001 indicates that this association is statistically significant and was adjusted by other variables of interest, discarding confusion and interaction among the model’s variables. See Table No. 9 High-risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. 34 The Program’s internal-use document “Intervention Guides, DHARTE “Arts Therapy as a Tool for the Social Prevention of Violence””27 mentions the life skills that the program promoted in the workshops carried out with student participants; these are: a) Communication skills/healthy relationships; b) Decision-making and problem solving; c) Building empathy; d) Cooperation, teamwork; and e) Emotional control skills. In the following paragraphs, there is a description of the relation between the life skills promoted by the program and the findings in the gathered data: Table No. 8: Life skills and acquisition of emotional intelligence skills among beneficiaries. Life skills promoted by DHARTE Life skills as an effect Emotional control skills Self-Awareness Self-esteem Self-control Building Empathy Empathy Healthy relationships, cooperation, teamwork Teamwork Communication Skills Assertive Communication Decision-Making and Problem Solving Action strategies Coupled with the aforementioned acquisition of social and personal skills, at a cognitive level, the younth incorporated action strategies and awareness in their decision-making to prevent their involvement in risky situations. As was described in the bivariate analysis section, variables identified in the ERI questionnaire were validated. To comply with the statistical significance, the confidence interval values28 were considered in the inferential statistics. The predictive analysis required testing the associated assumptions. The inferential analysis29 required considering time as a cross-section for data modelling, four Multiple Linear Regression models were created; the proposal of working with the dependent variable considered as continuous was due to the characteristics of the data and because of the 27 Intervention Guides, DHARTE “Arts Therapy as a Tool for Social Prevention of Violence" prepared by Alianza Heartland México with support from USAID. This document is being edited; therefore, the description of each of the skills is not included in this document. 28 The coefficients, IC95% and the value of p, were estimated for each of the models corresponding to the years of the intervention's duration: Model 2015 (n= 2992), Model 2016 (n=2821), Model 2017 (n=1680), and Model 2018 (n= 828). 29 See Annex H During the descriptive analysis and the correlation valuation, the variables had significant relevance. Having found the model that would be the best fit for the characteristics of the variables, that is, considering the continuous and categorical variables, the Regression Models Probit and Logit give the decrease in the statistical power of the data for the 2017 and 2018 samples, the solution was to show the change per response unit or category (given that the risk is managed on a continuous scale). In conclusion, Multiple Linear Regression was used. All of the above served so that the assumptions of linearity (scatter), normality of residuals (Kolmogorov-Smirnov), homogeneity of variance, and independence of residuals would be confirmed when evaluating the models. 35 relevance to carry out the interpretations. It is important to mention that other alternatives were used to perform the modelling (Probit, using an ordinal regression with a categorical dependent variable), but did not adjust correctly with the data. 𝒀𝒀 = 𝜷𝜷𝟎𝟎 + 𝜷𝜷𝟏𝟏𝑿𝑿𝟏𝟏+. . . +𝜷𝜷𝟔𝟔𝑿𝑿𝟔𝟔 + 𝑬𝑬 𝑌𝑌 = 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝑇𝑇 𝑇𝑇 𝑅𝑅 𝑅𝑅) 𝛽𝛽0 = 𝑜𝑜 𝑖𝑖 (𝑌𝑌, 𝑋𝑋 𝑖𝑖 ) 𝛽𝛽1…𝑛𝑛 = , 𝑟𝑟 𝑟𝑟𝑟𝑟 𝑟𝑟𝑟𝑟 𝑐𝑐𝑐𝑐 (𝑢𝑢 ) 𝑋𝑋1 = my parents never support me or motivate me to do what I like (significant factors, ERI) 𝑋𝑋2 = 𝑚𝑚𝑚𝑚 𝑛𝑛 𝑘𝑘 ℎ 𝐼𝐼 𝑚𝑚𝑚𝑚 (significant factors, ERI) 𝑋𝑋3 = 𝐼𝐼’ 𝑢𝑢 𝑜𝑜 ℎ 𝑟𝑟 (significant factors, ERI) 𝑋𝑋4 = in my family,there is yelling, insults, or beatings (significant factors, ERI). 𝑋𝑋5 = 𝐼𝐼 ℎ ℎ , , 𝑚𝑚𝑚𝑚 (significant/critical factors, ERI) 𝑋𝑋6 = 𝐼𝐼’ ℎ ℎ ℎ 𝑚𝑚𝑚𝑚 (significant/critical factors, ERI) 𝐸𝐸 = 𝑅𝑅 𝐸𝐸 All the proposed models were adjusted considering the variables: School (nominal categorical), sex (dichotomous), significant/critical factors considering the ordinal categorical variables (I have considered harming, cutting, or injuring myself, and I’ve thought about harming myself and dying) and significant factors also all ordinal categorical variables (my parents never support me or encourage me to do what I like; my parents or guardians never know where I spend my free time; I've used an object to assault or threaten someone; and yelling, insulting, or beatings are common in my family). Considering the characteristics of the variables for the data model, dummy variables were generated and included in the models (transformation to categorical variable), to be able to leave a reference category, and in that way infer the change from the coefficient in the categories of interest. 36 Table No. 9: High risk estimation in Multiple Linear Regression Models for the 4 years of the Program's intervention. NOTE: The coefficients have been estimated via Multiple Linear Regression Models in each of the study’s duration years (2015, 2016, 2017, and 2018). Each column corresponds to a particular model, and the associated statistical significance value (value of p) and the confidence intervals to 95% are included in addition to correlation coefficient. All the proposed models were adjusted by School a, sex b, significant/critical factors (I have considered harming, cutting, or injuring myself, and I’ve thought about harming myself and dying) and significant factors (my parents never support me or encourage me to do what I like; my parents or guardians never know where I spend my free time; I've used an object to assault or threaten someone; and yelling, insulting, or beatings are common in my family). The aforementioned critical/significant factorsc and the significant factorsd were selected to be considered in the models because of their statistical significance when performing the bivariate analysis (p-value <0.05, IC95% not including the 1). For more information see table No. 7. * Corresponds to the Pearson linear correlation coefficient; it is an index that measures the grade of covariation between different linearly related variables (multivariate analysis) once the multiple linear regression model has been run: it's interpretation is subject to the following parameters BETA-1 COEFFICIENT: It's the change for each unit that may increase Xn on average and will increase BETA units, also calculating the random error. ** Corresponds to the 95% confidence interval, that refers the margin error type II that is estimated will be allowed in the analysis. *** Considers the statistical significance value of P for those values less than 0.05. 37 VI. Conclusions and Recommendations A. Conclusions 1. Regarding the Effect Among Beneficiaries: The quantitative data show that in some of the schools in which the program was implemented, the risk decreased. However, there is no confirmation that this happened in a general manner in all schools, even though there is evidence that the proportion of high-risk students decreased. Qualitative data indicates that there was no difference by school in the way in which the intervention was welcomed by the students. The quantitative evidence shows that the intervention did not have the same effect among all of the teenagers participating in the program activities in the schools. There are differences in the effects among schools. The qualitative evidence shows that students acquired personal and social skills such as self-awareness, self-esteem, self-control, empathy, and the capacity to work in teams. It is worth highlighting that assertive communication, empathy, and having confidence in themselves are mentioned as sustainable effects in time. In summary, the program promoted and achieved the acquisition of life skills in the participating youth. The intervention provided by DHARTE sought to modify teenager’s individual conditions when interacting with violence, in order to avoid it. In conclusion, we can identify the key factors that contributed to decreasing the level of risk for youth involvement in violence and crime. These may be observed in the models in Table No. 9, the coefficients are identified by their statistical significance. The effect among schools is different. This may be explained considering the particularities of activity facilitation inside schools, as well as its acceptance and support on behalf of the administration/faculty/staff, response of the school community (students and parents), and participation in activities. The intervention had a homogeneous effect in regards to the sex of participants. Marginally, the risk results and their relation regarding the sex of the participants show the difference marked in 2015. 2. Regarding Replication Organizations such as INJUVE and Monstritos were receptive and enthusiastic about the adoption of the DHARTE Model when mentioning the training and consulting/advice, even though they didn't receive an intervention manual or guide. Moreover, they only relied on knowledge to apply the ERI instrument. The plans for the implementation and the capacity of relying on financial capital to implement the model were not detected in an activity startup phase, that is, it seems there are no resources assigned to this. However, the organizations rely on personnel that would be involved with the implementation. Lastly, an operational plan that includes the replication of the model for its implementation is still pending. Therefore, it is not possible to draw conclusions identifying direct results in the beneficiaries. It is likely that the strategy to promote the sustainability of the DHARTE Model among the replicators is not evident because of the delays in its programming. 38 3. In Relation to the Intervention Design Modification The evidence shows that the contents included in the workshops, as a medium to acquire life skills, will foster the permanence of self-awareness, self-control, communication, confidence, and empathy skills among the beneficiaries. The evaluation team considers that one of the strengths of the program design is the contents and activities that promote these learnings. The activities that were launched with greater emphasis in programming the Project activities during the third year were those related to the gathering of information of the ERI instrument about students and the inclusion of the parents in workshop activities. Including the parents’ participation did not have the expected impact, although it is an important factor for the decrease of the youth's involvement in violence. The recommendation regarding the inclusion of a gender specialist did not progress during the Program's 3rd year. 4. Intervention Effect The most important achievement among participants is the decrease in RISK associated with the “high-risk” triggers regarding self-destructive behaviors such as “I've considered and tried to carry out a plan to harm myself and die,” as well as the protection factor associated with family life “my parents never support me or encourage me to do what I like.” B. Recommendations • Given the diagnostic experience achieved, refine the ERI to increase sensitivity based on the ages of the young people selected to participate in the intervention. It may also be useful to incorporate collect data on variables such as family composition or proxies for income since these variables are important for designing the intervention with parents. • Future iterations of this program should follow-up with students after graduation in order to have the same number of data collection points for all students. This additional data may allow for better measurement of the intervention’s effect and thus, be able to ensure the causality. • Future iterations of this program should try to collect data by school, including dropout, attendance, completion rates, from the Control Escolar (School Control) kept by the Secretaría de Educación (Ministry of Education) of the state of Nuevo León, so that the program will not be dependent on the schools to provide said information. 39