[Month Year] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Jaime López-Aranda, Antia Mendoza, Lilian Chapa and Leonel Fernández. Final Project Evaluation, Scout Experience: Ready for Life 2 FINAL PROJECT EVALUATION, SCOUT EXPERIENCE: READY FOR LIFE AN ASSESSMENT OF THE PROVISION OF NON-FORMAL EDUCATION SERVICES TO AT-RISK YOUTH IN TIJUANA, MEXICO OCTOBER, 2015 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 CONTENTS Executive Summary ........................................................................................................................................................... 5 Evaluation Purpose & Evaluation Questions................................................................................................................ 7 Evaluation Purpose ........................................................................................................................................................ 7 Evaluation Questions .................................................................................................................................................... 7 Project Background ........................................................................................................................................................... 8 Evaluation Methods & Limitations ............................................................................................................................... 11 Methods ......................................................................................................................................................................... 11 Limitations..................................................................................................................................................................... 12 Findings, Conclusions & Recommendations .............................................................................................................. 14 Findings .......................................................................................................................................................................... 14 1. Project design evaluation ............................................................................................................................. 14 2. Quantitative analysis of observed outcomes .......................................................................................... 17 3. Qualitative analysis of observed outcomes ............................................................................................. 18 4. Structured observation findings ................................................................................................................. 21 5. Contribution to USAID/Mexico Country Development Cooperation Strategy 2014-2018 ...... 22 6. Contribution to ongoing Government of Mexico crime prevention efforts .................................. 23 7. Positive/Negative externalities ................................................................................................................... 24 8. Project sustainability/replicability ............................................................................................................... 24 Conclusions .................................................................................................................................................................. 26 Recommendations ...................................................................................................................................................... 26 Annexes ............................................................................................................................................................................. 29 Annex I: Terms of Reference ................................................................................................................................... 30 Annex II: Base questionaire focus-group interviews ........................................................................................... 35 Annex III: References ................................................................................................................................................. 37 4 ACRONYMS ASMAC Scouts of Mexico (Asociación de Scouts de México A.C.) CDCS USAID/Mexico Country Development Cooperation Strategy 2014-2018 FPE Final Project Evaluation GOM Government of Mexico M&E Monitoring and Evaluation PRUEBA ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional de Logro Académico en Centros Escolares) QPR Quarter Project Report RBM Results Based Management SERL Scout Experience: Ready for Life USAID U.S. Agency for International Development 5 EXECUTIVE SUMMARY PROJECT BACKGROUND, EVALUATION PURPOSE AND METHODOLOGY The Scout Experience: Ready for Life (SERL) project was a joint effort by USAID and Scouts of Mexico aimed at developing, implementing and testing non-formal education models for at-risk youth with the objective of enhancing their ability to play a productive role in their communities. It was based on the Scout Movement’s value system and non-formal educational techniques and was implemented in three at-risk communities (polígonos1 ) in Tijuana, Baja California Mexico from August 2012 to August 2015. This is the project’s final performance evaluation and as such was tasked with assessing its results and providing recommendations for future replication efforts. The evaluation used a mixed-methods approach to provide a more comprehensive understanding of the project’s results and its replicability, including a desk review of the project design and quantitative and qualitative analyses of the project’s outcomes. For the quantitative aspects of the analysis, the evaluation team opted to rely exclusively on the data collected by the project staff through its monitoring and evaluation (M&E) processes, given time and beneficiary availability considerations. For the qualitative aspects, the evaluation team relied on semi￾structured and focus group interviews as well as structured observation methods. The quantitative and qualitative components of the evaluation were conducted in parallel to maximize the evaluation team’s ability to synthetize its findings and provide recommendations within the allotted time. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The main findings and conclusions of the evaluation can be summarized as follows:  The SERL project was successful in reaching at-risk youth in the three polígonos and in establishing a continuous presence to provide them with services. It was also successful in transferring the leadership of the three scout groups it created to the community after the project’s conclusion— which will continue to provide some of the services that were provided by the project to the target population. Finally, it developed a sizable knowledge base that may help further the institutional development of Scouts of Mexico and other organizations.  As per the qualitative analysis conducted, there are observable improvements in the beneficiaries’ personal development and interpersonal skills, as well as in their leadership and academic skills. It was not possible, however, to conduct a meaningful quantitative analysis to measure these improvements or to link them to specific components of the program, because it was determined that the response rate to the different outcome evaluations conducted by the project’s staff was too low.  The project had significant positive externalities, both in regards to the beneficiaries’ family dynamics and to the volunteers’ leadership skills development—the volunteers were not at-risk youth but for 1A polígono comprises one or more “colonias” or officially demarcated neighborhoods that share similar characteristics and are geographically proximate. There may be several polígonos in a single municipality, as in the case of Tijuana. 6 the most part hailed from other scout groups in Tijuana and should therefore be considered local stakeholders. It was also found that the project promoted gender equality and female empowerment and strengthened at least one community-based organization.  Several components of the SERL project experience may be useful for the ongoing Government of Mexico (GOM) crime and violence prevention efforts.  The project is replicable in other at-risk communities at a much lower cost than the original SERL. The project’s inherent flexibility, as it builds on the strengths of the Scout method, makes it a relatively safer bet than other untested approaches. The evaluation team’s recommendations can be summarized as follows: 1. Set up a follow-up mechanism. It is not advisable to assume that the scout groups created by the project should operate like any other regular scout group, as there are significant risks and challenges associated with working with at-risk youth. It is recommended that both the Baja California Scout Province and the Tijuana scout groups designated as sponsors conduct quarterly follow-up reviews of the new groups’ membership levels, leadership-training processes, funding and threats. The objective of this follow-up is to ensure that the new groups have a formal support network and an “early-warning” system in place, should they require assistance. 2. Simplify the project’s objectives and indicators. Future implementations should take into account the difficulties inherent in establishing quantitative indicators for a beneficiary population that may vary significantly within the project’s life span, as well as the unavailability of standardized testing tools to measure scholarly performance and psychological development. It is recommended that new projects adopt basic performance indicators such as the ones introduced after SERL’s midterm review and evaluate whether additional ones are adequate. 3. Strengthen the M&E process and expand its scope. Efforts should be made to ensure that every new project gathers information about its beneficiaries from the start and is able to track them through time. If it is possible to establish a baseline measurement, further testing or assessment should be restricted to those individuals that participated regularly in the project’s activities. A mixed-methods approach should be considered to provide a more accurate representation of the project’s results, instead of relying on a purely quantitative one. 4. Foster more cooperation with other non-governmental organizations and local authorities. It is advisable to identify potential synergies with existing government and non-governmental programs and to promote more formal cooperation mechanisms with local authorities and non-governmental organizations working in the same area. However, it should be noted that this is not a straightforward process and should be previously reviewed with Scouts of Mexico and the province￾level authorities. 7 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE This is the Final Project Evaluation (FPE) for the Scout Experience: Ready for Life (SERL) project, implemented in Tijuana, Baja California, Mexico, from August 2012 to August 2015. It is part of the final reporting process and can be categorized as a performance evaluation—as opposed to an impact one, as discussed in the methodology section. As a performance evaluation, the FPE’s main purpose is to assess the project’s results and to provide recommendations for both USAID and the implementer, Asociación de Scouts de México A.C. (Scouts of Mexico, ASMAC), regarding future project design and implementation efforts. EVALUATION QUESTIONS As per the Terms of Reference (ToR) provided by ASMAC (see Annex I), the evaluation has one primary question and three secondary questions. The evaluation also considers an additional question presented by USAID during the exploratory phase of the evaluation regarding the replicability of the project and its implications for other crime and violence prevention efforts, in the context of USAID/Mexico’s Country Development Cooperation Strategy (CDCS) 2014-2018. Primary Evaluation Question (ToR) 1. Does the project design have the required elements to achieve the expected outcome level results in character, education and leadership in the intervention group? Secondary Evaluation Questions (ToR) 2. Does the project design have the required elements to measure the indicators related to outcomes and outputs on character, education and leadership in the intervention group? 3. What is the magnitude of the achieved results for each outcome and output level indicator, as well as the contribution towards the achievement of USAID Common outcomes? 4. Does project implementation foster any other development results or externalities than those reflected by official indicators, either for participants or non-participants in the intervention zones? Additional questions (CDCS) 5. Can the model be replicated in other communities? How can it be strengthened? 8 PROJECT BACKGROUND The Scout Experience: Ready for Life (SERL) project was a joint effort of USAID and Scouts of Mexico aimed at strengthening the capacity of at-risk youth (7 to 29 years old) to play a productive role in their communities. The project was designed and implemented during a three-year period (August 2012- August 2015) in the city of Tijuana, Mexico. The project fell under the Merida Initiative bilateral cooperation agreement and more specifically under the Initiative’s fourth objective or Pillar IV: build strong and resilient communities. Broadly speaking, projects implemented under Pillar IV are tasked with promoting the target communities’ ability to withstand the effects of crime and violence. And in the case of projects aimed at at-risk youth such as Scout Experience, the specific objectives are to prevent said youth from joining criminal organizations as well as increasing constructive engagement with their communities2. As stated in USAID/Mexico’s Country Development Cooperation Strategy (CDCS) 2014- 2018, citing USAID’s Policy on Youth in Development (USAID, 2012): […] violence, high school dropout rates and the challenges of the school‐to‐work transition can create conditions for vulnerable youth to be recruited into criminal networks, armed groups, youth gangs, and trafficking. As a result, activities are geared towards keeping youth in school, supporting self or salaried employment, and improving social integration and civic leadership. (USAID/Mexico, 2014, p. 23) The current CDCS also emphasizes the participation of local actors and the replicability of the models implemented, as well as the incorporation of a gender perspective. Successful projects are thus expected to be adopted and sustained by the target communities and other relevant local stakeholders after USAID financing ends. They are also expected to complement other existing projects, to contribute to the current Government of Mexico’s (GOM) crime and violence prevention efforts and, more importantly, to be replicated in other communities that were not targeted by the original effort3. From its inception, the SERL project had a dual-nature, serving both as an “educational innovation lab”, aimed at developing and testing non-formal education models for at-risk youth and fostering institutional development, and as a focused intervention in three polígonos4 of Tijuana: Camino Verde, Granjas Familiares and Mariano Matamoros. Both components of the project worked in parallel and provided feedback to each other, with the intervention activities serving as a development and testing ground for the laboratory’s hypothesis and methodologies. Some of the activities of both components are expected to continue semi-autonomously after the formal end of the project’s implementation, as is discussed below. The educational lab component of the project focused on developing methods and tools that 2 See Development Objective 1, Intermediate Result 1.3 of USAID/Mexico’s Country Development Cooperation Strategy FY 2014 – FY 2018 (USAID/Mexico, 2014) for a more comprehensive description of USAID’s development objectives, expected results and strategies regarding at-risk youth. 3 Even though the SERL project predates the current CDCS, USAID and the project management team requested that the evaluation incorporate these criteria. It should also be noted that the project management team made adjustments to better align it to the CDCS that will be discussed in further detail later. 4A polígono comprises one or more “colonias” or officially demarcated neighborhoods that share similar characteristics and are geographically proximate. There may be several polígonos in a single municipality, as in the case of Tijuana. 9 complement the existing scout youth program, foster the institutional development of Scouts of Mexico and strengthen the capabilities of the project’s volunteers to continue developing new innovative solutions for at-risk communities. These methods and tools created during the project in the areas of educational program, adults in scouting management and the Scout Experience Model were integrated in an online Collaborative Learning Platform also known as a replication toolkit. It is expected that this toolkit will evolve into a full-fledged knowledge-management platform that will enable replication, fundraising and collaboration with other scout organizations, the Mexican government and other interested third-party social entrepreneurs. Scouts of Mexico would be directly responsible for the operation and financing of the platform. The intervention component of the project focused primarily on the provision of non-formal educational services derived from the international Scout Movement principles and methods as well as the educational project of Scouts of Mexico, which were adapted to the specific circumstances of at-risk youth in the three polígonos5. These services were provided through three Youth Centers and, after the project’s second year, through the establishment of three Scout Groups affiliated with Scouts of Mexico—one youth center and one scout group for each polígono6. The specific services provided varied from center to center and across time and were the responsibility of project staff and a rotating roster of local, national and international volunteers—most of them members of the Scout Movement and, in the final stages of the project, some of them hailing from the target communities. It is worth noting that while funding for the youth centers ended with the project, the three scout groups are expected to continue operating semi-autonomously under the umbrella of Scouts of Mexico and to procure their own funding as all other scout groups in the country do7. Furthermore, it is expected that the scout groups will continue to provide some of the services that were provided by the project under the leadership of scouts recruited from the target communities, albeit at a smaller scale. The provision of non-formal educational services and capacity-building efforts are linked through a strategic framework that identifies three development aspects encompassing different dimensions of the beneficiaries’ personal development and four corresponding educational emphases (Figure 1). The key development aspects can be summarized as follows: a) Character. Development of personal and interpersonal skills (“learning to be and to be together”) and the adoption of the Scout value-system, commonly referred to as the Scout Promise and Law, as a basis for psychological resilience. b) Education. Development of language, mathematics and logic capabilities as well as specific soft employability skills through the scout merit badge system and a “learning-by-doing approach”. c) Leadership. Education for global citizenship and social leadership; development of competencies related to social project design and management—also applicable to volunteers. 5 More information about the Scout Movement’s principles and methods can be found at the websites of the World Organization of the Scout Movement (http://www.scout.org/) and the Scouts Association of Mexico (http://www.scouts.org.mx/). 6 A Scout Group is the smallest unit of the Scout Movement. It comprises different sections grouped by age. It is expected to operate semi-autonomously and to abide by the Scout Movement and its national association tenets and policies. 7 It should also be noted that the Granjas Familiares center remains operational under community leadership but the other two were closed upon the project’s end. 10 The four associated educational emphases to each development aspect can be summarized as follows: a) Compass (related to Character). Provides tools for the identification of risk and protective factors and psychosocial support needs of individual beneficiaries. Protocols for referral of specific cases to third-party support services for those individuals requiring specialized support or legal interventions. b) Lantern (related to Education). Focuses on reducing the beneficiaries’ dropout rate, enabling effective school transitions and literacy. c) Insignia (related to Education). Focuses on developing soft employability skills using the scout merit badge system and the multiple intelligence approach. d) Young Citizens (related to Leadership). Provides opportunities for social leadership skill￾development for both the beneficiaries and the volunteers themselves. It is based on the Scouts of the World program. Figure 1: Key development aspects and educational emphases Educational Emphases (Volunteers' skills development) Key beneficiary development aspects Purpose At-risk youth capacity strengthening Character Compass Education Lantern Insignia Leadership Young Citizens 11 EVALUATION METHODS & LIMITATIONS METHODS The evaluation used a mixed-methods approach to provide a more comprehensive understanding of the project’s results and its replicability, including a desk review of the project design and quantitative and qualitative analyses of the project’s outcomes. For the quantitative aspects of the analysis, the evaluation team opted to rely exclusively on the data collected by the project staff through its monitoring and evaluation (M&E) processes, given time and beneficiary availability considerations. For the qualitative aspects, the evaluation team relied on semi-structured and focus group interviews as well as structured observation methods. The quantitative and qualitative components of the evaluation were conducted in parallel to maximize the evaluation team’s ability to synthetize its findings and provide recommendations within the allotted time. (See Table 1 for a breakdown of the methods and how they relate to the evaluation questions; more detailed information about data collection instruments and protocols is included in Annex 2.) Table 1: Evaluation questions and methodology (N/A = Not applicable) Evaluation questions Data collection / Analysis method Data sources Sample size / Number of Participants 1. Does the project design have the required elements to achieve the expected outcome level results in character, education and leadership in the intervention group? Design Evaluation (Desk review) Project documentation N/A Interviews Project Staff Local authorities Local non-governmental actors 12 interviews Focus group interview Project volunteers 7 volunteers 2. Does the project design have the required elements to measure the indicators related to outcomes and outputs on character, education and leadership in the intervention group? Design Evaluation (Desk review) Project documentation N/A 3. What is the magnitude of the achieved results for each outcome and output level indicator, as well as the contribution towards the achievement of USAID Common outcomes? Statistical analysis of the project M&E data Project M&E data N/A Focus group interview Beneficiaries Parents Volunteers 26 beneficiaries (divided by age in 3 focus groups, random sample selection) 7 parents 7 volunteers 12 Evaluation questions Data collection / Analysis method Data sources Sample size / Number of Participants 4. Does project implementation foster any other development results or externalities than those reflected by official indicators, either for participants or non￾participants in the intervention zones? Structured observation Project geo-referenced database Site visits N/A Focus group interview Parents 7 parents (all female) Interviews Local authorities Local non-governmental actors 8 interviews 5. Can the model be replicated in other communities? How can it be strengthened? Semi-structured interviews Focus group interviews Desk review Project Staff Volunteers Parents Local non-governmental actors Local authorities N/A LIMITATIONS The project implemented an ambitious M&E plan that covers most of the project’s activities and produced a significant amount of data on the characteristics and progress of some of the beneficiaries. It should be noted however that the project’s staff was forced to make significant adjustments and revisions to its M&E plan during the implementation phase and faced significant problems in tracking both individual beneficiaries’ progress, and to conduct standardized testing of language and math abilities, as well as psychological resilience. This was partly due to the mid-project discontinuation of official standardized data sources on educational achievement (Prueba Enlace8), as well as the identification of data collection problems stemming from the lack of adequate volunteer and technological resources. The M&E area was able to address most of these issues and strengthen its processes, but the resulting datasets show significant gaps and are thus of limited use in reaching statistically valid conclusions for the beneficiary population as a whole, even though they do allow to analyze some of the characteristics and progress of a small subset of beneficiaries. Given the limitations of the available quantitative data, the final assessment relies more on its qualitative analysis component to reach its conclusions. Given the fact that this is a performance, rather than impact, evaluation, this is considered appropriate and efforts were made to reduce the biases inherent to qualitative methods and to enhance the representativeness of their results. In order to reduce the bias inherent to focus group and semi-structured interviews, all participants were assured that their answers would remain anonymous and no project staff member observed or participated in the 8 Prueba Enlace stands for Evaluación Nacional de Logro Académico en Centros Escolares (National Evaluation of Academic Achievement in Schools). The Prueba Enlace was a national standardized academic performance evaluation administered to elementary and middle-school students from 2006 to 2014. It measured language, communication and mathematical skills. Prueba Enlace will be replaced in 2015 by a new battery of tests named Planea, which stands for Plan Nacional de Evaluación de Aprendizajes (National Learnings Evaluation Plan), which encompasses a broader skill set. 13 interviews unless they were being interviewed themselves. Respondents were also informed that the evaluation team was not involved in the project. The evaluators were satisfied that none of the stakeholder and third-party interviewees had a personal or institutional stake in the replication of the project. Furthermore, the beneficiaries that participated in the focus group interviews were selected through a stratified random sampling method that considered gender, section (age group) and polígono of origin—all other interviewees were proposed by the project’s staff and approved by the evaluation team. Finally, structured observation was undertaken only at Camino Verde and Granjas Familiares on a limited basis and was thus complemented with the geo-referenced data collected by the project’s staff. There was no structured observation of Mariano Matamoros due to time and personnel constraints. However, one of the evaluators had visited the community previously and her experience, the beneficiaries’ responses and other available secondary sources confirmed that it has similar characteristics to the other two. The evaluation team would also like to acknowledge that the project’s staff provided video interviews with the beneficiaries which, upon review, were found to be mostly consistent with the findings of the focus group interviews. However, these videos were ultimately determined to be unsuitable for inclusion in the final analysis due to the following considerations:  The interviews were collected by the project’s staff and the interviewee-selection criterion was not clearly defined. This may have introduced some bias in the beneficiaries’ responses.  They do not reflect regular interactions.  The interviews reference processes, behavior, performance and attitudes but do not delve deeper into any of these aspects. Furthermore they are not referenced to a specific timeframe, which makes it difficult to ascertain the beneficiaries’ progress within the intervention period. 14 FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS The Scout Experience project was designed using a results-based management (RBM) approach instead of a logical framework one. The RBM approach has the distinct advantage of providing more flexibility to adjust to the conditions on the ground, by incorporating feedback from the implementation process, which is particularly useful when, as in the case of the Tijuana polígonos, there is little or no information available at the project design stage. The project’s design is thus primed for adaptation and prioritizes the incorporation of the tools developed by its education innovation lab component, as well as the information gathered by its staff and volunteers over the course of their activities. This approach led to significant adjustments over the project’s life span, which makes it difficult to provide a simple assessment of its design or its outputs and outcomes. In order to provide a more comprehensive view of the project and its results, the findings section is organized in terms of the different aspects of the analysis performed by the evaluation team. It avoids a detailed description of the project’s activities, which can be found in the Project Background section of this report and on the project’s website9. Instead, it focuses on specific aspects of the project that were estimated to have the most impact on its outputs and outcomes and, more importantly, on its replicability and sustainability. 1. Project design evaluation While the potential beneficiary population is relatively well-defined throughout the Scout Project documentation, it is nevertheless unclear how it defines its intervention group. The project’s indicators and USAID’s common indicators refer only to at-risk youth who participated in the project’s activities, but make no distinction as to the level and regularity of this participation. This would put the youth beneficiaries who participated sporadically in the project’s activities, including one-time events, on an equal footing with those who received non-formal education services and continued support throughout the project’s implementation—including those who eventually became members of one of the scout groups. This is particularly important in light of the differences in outcomes that would be expected between sporadic and regular participants. While both are arguably actual beneficiaries of the project, it would be reasonable to expect a great variance in the observed outcomes of each type. It is possible that a beneficiary that participated only sporadically in the project’s activities may have acquired some skills and received a certain level of support. The project’s own impact theory, however, assumes that lasting developments in character, education and leadership require a more regular participation—if not necessarily the formal adoption of the Scout Movement value system. Furthermore, the project’s effects 9 The project’s website can be found at http://www.experienciascout.org.mx/?lang=en. 15 on sporadic participants would be unobservable, as there is no way to meaningfully measure their progress across time: it is not possible to evaluate them at regular intervals nor can they participate in any qualitative assessment exercises10. The project partially addressed this issue by developing an alternative set of performance indicators, additional to the one was established by the Cooperative Agreement (CA) and the USAID Pillar IV Common Indicators. This second indicator set was designed mid-project (May 2014) and incorporated in the Monitoring and Evaluation Plan (MEP). It is also significantly different from the original one, as shown in Table 2, highlighting the challenges that the project encountered in collecting the necessary data for its original indicators as well as the project’s own evolution. Table 2: Outcome and output level indicators, Scout Experience: Ready for Life (SERL) Outcome/Output (narrative summary) Original indicators (CA) Additional Indicators (MEP) Outcome 1: Increase Youth Capacity (Young people 7-29 years of age in Tijuana increase their capacity to play a productive role in the community) Percentage of out-of-school youth participating in SERL activities who have not finished primary, secondary or high-school and enroll in a formal education system. Number of in-school beneficiaries participating in the SERL program that stay in primary or secondary school. Percentage of students participating in SERL activities that successfully transition from primary to secondary school. Percentage of young people participating in SERL activities that show an improvement in their psychological resilience evaluation score. Output 1.1 Character (Young people 7-29 years of age in project target zones have access to non-formal education activities that strengthen their skills and capabilities related to sociability, affectivity, character, creativity, and corporality.) Percentage of youth between 7 and 29 years of age participating in the SERL project that acquire the knowledge, attitudes and practices pursued by the educational program of the youth center. Number of Scout beneficiaries participating in the SERL program that have received psychosocial support and preventive education to increase their awareness to anti-social or violent behavior (gender￾based violence, intra-family violence, bullying, human trafficking.) 10 All of the beneficiaries participating in the focus group interviews, for example, were part of a scout group and thus regular participants in the project, as the logistics of tracking down sporadic beneficiaries for such exercises made their inclusion unviable. 16 Number of at-risk youth between 7 and 22 years of age that acquire the knowledge, attitudes and practices pursued by the educational program of the youth center. Number of at-risk youth between 7 and 29 years of age that have received at least one hour of educative program SERL activities. Output 1.2 Education (Young people 7-29 years of age in project target zones have access to non-formal education activities that strengthen their skills and capabilities in math, language, and logic, as well as their competences in ecology, humanities, communication, security, technology and sports.) Percentage of enrolled students participating in SERL activities that show improvement in their language evaluation score. Number of at-risk youth receiving reading interventions at the primary and secondary level. Percentage of enrolled students participating in SERL activities that show improvement in their math evaluation score. Number of merit badges earned that strengthen the skills for life inventory (ecology, humanities, communication, security, technology and sports) of at￾risk youth participating in SERL activities. Percentage of youth between 7 and 29 years of age participating in SERL activities that show improvement in their skills for life evaluation score. Output 1.3 – Leadership (Young people 7-29 years of age in project target zones have access to non-formal education activities that strengthen their skills and capabilities to design and implement projects in the fields of peace, environment, and social development.) Percentage of youth between 7 and 29 years old that have been or are members of local gangs participating in the youth involvement program have decided to reenroll in school or a professional development program. Number of at-risk youth participating in the SERL program that show a voluntary commitment to follow specific values and principles related to positive citizenship and community leadership. Total number of community service hours implemented by at-risk youth destined to build up community sense of belonging, pride, responsibility and environment preservation. USAID Pillar IV Common Indicators (Applicable to all Pillar IV projects, as discussed in the Project Background section) 70% of at-risk youth beneficiaries (aged 6-16) participating in USAID-funded programs that have continued their academic education in school. N/A 2500 at-risk youth (aged 7-29) engaged in USAID community project activities. Not applicable. Source: Annual Implementation Plan 2014-2015, Monitoring and Evaluation Plan (Plan de Monitoreo y Evaluación) The additional set of indicators has clear advantages over the original one. First, it provides a more accurate measurement of outputs in terms of character, education and leadership. The original indicators were better suited to measuring the effects of the services provided and were therefore unsuitable to measure outputs. Moreover, as is stated in the MEP, and corroborated by staff and volunteer interviews, it turned out to be very difficult to find, or develop, adequate standardized 17 evaluation tools for psychological resilience and language and math skills; and data collection was spotty or limited at best. By focusing on specific services provided, number of hours dedicated to specific tasks and individual benchmark achievements, the new indicators are more easily measured and more precise. The second advantage of the additional set of indicators is that it points to a relatively more precise definition of the intervention group, by distinguishing between sporadic beneficiaries and scout beneficiaries, who constitute the core beneficiary group and could therefore be characterized as the actual intervention group. Indicators related to merit badges—which are only awarded to scouts—and the provision of psychosocial support services to scout beneficiaries—as opposed to beneficiaries in general—provide a more accurate measure of the project’s true reach and, more importantly, are focused on those beneficiaries that might actually show lasting outcomes. It should be noted, however, that while the midterm adjustments were a significant improvement, the project’s original design did not make enough provisions for the requirements of the M&E process in the scale that it envisioned originally. It did not establish a baseline measurement for the beneficiaries nor did it establish mechanisms to track their progress efficiently through time; and it did not allocate enough human resources for these tasks during its first stages. As a result, as is discussed next, it is not possible to determine the magnitude of the results achieved. 2. Quantitative analysis of observed outcomes The evaluation team had access to the results of three surveys applied by the youth centers in June 2014, January 2015 and June 2015. It also had access to the results of the language, math and resilience tests applied to some of the beneficiaries, as well as to the attendance records compiled by each scout group and the results included in the Quarter Project Reports (QPR). After analyzing the data, it was determined that it was not possible to conduct a statistical analysis of the observed outcomes, as the beneficiary response rate is too low (Figure 2). Moreover, as was discussed before, the introduction of a new set of indicators that are considered more appropriate to measure the project’s outputs and outcomes discouraged the use of those included in the QPR. Figure 2: Response rate of outcome evaluations 18 3. Qualitative analysis of observed outcomes The focus-group interviews with the beneficiaries—who were grouped by ages and randomly selected in order to avoid selection bias—provided valuable information about how the four educational emphases resulted in positive changes in their lives. It was also possible to obtain valuable information regarding positive externalities regarding their families and school performance, which was confirmed by the focus-group interviews with parents11 and volunteers,12 and the interviews with other relevant actors. Nevertheless, it is necessary to point out that it would have been desirable to make use of a richer quantitative data set in order to supplement the qualitative analysis. Findings by development aspect are described next by age group: Character Age group 1 (7-11 years old) The interviewees had participated in the project for 3 to 12 months prior to the interviews. They said they became interested in participating in the project by word of mouth from their neighbors, or by watching other participants engage in outdoor activities. They did not identify nor mention positive changes in their family or school interactions while in the program, but they noted that they learned girls and boys can participate equally in all activities. They pointed out the Scout activities they enjoy the most are games and camping. The children said they trust their peers and were open and receptive to the comments shared during the session. The group also exhibited the ability to detect problems in their areas of everyday interaction, including trash disposal and local neighbor conflicts. The most significant aspect, however, was that some of the children expounded on the risks and changes in their daily routine during the rainy season. They mentioned that their homes would sometimes flood, damaging their furniture and their clothes and forcing them to sleep with their parents. Finally, most of the children participating in the focus group said their mothers, and one of the fathers, participate with them in scout activities. Age group 2 (11-14 years old) In terms of sociability, the beneficiaries expressed that they took into account the opinion and advice of their parents and teachers, but emphasized the importance of their scout leaders’ perspective. When talking about changes detected in terms of the quality of their relationships with siblings, neighbors or classmates, one of the adolescents reported he had stopped beating his sisters, which was a frequent occurrence before his participation in the project due to his family’s negative environment. The same youth said he’s receiving psychological support now. 11 All of the parent focus-group participants were female and at least one of their children was a member of one of the scout groups. 12 All of the volunteers interviewed were scouts from other scout groups in Tijuana. Two participants from other cities participated at first via remote conference but the connection was unstable and ultimately terminated. 19 The group also stated clearly that they learned during the project’s activities that boys and girls are able to do the same things. One of the youths said that because of what he now knows about gender equality, he talked about it with his family where, he reported, "machismo" is very common. The interviewees expressed they had enough trust in their fellow Scout group to share their problems and are willing to provide support if any of them have personal, family or school problems. In the same vein, they expressed they feel "like family" within the group and that they can count on it when in trouble. Age group 3 (14-21) The interviewees mentioned that they feel supported by the “brotherhood” between them: “they are my age, so they face the same problems I do.” One of the participants in the focus group mentioned she improved the way she treats her schoolmates since she became a scout, while another referenced she used to be disrespectful to her parents but has much better communication with them now, especially with her mother. “Today I’m a person who thinks before acting,” emphasized another one. Finally, the interviewees stated that it’s clear to them that men and women are worth the same and both genres can be successful in any activity. Some of them said they learned about gender equality in the program, while others said they already knew about it. Education Age group 1 (7-11 years old) The children stated that they are performing well in school and expressed appreciation for subjects such as math, science and language. They said that during the games they play as part of the scout activities they often have to use math skills. When asked if they knew about cases of children not attending school, all of the participants said that all of their peers were already attending school before joining the program. They stated that those children who required additional academic assistance received it from the volunteers. Age group 2 (11-14 years old) The group was enthusiastic about some specific competencies acquired during the project’s activities, such as language lessons, sports, public speaking –which requires research– and others related to the environment. They also expressed a sense of pride in earning merit badges. When asked about changes in their academic performance since their participation in the Scout Experience program, one of the group participants said that “as an elementary level student I used to miss a lot of school days and delivered no homework” because she had no interest in school, whereas now, as a secondary level student, she considered herself outstanding. Another participant said the leaders and volunteers helped them with school duties. The group also admitted, however, that some of their fellow scouts are still outside of the formal education system, mainly because of economic difficulties. 20 Age group 3 (14-21 years old) The group emphasized they do better in school because of the support they receive as part of the Scouts program. Of particular note is the case of a youth who re-enrolled in a formal adult￾focused education system with the support of the program. He said he had dropped out of school because of economic difficulties, but wanted to re-enroll in order to get a job and got the help and motivation to do it from a volunteer. Leadership Age group 1 (7-11 years old) Not applicable. Age group 2 (11-14 years old) The group mentioned vandalism and the presence of garbage as their communities’s main problems. When asked what they could do to address these issues, most of them said they would ask the vandals to join the Scouts. About the waste disposal problem in their communities, they pointed out that they have actually participated in cleaning activities. Age group 3 (14-21 years old) When asked about any initiatives they may have undertaken as part of the Scouts, they talked about two that were particularly meaningful to them. The first was petitioning the municipal authorities to install garbage bins in a problematic outdoor spot in Mariano Matamoros where neighbors disposed of their trash in the open. When they did not get a response, they planted trees in the spot and were happy to observe that the trash disposal decreased drastically. The second one was working on a project to get more youth to join the Scout program by visiting schools and public spaces. The focus-group interviews with parents and volunteers and the semi-structured interviews with local government officials and local non-governmental actors supported the observations of the beneficiaries’ interviews. The main findings can be summarized as follows: Parents’ focus group interviews The parents pointed out that the Scout Group is a healthy alternative for their kids, who would otherwise spend time on the street or at home with no physical activity. A few of the parents stated that they had considered the scouts as an activity for their children before but found it inaccessible due to financial or logistical issues. Others pointed out that they had forcibly signed their kids up, but after a while—and particularly after the first camping trip—they became extremely motivated. All of the parents agreed that they led very stressful lives and that it had a negative impact on their family dynamics. The scout activities provided a way to reverse some of those dynamics and improve their relationship with their children. Those who assumed leadership roles in the scout groups considered it to be an extraordinary learning experience. 21 Volunteers’ focus group interviews The volunteers pointed out that their learning curve was rather steep. While none of them had any previous pedagogical experience, they all felt they could rely on their scout experience and the tools made available to them in order to fulfill their role. Some of them pointed out that the youngest kids had no discipline at first, but the scout method was the key to developing it. All agreed that the beneficiaries responded favorably to the scout method and some of them pointed out that class-barriers were not significant after a while, although they were at first. Finally, there was a consensus that the scout uniform signaled trustworthiness and served both as protection and as a recruitment tool. Other interviews All of the local officials and non-governmental actors interviewed pointed out that the more accomplished beneficiaries were excellent and self-confident communicators. The municipal officials pointed out that the beneficiaries were polite and disciplined, while some of the non￾governmental actors stated that the beneficiaries stood out amongst their peers. 4. Structured observation findings As was noted before, structured observation was conducted in Camino Verde and Granjas Familiares, but not in Mariano Matamoros. The evaluation team also reviewed reports from local and federal authorities regarding all three polígonos and found that both Mariano Matamoros and Camino Verde were considered to be high-risk areas, whereas Granjas Familiares was considered to be a priority by the Federal Government but a mid-level risk area by the state and municipal authorities. The evaluation team also reviewed the data collected by the project’s staff regarding risk and protective factors and the location of transportation and education services, which was geo-referenced and was used to determine the optimal location of the youth centers. Please see Table 3 for a summary of the findings. Table 3: Structured observation results Polígono Risk factors Positive community assets Camino Verde  Slum with a large land area.  Considered a high-risk neighborhood by local and federal authorities.  Deficient public services (public safety, health, education, transportation, garbage collection).  Landslide/flooding risk.  Gang activity.  Child prostitution activities.  Drug trafficking.  Presence of several non￾governmental and governmental social development programs and projects, Granjas Familiares  Gang activity.  Drug trafficking.  Landslide/flooding risk.  Strong neighborhood association. 22 5. Contribution to USAID/Mexico Country Development Cooperation Strategy 2014-2018 The evaluation team was able to identify a potentially significant contribution to USAID/Mexico Country Development Cooperation Strategy 2014-2018 (CDCS) Development Objective 1: Crime and violence prevention models replicated by local stakeholders” and specifically to the “Intermediate Result 1.3: At‐ risk youth capacity to play a productive role in their communities increased”. It also identified a potentially significant contribution towards USAID’s Gender Equality and Female Empowerment Policy concerning the outcome defined as: “Increase capability of women and girls to realize their rights, determine their life outcomes, and influence decision-making in households, communities, and societies”. The evaluation team was not able to assess the magnitude of these contributions, which would require a separate impact evaluation effort, but they can nevertheless be outlined as follows: a) IR 1.3: At-risk youth capacity to play a productive role in their communities increased As noted previously, the program was successful in reaching at-risk youth in all three polígonos. And even though it is not possible to ascertain at this time what role individual beneficiaries will play in their communities in the future, it was observed that the beneficiaries had engaged in activities aimed at improving their surroundings and some had even started to develop initiatives of their own. More importantly, all three scout groups have developed a positive group identity that serves as a useful counterpoint to potentially pernicious or dangerous alternatives, such as gangs. As scouts, beneficiaries appear to become more aware of the need to set an example for their peers. b) Increase capability of women and girls to realize their rights, determine their life outcomes, and influence decision making in households, communities, and societies. As was discussed previously, focus group interviews enabled the evaluation team to identify a clear and consistent bias towards gender equality amongst the beneficiaries, which in some instances can be attributed to their participation in the project. It’s worth noting that female and male beneficiaries were expected to participate in the same activities and assume similar leadership roles if applicable. The same was observed amongst the volunteers from the regular scout groups and amongst the adult volunteers from the community. Moreover, across all age groups, females represented an average of 45% of the total number of beneficiaries13, which is a significant achievement given the context of at-risk youth in Mexico (Figure 3). 13 Considering only those beneficiaries for which age and gender data was available and that participated in the group’s activities on a regular basis. 23 Figure 3: Gender distribution of beneficiaries by age group* 6. Contribution to ongoing Government of Mexico crime prevention efforts The Scout Experience: Ready for Life (SERL) project makes two valuable contributions to the Government of Mexico’s (GOM) crime and violence prevention efforts within the framework of the GOM’s National Program for Social Prevention of Crime and Violence (PNPSVD). The first is SERL’s capacity to reach at-risk youth with a secondary-level prevention approach14, which is aligned to the PNPVSD’s Objective 2: “Reduce the vulnerability of priority attention communities against crime and violence” (Secretaría de Gobernación, 2014). The second is the learning process about the design and implementation of M&E processes in at-risk polígonos. As has been established in previous studies, identifying a clear target population and reaching it have proven to be significant challenges for the interventions implemented under the PNPSVD umbrella (México Evalúa, 2015; México Evalúa, 2014). In this respect, it should be highlighted that the SERL project has a clear and well-defined target population—i.e. at-risk youth aged 7 to 29—and the ability to reach it effectively, as could be observed in the focus group interviews, in which it was determined that the beneficiaries were exposed to one or more acknowledged risk-factors, including school desertion and domestic violence15. In this respect, the evaluation team was able to ascertain that several beneficiaries re-enrolled in a formal education system as a result of their involvement in the project and several others reported a significant improvement in their family dynamics, which constitute protection factors against crime and violence. The evaluation team was also able to observe an increase in the beneficiaries’ empathy—evidenced by their ability to concern themselves about other scouts’ problems—as a result of their participation in the Compass educational emphasis, which is also highly relevant in preventing future criminal or violent behavioral patterns (Farrington & Welsh, 2007, p. 52). 14 In a public-health model, secondary-level attention refers to actions targeted at communities in which there are significant risk-factors for either victimization or criminal behavior (UNODC, 2010). 15 See Farrinton & Welsh and (2007, p. 55) and Secretaría de Gobernación (2014, p. 24). 0 10 20 30 40 50 60 7 to 10 11 to 15 16 to 18 52 55 7 43 44 6 Female Male Source: Attendance records 2015. *Only includes beneficiaries for which gender and age data was available. 24 SERL’s approach may thus prove to be a valuable reference for other projects targeted at at-risk youth throughout the country, particularly in regards to the identification and recruitment of beneficiaries, but also to the services provided to them. The project’s M&E learning process, on the other hand, may serve as a valuable reference for other projects aimed at at-risk youth, and particularly so as to the establishment of baseline measurements and tracking beneficiaries’ progress. The issues encountered with the use of standardized testing tools and indicator design are common to other projects that the evaluation team has assessed, and therefore the experience of SERL could and should be taken into account in future implementation efforts 7. Positive/Negative externalities The analysis of both the parents and volunteers focus-group interviews, as well as the unstructured interviews with other stakeholders point out that the creation and operation of the scout groups may have had significant positive externalities. The parents’ group highlighted an improvement in family dynamics because of changing beneficiary and parent attitudes, including an improvement in their communication skills and after-school activities. The volunteer group also described a significant increase in their leadership capabilities and reported a better understanding of the problems faced by at-risk communities in their city, knowledge that would have been otherwise unavailable to them. Even though these volunteers were not, in general, at-risk youth, this should also be considered a positive externality. It is relatively less clear what the impact of the program was on local government institutions and other non-governmental organizations working in the polígonos. In the case of Granjas Familiares, the presence of the Scout Group has strengthened the activities of the local community-based organization, but this is not necessarily the case of Camino Verde and Mariano Matamoros. Furthermore, as is discussed next, there is some evidence that bureaucratic inertia and competing agendas might have limited the scope of the linkages between the project and governmental and non-governmental stakeholders. In any event, no negative externalities could be identified. 8. Project sustainability/replicability As was previously discussed, upon the completion of the project’s implementation phase the process of transferring some of the activities of the education innovation lab to Scouts of Mexico, and the scout groups to local community leadership supported by the Baja California Scout Province, as well as other scout groups already operating in Tijuana—designated as “sponsor groups”—began. With regard to the educational innovation lab component of the project, the evaluation team was provided with the documentation that will be included in the Collaborative Learning Platform/Replication Toolkit, which will be operated by Scouts of Mexico—the online system was accessible but not fully operational at the time of the evaluation. The transfer of leadership of the scout groups, including youth center facilities16, was already complete by the time this report was elaborated. This transfer included granting a revolving membership fund, which provides funding for up to 100 membership fees for volunteers and 16 The groups in Camino Verde and Mariano Matamoros operate in locations provided by the local government or community leaders to Scouts of Mexico. Granjas Familiares operates in the facilities of a Community Based Organization under a written agreement with Scouts of Mexico. 25 beneficiaries in each group, as well as setting-up Facebook accounts to facilitate internal and external communications—training was provided for the users. As was also previously discussed, the evaluation team was asked to evaluate the replicability of the project to other polígonos. In this context, the sustainability of the project results should be understood strictly in terms of the future viability of the knowledge management platform, and that of the scout groups formerly sponsored by the project. Likewise, the replicability of the project refers strictly to its intervention component—and not to the educational innovation lab effort, which will be discontinued as a centralized effort after delivering its final products and transformed into an open and collaborative one. In order to assess the project’s replicability, the evaluation team developed a simple model of the project’s intervention component, which considers only the most basic requirements for any future replication effort. Another model was developed to approximate the operation of a basic scout group and compare it to the circumstances of the groups that were sponsored by the project. It then sought the interviewees’ perspectives on different aspects of both models in order to gather a more comprehensive understanding of their validity and the present conditions in the target communities and thus, of the prospects for the project’s sustainability, results and replicability. It was determined through this process that for any replication effort to be successful, as well as to ensure the sustainability of the Scout Experience project’s results, three key conditions must be met: a) Recruitment and training of local scout leaders. As with any other scout group, the key to the survivability of scout groups created in at-risk communities depends on their leaders’ ability to recruit, motivate and sustain their members—and even more so in a relatively hostile environment. Recruiting and training leaders from the community are therefore crucial steps towards achieving long-term sustainability. In the case of the Scout Experience project, the interviewees agreed that the current leaders are committed, but some of them expressed concerns about their technical capabilities and their ability to collaborate effectively and productively with other scout leaders. b) Logistical and technical support from the national scout association and of at least one local scout group, as well as additional sources of funding. Through the interviews with staff, volunteers, parents and other local actors it was determined that the target communities would not have been able to start a scout group such as the ones resulting from the interventions on their own. It was also determined that the new groups will not be able to provide the same level of services as they did under the project, which was a cause of concern for some of the interviewees. Furthermore, the leaders themselves expressed their need of continued technical assistance and training from other scout groups and Scouts of Mexico. It was also determined that local authorities and non-governmental actors, while appreciative of the project’s results, are not willing nor able to provide support to the new scout groups at this stage, citing lack of technical skills and funding—and such support might contravene Scouts of Mexico’s preferences regarding third-party involvement. c) Ability to reach out to the target community through local leaders and local government institutions, as well as non-governmental organizations already working in the area, if any. Granjas Familiares scout group has found a solid footing in the community through the sponsorship of the local neighborhood association, which allowed the youth center to continue operating with no disruption. The Mariano Matamoros group has access to 5,000 square-meters provided by the 26 State Government through an agreement with Scouts of Mexico as well as to a local municipal sports ground (Unidad Deportiva Mariano Matamoros), but it no longer has access to the original Youth Center—and neither does Camino Verde. The Camino Verde group has found temporary accommodations at a local culture center run by a community-based organization and has access to a nearby public park (Parque Lineal), but at least two of the interviewees expressed concern about the long term viability of that arrangement. While regular scout groups are expected to make use of available local public facilities to conduct their activities, the project’s own initial assessment highlights the fact that logistical and security concerns might hinder the development of scout groups in at-risk communities. CONCLUSIONS It’s clear that the Scout Experience was successful in reaching at-risk youth in Camino Verde, Mariano Matamoros and Granjas Familiares. It is also clear that the program had positive effects in at-risk youth in terms of character, education and leadership, although it was not feasible to determine exactly in what measure. Moreover, leveraging the scout movement principles, methods and brand proved to be a crucial and effective strategy. It is still too soon, however, to determine whether the project’s exit strategy was successful. While the transfer of the scout groups created by the project to community leaders occurred without significant issues, there appears to be a consensus that the groups and the new leaders face significant challenges ahead. As in the case of the knowledge platform, the continued support of Scouts of Mexico will be crucial for the mid- and long-term success of the project. It is incumbent on the organization to make the most of the institutional learning process that took place and determine what priority it will give to this kind of initiatives in the future. Finally, it can be established that given the right conditions, the project is replicable in other at-risk communities at a much lower cost, taking advantage of the tools and methods that were tested by Scout Experience. More research and a more precise tracking of beneficiaries would nevertheless be necessary to ascertain the relative usefulness of any replication effort. But it should be noted that the project’s inherent flexibility, as it builds on the strengths of the scout method, makes it a relatively safer bet than other untested approaches. RECOMMENDATIONS In order to enhance the replicability and sustainability of the Scout Experience: Ready for Life (SERL) project, the evaluation team makes the following recommendations: 1) Set up a follow-up mechanism. There are still many unknowns about the beneficiaries’ future and the viability of the new scout groups. It is not advisable at this point to assume that the groups should operate like any other regular scout group, as there are significant risks and challenges associated with working with at-risk youth. Without decisive and continued support, the results obtained might prove to be short-lived. The primary responsibility for the follow-up should be formally assigned to both the Baja California Scout Province and the Tijuana scout groups that were designated as sponsors. It should be conducted on a quarterly basis and the results should be reported to the national association with regard to four 27 key aspects: a. Membership levels. Has the group been able to continue recruiting new members and maintain a viable membership level? b. Training. Have group leaders received further training? Are training options being fully exploited? Why or why not? c. Funding. Has the group been able to sustain adequate funding levels in order to conduct a minimum acceptable level of activities? d. Threats. Are there any new threats to the group’s continuous operations or have the originally identified ones gained further importance? Please note that the evaluation team does not recommend that local scout groups or the province organization conduct further evaluation of observed results at the individual beneficiary level. The aim of the follow-up mechanism is to ensure that the new groups have a formal support network and an “early-warning” system in place should they require assistance. The resulting reports may be relatively brief and need not entail a formal review process, but they should be taken into account by any future implementation efforts. 2) Simplify the project’s objectives and indicators. The replication of the educational innovation lab is both unnecessary and unadvisable. Future interventions should take advantage of the knowledge acquired during the current process and develop new solutions in a collaborative manner. Before designing further interventions, implementers should review the materials already uploaded to the Collaborative Learning Platform/Replication Toolkit and determine what will be the scope of their own projects. Specifically, future implementations should take into account the difficulties inherent in establishing quantitative indicators for a beneficiary population that may vary significantly within the project’s life span, as is the nature of scout groups. They should also take into account the unavailability of the standardized testing tools to measure scholarly performance and psychological development, as establishing performance indicators linked to either aspect may not be particularly useful. It is recommended that new projects adopt basic performance indicators, such as the ones that SERL introduced after the midterm review and determine whether it is possible to include more in-depth individual beneficiary development indicators depending on their circumstances. 3) Strengthen the M&E process and expand its scope. Efforts should be made so that every new project gathers information about the beneficiaries from the start and is able to track them through time. This entails determining whether or not it is possible to establish baseline values for the indicators on an individual basis, linked to a unique ID number for each individual beneficiary. If it’s not possible to establish these baseline values, the unique ID number should still be used for attendance and participation purposes. This should be considered a key staff and volunteer responsibility. Tracking the beneficiaries’ attendance should help determine which individuals could be the subject of further standardized evaluation—those beneficiaries who participated only sporadically should not be subject to further testing or assessment, as any observed results would not necessarily be linked to the project’s activities. The M&E area should be able to determine after a pre-determined amount of time whether there are enough subjects to conduct a quantitative analysis of the observed results—assuming it was possible to establish an individual baseline for the beneficiaries and that further testing is viable. If this is not the case, the monitoring and evaluation area should 28 incorporate qualitative or mixed-methods approaches that may shed light on group and community dynamics even if they can’t be traced to a particular individual. It should also be taken into account that even if there are enough baseline and progress assessments to analyze a representative sample, complementing the quantitative findings with a qualitative approach would provide a more accurate representation of the project’s results. Finally, if electronic surveys are used to measure individual progress, it is important to minimize the use of open-ended questions. This will save time when processing data and will make the resulting information more reliable. 4) Foster more cooperation with other non-governmental organizations and local authorities. Projects will themselves be more resilient if they are more integrated into the existing program and project ecosystem. More effort should also be put in identifying synergies with existing government and non-governmental programs and to promote more formal cooperation mechanisms. As was observed in the case of SERL, this is not a straightforward process and it should be consulted with ASMAC and province-level authorities. 29 ANNEXES 30 ANNEX I: TERMS OF REFERENCE Terms of Reference – Final Project Evaluation Scout Experience: Ready for life Asociación de Scouts de México, A.C. In cooperation with the United States Agency for International Development (USAID). The purpose of this document is to provide general guidance to research and evaluation institutions to perform a final program evaluation to the Scout Experience: Ready for life project implemented in the city of Tijuana by Scouts of Mexico in cooperation with USAID. The elements to be assessed in this evaluation consist primarily of: data quality, monitoring and evaluation system, program performance, development results and contribution of the prevention model to the overall youth and adolescent development environment. A combination of quantitative and qualitative elements is expected as part of this work. Context The Scout Experience: Ready for Life project is a joint effort of Scouts of Mexico and the United States Agency for International Development (USAID) to support the Mexican Government in achieving the objectives of Pillar IV “Build strong and resilient communities” of the bilateral Merida Security Initiative. After more than 30 months of implementation of this project in Tijuana, Mexico an external program evaluation is required as part of the final reporting process. Scouts of Mexico is a civil society organization with over 87 years of existence and more than 40,000 members across the country. It provides a non-formal educational program for boys and girls aged 7 to 21 years of age and offers volunteering opportunities for adults 18 years and older. The Scout Experience: Ready for life project is an Educational Innovation Lab focused on developing innovative, replicable and sustainable solutions to strengthen the capacity of at-risk children and youth in Mexico and around the world, to play an active role in their community. It supports young people in the heart of Mexico’s most at-risk communities to strengthen their character, improve their education and play an active role in society. Through the opening of Scout Groups with an intensive youth program 3 days per week, we work to ensure every child and youth is socially competent, capable of solving problems, autonomous and with a sense of purpose. Evaluation Rationale The main purpose of the final evaluation of this project is to document the results achieved by the intervention, intended and unintended, established in the project framework and provide recommendations to Scouts and USAID for future project design and implementation. The final evaluation will also provide quantitative information regarding progress on the indicators and targets established in the Monitoring and Evaluation Plan approved by USAID with particular focus on USAID Common Outcomes and their associated indicator. 31 Primary Evaluation Question Does the project design have the required elements to achieve the expected outcome level results in character, education and leadership in the intervention group? Secondary Evaluation Questions Does the project design have the required elements to measure the indicators related to outcomes and outputs on character, education and leadership in the intervention group? What is the magnitude of the achieved results for each outcome and output level indicator, as well as the contribution towards the achievement of USAID Common outcomes? Does project implementation foster any other development results or externalities than those reflected by official indicators, either for participants or non-participants in the intervention zones? Audience The primary audience of this evaluation will be the USAID Mission in Mexico and Scouts of Mexico, Scouts of Mexico and USAID Mexico will be responsible for sharing the main findings, recommendations and lessons learned with key stakeholders and to use them as an input for future projects. Evaluation and Design Methodology The consultant should use a mixed methods evaluation process. The final evaluation will report quantitative and qualitative information to respond to primary and secondary questions. For the quantitative part, USAID Mexico has approved a set of project indicators that provide information about the progress of the expected results; these indicators are stated in the Monitoring and Evaluation Plan. Scouts will provide to the consultant a database with all the information collected for the indicators established in the M&E Plan. Additionally, Scouts of Mexico has additional development indicators that will be included in the assessment. For the qualitative section, Scouts suggests the consultant incorporates the following tools:  A desk review of the project proposal, project reports and other relevant documentation related to the Project (i.e. USAID Sub-Award Agreement, USAID Program and Evaluation Policies, Project Quarterly and Annual Reports, M&E Plan, USAID Strategic Plan for Mexico, USAID Common Indicators for Crime Prevention Projects and Scouts of Mexico Educational Objectives).  Interviews and focus groups with key beneficiaries and stakeholders to assess “civic engagement and build of sense of community pride and responsibility” fostered by the activities of the project among youth beneficiaries, volunteers, parents, community leaders, local authorities and other stakeholders on the field. Potential key informants may include (but are not limited to):  Relevant civil society organizations working in the same field (i.e. Crime and Violence Prevention Program, International Youth Foundation, etc.)  Project beneficiaries (youth, parents and volunteers)  Project staff  Government counterparts 32  Stakeholders reached throughout the project cycle: local authorities, community leaders, etc.  Qualitative analysis of video-recorded testimonials from beneficiaries covering areas such as “character, education and leadership”. Pre Assessment The Evaluation Team, making use of their judgment, should decide (weighing collection time versus reliability) if enough data is available to make statistical analysis or if more data collection is required. For the qualitative analysis, the evaluation team must weight if available transcripts and videos are adequate to cover the research topics broadly enough or if more data collection is needed. Scouts of Mexico will provide to the evaluation team all the available qualitative and quantitative data previously collected as well as an exhaustive list of volunteers, community leaders and key informants contact details. The final evaluation team will require an undetermined number of field visits to the targeted zones in Tijuana, Baja California to validate the results of the intervention. Evaluation Products The deliverables expected from the final project evaluation are:  Evaluation work plan and detailed methodology, including data collection tools to be approved by USAID Mexico and Scouts of Mexico. Although the evaluation team may use their own tailored tools for data collection they are encouraged to make use of previously collected data in the field by Scouts of Mexico for their analysis (in both cases, the tools should be presented in the annex).  Fieldwork report to be delivered at the end of the data collection stage (the evaluation team should state if they decide to solely use previously collected data by Scouts of Mexico, for such case no report is required).  Draft evaluation report (in Spanish and English) based on the approved Monitoring and Evaluation Plan and including the statistical analysis of the database provided; to be submitted electronically no later than September 14 2015. The format of the report should follow the guidelines and policies listed in the annex.  Once the Draft Evaluation is delivered, Scouts of Mexico has 72 hours to comment on the Draft Evaluation Report and provide feedback to the Evaluation Team (No later than September 18, 2015).  Final Evaluation Report (in Spanish and English) that incorporates Scout’s responses to the Draft Evaluation Report must be submitted no later than 5 days later (No later than September 25, 2015). All of the above deliverables should be in compliance with USAID Monitoring and Evaluation guidelines and quality standards, as well as the policies of USAID TIPS Document series, ADS 252, 253, Annex for ADS 201 and 203. 33 Evaluation Team Composition and Desired Qualifications Team size A team of researchers and research assistants to conduct the evaluation with the following 3 key roles and their corresponding research assistants: Role 1: One Senior Researcher in charge of coordinating the Final Project Evaluation Main Functions  Provide technical direction and overall guidance to the evaluation team for all activities under the project.  Lead the evaluation (mixed method evaluation) and all data collection activities.  Desired qualifications  At least 15 years of experience conducting research on social sciences subjects.  Previous experience conducting participatory and quasi-experimental evaluation and the ability to write convincing findings and recommendations is required.  Strong organizational, writing, and communications skills.  Fluent in both Spanish and English speaking, reading, and writing.  Previous experience in formal o non-formal education research is a plus. Role 2: At least one researcher with experience in quantitative methods Main Functions  Responsible for the design, quality development and production of all data quality assessment reports.  Assist in the production of evaluations reports and any quantitative assessment.  Analyze quantitative data and the ability to translate key findings into wide written reports for a broad audience. Desired qualifications  At least 10 years of experience conducting quasi-experimental evaluations.  Strong skills in strategic analysis, data collection and analysis, identifying trends in data.  Experience writing findings and recommendations in assessment or evaluation reports. Role 3: At least one researcher with experience in qualitative methods. Main functions  Supports the Senior Researcher in the elaboration of the Final Report by collecting data via qualitative approaches and incorporates the views of participants and different stakeholders into the Evaluation Report.  Complement the Evaluation Report with study cases that support conclusions obtained via quantitative methods. Desired qualifications  Experience employing qualitative approaches (i.e. focus groups, interviews, participant observation, etc.) in social science evaluations.  Experience with logic model or outcome mapping as means of participatory evaluation. 34  Experience conducting social studies with vulnerable populations (preferably with communities of migrant origin) or experience in Youth Studies research is a plus. Management of Evaluation Budget Financial and human resources will be assigned and transferred to the Evaluation Team upon agreement of these Terms of Reference prior the start of the Final Evaluation Report. Background References USAID policies, procedures and tools can be found at: Education http://www.usaid.gov/sites/default/files/documents/1870/Education%20Strategy.pdf Evaluation http://www.usaid.gov/evaluation Youth Development http://www.usaid.gov/policy/youth Gender and Female Empowerment http://www.usaid.gov/policy/gender-female-empowerment Local Systems http://www.usaid.gov/policy/local-systems-framework Resilience http://www.usaid.gov/sites/default/files/documents/1870/USAIDResiliencePolicyGuid anceDocument.pdf Other useful references are: Results Based Management – United Nations Development Group https://undg.org/home/guidance-policies/country-programming-principles/results- based￾management-rbm/ Standards for Evaluation in the UN System – United Nations Evaluation Group http://www.uneval.org/document/detail/22 Guidelines for Scouting with Children in Especially Difficult Circumstances – World Organization of the Scout Movement http://scout.org/node/6033 35 ANNEX II: BASE QUESTIONAIRE FOCUS-GROUP INTERVIEWS Presentación: nombre y colonia, ¿en qué año están? 1. ¿Cómo están, cómo se sienten? ¿Cuáles son los problemas más importantes en su colonia? (Importante ver si identifican el crimen como uno de los problemas importantes) 2. ¿Por qué entraron a Experiencia Scout? CARÁCTER En su grupo o sección o manada, ¿sienten confianza de decirle a sus compañeros cómo se sienten? (A) Si ustedes ven o se enteran de que un compañero suyo tiene un problema, ¿qué hacen al respecto? (A) ¿Toman en cuenta la opinión o consejos que les dan sus papás, abuelos, o líderes en su grupo scout? (S) ¿Notan algún cambio en ustedes en la forma de llevarse con sus hermanos, vecinos o compañeros de escuela? (S) ¿Consideran que los niños y las niñas son iguales o hay cosas que sólo deben hacer los niños y cosas que solo deben hacer las niñas? (S) ¿Pueden darme un ejemplo de algo que haya cambiado en ustedes desde que están en los Scouts? (groserías o peleas) (C) ¿Pueden darme un ejemplo de cómo en su vida cumplen con la promesa Scout? (SOLO 2DO Y 3ER GRUPO) (C) ¿Les interesa aprender cosas nuevas? Por ejemplo, algún juego nuevo, algún deporte (CR) ¿Pueden contarme de algún proyecto o actividad que ustedes hayan propuesto y hayan hecho con los Scouts, como algún invento, nudo? (CR) ¿Han cambiado hábitos o costumbres para mejorar su salud o para ser más fuertes? (CO) EDUCACIÓN ¿Cómo van en la escuela desde que están en los Scouts? 36 ¿Alguno de ustedes no estaba en la escuela cuando entró a los Scouts? ¿Qué los animó o qué fue lo que hizo que decidieran regresar a la escuela? LIDERAZGO ¿Pueden darme un ejemplo de ideas o proyectos que hayan tenido y propuesto en los Scouts? Y en su colonia o escuela? ¿Cuáles son los principales problemas de su colonia? ¿Qué les gustaría hacer para solucionarlos o qué propondrían? 37 ANNEX III: REFERENCES Farrington, D., & Welsh, B. (2007). Saving children from a life of crime: Early risk factors and effective interventions. Oxford: Oxford University Press. IEG. (2012). Designing a Results Framework for Achieving Results: A How to Guide . Obtenido de http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.p df México Evalúa. (2015). Prevención del Delito en México: ¿Cuáles son las prioridades? Mexico: Mexico Evalúa. México Evalúa. (2015). Prevención del Delito: ¿Dónde quedó la evidencia? México: México Evalúa. OECD. (2009). Measuring Government Activity. Paris: OECD. Scouts of Mexico. (2015, May 15 ). The Project. Retrieved September 12, 2015, from Scout Experience: Ready for Life: http://www.experienciascout.org.mx/?page_id=730&lang=en Secretaría de Gobernación. (14 de Febrero de 2014). Programa Nacional de Prevención Social de la Violencia y la Delincuencia. Diario Oficial de la Federación. UNODC. (2010). Handbook on Crime Prevention Guidelines: Making them Work. NY: Criminal Justice Handbook Series. USAID. (October de 2012). Youth in Development Policy. Obtenido de https://www.usaid.gov/sites/default/files/documents/1870/Youth_in_Development_Policy_0.pdf USAID/Mexico. (April de 2014). Country Development Cooperation Strategy FY 2014 - FY 2018. Obtenido de https://www.usaid.gov/sites/default/files/documents/1869/Mexico%20CDCS%202014%202018.pdf