EVALUATION MID-TERM PERFORMANCE EVALUATION for EDUCATION FOR CHILDREN AND YOUTH ACTIVITY 2011-2017 August 2017 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Mendez England & Associates (ME&A) under contract AID-OAA-I-15-00024 / AID-519-TO-16-00002. 2 MID-TERM PERFORMANCE EVALUATION for EDUCATION FOR CHILDREN AND YOUTH ACTIVITY 2011-2017 August2017 AID-OAA-I-15-00024 / AID-519-TO-16-00002 Cover Photo: Students in School República de Nicaragua Credit: Amalia Alejandro This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Gary Woller, Team Leader; Ms. Cristina Accioly, Education Specialist; Mr. Carlos Umaña, Research Specialist; Ms. Margarita Sanchez, Gender Specialist; Ms. Amalia Alejandro, Research Specialist and Ms. Evelyn Jacir de Lovo, Policy Advisor. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 ABSTRACT The USAID/El Salvador Education for Children and Youth (ECY) Activity seeks to improve educational opportunities for lower secondary school students and out-of-school youth (OSY) and to create safer communities by increasing access to education and offering youths alternatives to criminal gang activity. This mid-term performance evaluation examines how stakeholders perceive ECY’s success in achieving these objectives and how its interventions are contributing to reducing youth gang and criminal activity, improving the learning environment in schools, reducing the student dropout rate, improving scholastic performance, and increasing the number of OSY who return to school. The evaluation concludes that, during the period of January 2013 through December 2016, ECY has had a positive effect on outcomes in each of the above areas by increasing or improving: student interest and involvement in learning, student self-esteem, cooperation among students, respect for rules and fair play, outlets for positive self-expression, strategies for resolving disputes and dealing with stress, access to psychological counseling, self-image, belief in future life prospects, emphasis on inclusion, and coexistence and harmony within the school community. Particularly effective interventions for achieving the above results include active learning methods, sports activities, and coexistence activities. The evaluation also found several issues that threaten the long-term sustainability of ECY interventions. Paramount among these issues are the failure to integrate the interventions in the Ministry of Education’s Full-Time Inclusive School initiative and the lack of follow-up technical assistance. ECY is currently implementing strategies to address these and other issues raised in the evaluation. 4 CONTENTS ABSTRACT 3 ACRONYMS 7 EXECUTIVE SUMMARY 9 1.0 INTRODUCTION 19 1.1 The Education for Children and Youth Activity 19 1.2 Evaluation purpose 19 2.0 EVALUATION METHODOLOGY 20 2.1 Evaluation Questions 20 2.2 Evaluation Scope 20 2.3 Data Collection Methods 21 2.4 Data Analysis 23 2.5 Evaluation Team and Timeline 23 2.6 Methodological Limitations 23 3.0 FINDINGS AND CONCLUSIONS 23 3.1 Evaluation Question 1 23 3.1.1 Findings 23 3.1.2 Conclusions 30 3.1.3 Recommendations 31 3.2 Evaluation Question 2 31 3.2.1 Findings 31 3.2.2 Conclusions 33 3.2.3 Recommendations 33 3.3 Evaluation Question 3 34 3.3.1 Findings 34 3.3.2 Conclusions 35 3.3.3 Recommendations 36 3.4 Evaluation Question 4 36 3.4.1 Findings 36 3.4.2 Conclusions 38 3.4.3 Recommendations 38 3.5 Evaluation Question 5 39 3.5.1 Findings 39 5 3.5.2 Conclusions 44 3.5.3 Recommendations 45 3.6 Evaluation Question 6 46 3.6.1 Findings 46 3.6.2 Conclusions 47 3.6.3 Recommendations 47 ANNEX 1: ECY MID-TERM PERFORMANCE EVALUATION STATEMENT OF WORK 48 ANNEX 2: ORIGINAL EVALUATION QUESTIONS AND SUB-QUESTIONS 63 ANNEX 3: EVALUATION MATRIX 65 ANNEX 4: BIBLIOGRAPHY 69 ANNEX 5: EVALUATION METHODOLOGY 71 ANNEX 6: LIST OF KEY INFORMANTS INTERVIEWED 76 ANNEX 7: DISCUSSION GUIDES 78 ANNEX 8: MINI-SURVEYS 85 ANNEX 9: EVALUATION TEAM 88 ANNEX 10: GOLDTREE LIEBES CASE STUDY 89 ANNEX 11: RISK/PROTECTIVE FACTORS ADDRESSED BY ECY 94 ANNEX 12: DETERMINANTS OF SCHOOL RISK RATING 96 ANNEX 13: GENDER INCLUSION REPORT 97 ANNEX 14: ECY IMPLEMENTATION ADAPTATIONS 101 ANNEX 15: ECY INTERVENTIONS 104 ANNEX 16: EVALUATION TIMELINE 105 ANNEX 17: DISCLOSURE OF ANY CONFLICTS OF INTEREST 110 6 TABLES Table 1. Evaluation Questions and Sub-Questions ..................................................................................... 20 Table 2. Number of KIIs and People Interviewed by Stakeholder Group .................................................. 21 Table 3. Number of KIIs and FGD Participants by Stakeholder Group ..................................................... 22 Table 4. Number of Mini-Surveys by Stakeholder Group .......................................................................... 22 Table 5. Average Mini-Survey Scores on the Activity’s Perceived Effect on Key Outcomes ................... 23 Table 6. Trends in Key Activity Results from the ECY Performance Monitoring System ........................ 24 Table 7. Factors Perceived to Contribute Most to Improving Key Schooling Outcomes ........................... 26 Table 8. Average Mini-Survey Scores on Activity Inclusivity ................................................................... 34 Table 9. Average Mini-Survey Scores on the Implementation of Activity Interventions .......................... 39 Table 10. Average Mini-Survey Scores by IP Staff on the ECY M&E System ......................................... 46 7 ACRONYMS Abbreviation Definition ADS Automated Directives System AOR Agreement Officer’s Representative AIS Salesian Association (Asociación Institución Salesiana) APA Learn, Practice and Apply (Aprende, Practica y Aplica) ATP Technical Advisor (Asesor Técnico Pedagógico) CDCS Country Development Cooperation Strategy ECY Education for Children and Youth Activity DO Development Objective EQ Evaluation Question ET Evaluation Team FEDISAL Foundation for Integral Salvadoran Education (Fundación para la Educación Integral Salvadoreña) FGD Focal Group Discussion FHI360 Family Health International 360 FTIS Full-Time Inclusive Schools (Escuela Inclusiva de Tiempo Pleno) FUNPRES Pro-Education Foundation of El Salvador (Fundación Pro-Educación de El Salvador) FUSALMO Salvador del Mundo Foundation (Fundación Salvador del Mundo) GIZ German Cooperation Agency for Development GOES Government of El Salvador (Gobierno de El Salvador) ICT Information and Communication Technology IP Implementing Partner IR Intermediate Result KII Key Informant Interview KM Knowledge Management LEPINA Law for the Integral Protection of Children (Ley de Protección Integral de la Niñez y Adolescencia) MINED Ministry of Education (Ministerio de Educación) M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MEP Monitoring and Evaluation Plan MINED Ministry of Education OSY Out-of-School Youth PAES Learning and Aptitude Test (Prueba de Aprendizaje y Aptitudes para Egresados de Educación Media) PAP Psychological First Aid (Primeros Auxilios Psicológicos) PESS Plan El Salvador Seguro QA Quality Assurance RCC Creative Conflict Resolution (Resolución Creativa de Conflictos) 8 Abbreviation Definition SEL Social Emotional Learning SOW Statement of Work Sub-EQ Sub-Evaluation Question TA Technical Assistance UDB Don Bosco University (Universidad Don Bosco) USAID United States Agency for International Development USG United States Government 9 EXECUTIVE SUMMARY INTRODUCTION This Executive Summary presents an overview of the primary findings, conclusions and recommendations from the mid-term performance evaluation of the USAID/El Salvador Education for Children and Youth (ECY) Activity. ECY seeks to improve educational opportunities for lower secondary school students and out-of-school youth (OSY) and to create safer communities by increasing access to education and offering youths alternatives to criminal gang activity. The ECY evaluation examined how stakeholders perceive ECY’s success in achieving these objectives – specifically how its interventions are contributing to reducing youth gang and criminal activity, improving the learning environment in schools, reducing the student dropout rate, improving scholastic performance, and increasing the number of OSY who return to school – and how Activity implementation is contributing to that success. USAID/El Salvador, the Government of El Salvador and other stakeholders will use the results of this evaluation to inform future Activity design or follow-up initiatives and to guide lower secondary school educational programming in the country. ACTIVITY BACKGROUND ECY seeks to: 1) increase and improve educational opportunities for vulnerable and disadvantaged lower secondary students and OSY; and 2) help create safer communities by providing Salvadoran youth with productive alternatives to criminal gang activity by addressing issues of inequitable access to education, particularly for at-risk youth living in environments affected by lawlessness, violence and crime. The evaluation covers ECY Objective 1- Sustaining improved educational outcomes for lower-secondary school students; and Objective 2 - Increase access to educational opportunities for OSY. EVALUATION METHODOLOGY The evaluation seeks to answer six evaluation questions (EQs) and nine sub-questions. To answer these evaluation questions and sub-questions, the evaluation used a mixed-methods data collection approach consisting of the following qualitative and quantitative methods: 1) document review; 2) performance indicators review; 3) 75 key informant interviews (KIIs) with 79 participants, 4) 75 focus group discussions (FGDs) with 428 participants; and 5) 377 mini-surveys. ECY stakeholders interviewed include representatives of the following groups: USAID, implementing partner (IP) staff, Ministry of Education (MINED) at the national and departmental levels, school directors, teachers, students, parents, OSY, community leaders, and other development organizations. For Component 1, the evaluation covered 22 schools in 13 municipalities. The 22 total schools visited are broken down into the following overlapping categories: a) 7 Cohort 1 schools, 15 Cohort 2 schools; b) 10 high-risk schools, 7 medium-risk schools, 5 low risk-schools; c) 16 urban schools, 6 rural schools. In addition, of the 22 total visited, 5 of the schools are participating in Plan El Salvador Seguro (PESS), and 2 are schools for the deaf. For Component 2 the evaluation covered 6 OSY ‘flexible modality’ groups in 5 municipalities. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The primary findings, conclusions and recommendations are summarized in the following Results Matrix. As recommendations are often the result of multiple findings and conclusions, this is reflected in the Results Matrix. 10 RESULTS MATRIX Findings Conclusions EQ1: How do stakeholders perceive the Activity’s success in addressing key outcomes?1 ECY interventions have contributed to achieving to one or more key outcomes in all sample schools. ECY interventions as a whole are effective strategies for achieving key outcomes. ECY interventions operate jointly to achieve key outcomes. ECY interventions operate as a package to achieve outcomes, albeit through different combinations of interventions and causal mechanisms. EQ1.1: To what extent do stakeholders perceive that Activity interventions have been effective in achieving key outcomes? ECY stakeholders most frequently cite Learn, Practice and Apply (APA), sports and coexistence committees as useful. APA, sports and coexistence committees are particularly effective interventions for improving a range of key outcomes. The most critical factor in preventing early youth gang infiltration is incentivizing youth to attend and remain at school during the day. Offering students opportunities to participate in extra-curricular activities outside class is an effective way to keep students at school. Most schools, however, lack the human, financial and physical resources to do so. ECY interventions as a whole address multiple risk/protective factors of youth crime prevention and contribute to a safer school-learning environment. ECY interventions are effective in addressing multiple risk/protective factors of youth crime prevention and in contributing to improving school safety. Student dropout is largely affected by economic and social factors outside of ECY’s control ECY interventions are unlikely to achieve significant improvements in student dropout rates. Vocational orientation helps youth to envision an expanded set of life options and provides them with the confidence and tools to achieve them. Vocational orientation is a particularly effective intervention to motivate youths to complete the ninth grade and attend high school. 1 Key ECY outcomes include: a) early gang infiltration, b) critical drivers and protective factors of crime prevention, c) safe school learning environment, d) student dropout/retention, e) student scholastic performance, and f) OSY returning to school. Recommendations ●ECY should provide assistance to schools to help them forge links with community-based actors to leverage additional resources for their full-time school activities. • USAID should support ECY to continue testing different sufficiency test modalities without penalizing ECY for failing to achieve output performance targets in terms of number of OSY reached. ●ECY should involve members of OSYs’ social support structures (e.g. family and friends) in an induction prior to beginning the flexible modality course, or otherwise involving them in activities at different points during the course. • ECY should provide flexible modality instructors a more intensive induction (e.g., 2 days) to ensure their command of the subject matter and their capacity to teach within the flexible modality environment and arm them with tools and resources (to the extent possible) to prepare OSY to pass the sufficiency test. ●ECY should provide ongoing feedback to flexible modality instructors on their class performance and create formal opportunities for instructors to meet together, and with relevant IP staff, to share experiences and lessons learned. 11 Findings Conclusions Recommendations APA makes learning more interesting and fun, teaches students to work independently and in groups, encourages students to invest in learning, and increases students’ self-confidence. APA is an effective intervention for improving the classroom-learning environment and increasing student scholastic performance. Fewer than 10 percent of OSY in ECY’s sufficiency test modality pass the sufficiency test. ECY’s sufficiency test modality is an ineffective intervention for helping OSY to return to school and meet their education/career goals. 12 Findings Conclusions EQ2: In what ways has MINED involvement helped or hindered Activity implementation? MINED’s primary contribution to ECY has been as convener at the department level by inviting stakeholders and permitting teachers to participate in ECY events and lending visibility and credibility to ECY within the country’s educational establishment MINED’s convening role has been critical in helping ECY achieve its current intervention outreach in terms of departments, municipalities, schools, directors and teachers. ECY has yet to be integrated in MINED’s vision and planning at either the national or departmental levels. The lack of oversight and guidance from MINED national level has trickled down to the departmental and school levels, resulting in a lack of oversight and accountability for ECY. Departmental Directors and ATPs are left entirely to their own discretion on whether and how to engage with ECY, with resulting wide variations in engagement. There exists a lack of oversight and accountability from MINED down to the departmental and school levels that adversely affects ECY implementation and results. Recommendations ●ECY and MINED should negotiate more efficient arrangements for communicating information about ECY activities to Departmental Heads, ATPs and school directors. ●USAID and MINED should identify and assign an Activity counterpart within MINED with access to key decision￾makers to plan and coordinate Activity implementation in the final Activity phase. ●MINED should establish clear guidelines with clear lines of accountability for Departmental Directors and ATPs to engage with the Activity. ●ECY should agree on a strategy with MINED to implement a capacity building strategy for key actors, including MINED national officials, Departmental Directors, ATPs, school directors. 13 Findings Conclusions EQ3: What have been the Activity’s outcomes with regards to gender and social inclusion? ECY interventions are equally open for girls, boys and disabled students. There is no evidence that schools discriminate against either sex or against disabled students in implementing ECY interventions. Challenges to inclusivity include lack of guidance from MINED on helping students with severe physical, learning and hearing disabilities. ECY training does not equip schools or OSY flexible modality courses to deal with students with severe physical, learning or hearing disabilities. Flexible modality courses do not address barriers to female participation and are disproportionately male. ECY’s flexible modality courses implicitly disadvantage female OSY by not addressing factors that contribute to their disproportionate share in the OSY population. EQ3.1: How have Activity interventions addressed issues related to gender and social inclusion? ECY has taken a number of direct and indirect approaches to promote gender and social inclusion. ECY has made a proactive effort to address issues related to gender and social inclusion with a number of successes. Schools have taken a gender and disability blind approach to inclusivity. Gender/disability blind approaches assume away structural, but hidden, constraints to female and disabled participation. Thus inclusivity gains reported by schools and in this evaluation are likely overstated. Recommendations ●ECY should investigate barriers to female participation in flexible modality courses and implement policies to address them, including, for example, assistance with transport and childcare. ●ECY should include instruction about specific barriers to female and disabled participation in its inclusivity activities with specific guidance on how to address them at the school level. ●ECY should design interventions that explicitly take into account and focus on addressing the underlying causes of exclusion. ●ECY should adapt its training curricula and technical assistance to address the unique learning and emotional needs of students with severe physical, learning or hearing disabilities, including deaf students. 14 Findings Conclusions EQ4: How sustainable are the Activity’s outcomes? Several factors threaten ECY sustainability, including: failure to integrate ECY with MINED, lack of follow-up TA to schools and teacher networks, lack of oversight and accountability, and lack of leadership by school directors. On the whole, ECY interventions and results are not sustainable at this point. Identified sustainability issues need to assume top priority over the remaining life of the Activity. EQ4.1: How have Activity interventions addressed issues related to sustainability? ECY has implemented four sustainability strategies but with limited to mixed success. ECY’s sustainability strategy has not been effective to date; however, ECY management is embarking on a new sustainability strategy focused on strengthening 1) its relationship with MINED and 2) follow-up TA. Recommendations ●ECY and MINED should jointly develop a formal, articulated structure for conducting integrated planning. ●ECY should identify a counterpart within MINED specifically tasked to work with the Activity and who has access to key decision-makers within the Ministry. ● MINED should establish a set of clear guidelines with clear lines of accountability for Departmental Directors and ATPs to engage with ECY. ● ECY should develop a new sustainability strategy that addresses all identified threats to sustainability and prioritize the strategy over the remaining life of the Activity. ●ECY should provide schools with financial or other resources to create APA guides in the short-term, while simultaneously helping schools link to other sources of financial support for this purpose over time. ●ECY should direct follow-up TA to help school directors, teachers and ATPs on how to manage learning networks and how to deliver value-added to network participants. ●MINED should establish a set of positive or negative incentives for Departmental Directors, ATPs, school directors and teachers to engage actively with ECY, without which the sustainability, replication and scale-up of ECY interventions is questionable 15 Findings Conclusions EQ5: What internal organizational capacities of the IPs have contributed to success or constraints/bottlenecks related to Activity outcomes? Stakeholders are highly satisfied with the capacity of ECY IP staff and the relevance, quality and usefulness of ECY Interventions. IPs are experts in their assigned fields, IP staff are highly qualified, and the quality of implementation by IPs has been consistently high. Challenges integrating its IPs have adversely affected ECY implementation in terms of delays in implementation, lack of coordination among IPs at assisted schools, and disjointed and ineffective quality assurance and follow-up TA. Challenges integrating the seven disparate implementing partners (IPs) into a coherent institutional entity have constituted significant constraints/bottlenecks to ECY implementation. EQ5.1: To what extent did Activity implementation adhere to planned implementation? What factors have affected implementation? ECY actual implementation adhered closely to planned implementation. ECY, for the most part, executed its work plan. Variations that occurred were mostly the result of adaptations made by ECY management. The loss of ECY’s counterpart at the MINED national office significantly affected ECY implementation and results See EQ2 and EQ4.1 EQ5.2: What was the quality of general Activity implementation? What are its strengths and weaknesses? How could it be improved? Important implementation issues include: weak follow-up TA, lack of resource sharing by schools, limited coverage of teachers in trainings, selection of the ‘wrong’ people to participate in trainings, large number of interventions at assisted schools, limited involvement by key stakeholders, and low sufficiency test pass rates. ECY has a number of implementation problems that have adversely affected its efficiency, effectiveness and sustainability. Recommendations ●ECY should create a single team responsible for conducting follow-up TA and bring it under the FEDISAL umbrella. ●ECY should tailor the number of interventions it implements in a given school consistent with each school’s interests, resources, and capacity. ●ECY should exercise greater discretion over which teachers participate in trainings -- perhaps using a pre￾training screening process – and reject unqualified teachers. ●If feasible, ECY should include all lower secondary school teachers in APA training workshops, and also provide all participating teachers the same level of training. ●ECY should include school directors in training workshops to the extent possible but at minimum in the APA training workshops. ●ECY should develop a formal set of processes by which IPs can communicate with other IPs when they find issues in the field relevant to the interventions supported by the other IPs. ●ECY should include ATPs and parents more often in training workshops and on-the-ground activities. ●For schools that are unable to participate in resource sharing interventions (e.g., robotics) ECY should involve them in a different integrated sharing network, provide resources directly to them, exempt them from participation, or offer them alternative methods to receive similar benefits. ●ECY should extend the training duration where appropriate, possibly by staggering training sessions and 16 Findings Conclusions Recommendations ECY is aware of its implementation issues and is implementing adaptations to address them. The ECY senior management team demonstrates a substantial capacity for recognizing and learning from its weaknesses and devising adaptations to address them. providing follow-up TA in-between sessions, to determine training effectiveness and where to target follow-up TA. ●USAID and ECY should conduct rolling assessments using diverse evaluation methodologies so as to determine intervention effectiveness, identify key drivers and barriers to success and recommend mid-course adaptations. ●ECY should conduct formal assessments of each major adaptation it pilots so as to determine its relative effectiveness. ●USAID and ECY should conduct rolling assessments of interventions using diverse methodologies and make appropriate adaptations to ensure that interventions are ready to hand over to MINED at Activity end. EQ 5.3: To what extent are ECY interventions consistent with MINED’s policy priorities? ECY is closely aligned with the principles and practices found in MINED’s Full-Time Inclusive School (FTIS) strategy ECY explicitly designed its interventions to be consistent with and support MINED’s FTIS strategy. 17 Findings Conclusions EQ6: What are the internal and external challenges, resolved and unresolved, related to the development and utilization of the monitoring system to track performance of the Activity components for the Activity’s key outcomes? ECY lacks formal knowledge management (KM) processes to disseminate information to IPs, MINED or schools or to use information to inform Activity strategy or implementation. The primary challenge related to the ECY M&E system is the absence of formal KM processes, which in turn inhibits institutional learning and implementation adaptations. EQ6.1: How relevant/useful are the performance monitoring indicators being collected and reported by the Activity? What, if anything, is missing? ECY’s two primary reporting frameworks (MEP and process indicators) are well designed and credible, and the current M&E staff is capable and effective and generates a substantial volume of information. The performance monitoring indicators being collected and reported by ECY are relevant and useful. There are no notable omissions from either. ECY IPs are not accessing or using information generated by the M&E system. There does not exist a learning or KM culture within or across ECY IPs. EQ6.2: What KM processes have been established at USAID and within Activity IPs to generate learning, and how well are they functioning? See EQ6. Notwithstanding the absence of formal KM processes, ECY senior management is well informed about Activity performance and uses this information to make implementation adaptations. ECY is working to integrate more knowledge management features in its M&E system. ECY’s KM strategy is a significant improvement, but it still lack formal design features to collect, disseminate and use both explicit and tacit information. Recommendations ●ECY should develop and implement KM processes aimed at creating a learning culture across Activity IPs that includes processes for collecting, disseminating, and using both explicit and tacit information across Activity IPs for the purpose of informing planning and decision-making. ●For schools falling in the red category in ECY’s new signal system, ECY should distinguish between schools that have potential to improve and those that do not have prioritize follow-up TA accordingly. ● In planning any follow-on Activity, USAID should require that that the Activity designs and implements a formal KM system that both complements and goes beyond the set of performance indicators in the Monitoring and Evaluation Plan and that utilizes a range of qualitative and quantitative methods focused on management utility rather than external accountability. 18 Findings Conclusions Recommendations EQ6.3: To what extent are USAID and IPs using performance information to evaluate Activity performance, generate learning, and make adaptations? IP staff generally perceives that the ECY M&E system generates useful information that is disseminated and acted on. Notwithstanding, IP staff and other stakeholders are not well informed about Activity performance, nor do they generally use performance information for planning or decision-making. 19 1.0 INTRODUCTION 1.1 THE EDUCATION FOR CHILDREN AND YOUTH ACTIVITY The USAID/El Salvador Education for Children and Youth (ECY) Activity seeks to reduce educational gaps in El Salvador through improving educational opportunities for vulnerable and/or disadvantaged lower secondary students and at-risk youth, aged 12-24 years living in municipalities with high crime rates.2 ECY further seeks to help create safer communities by providing at-risk youth with productive alternatives to criminal gang activity and addressing issues of inequitable access to education, including environments affected by lawlessness, violence, and crime. Funded at $25 million, ECY started in January 2013 and will end in December 2018. The ECY mid-term evaluation covers ECY Objective 1 – Sustaining improved educational outcomes for lower secondary school students; and ECY Objective 2 – Increase access to educational opportunities for out-of-school youth (OSY). Under Objective 1, ECY supports and coordinates its interventions with the El Salvador Ministry of Education (MINED) under the umbrella of MINED’s Full Time Inclusive School (FTIS initiative. Under Objective 2, ECY seeks to reintegrate OSY back into a formal school setting in support of the Government of El Salvador’s (GOES) education program EDUCAME,3 which offers six “flexible modalities” for OSY from 11 years of age to continue their formal schooling outside of the traditional education system. Of these six flexible modality offerings, ECY supports the traditional distance, semi face-to-face, night, and sufficiency test modalities. As part of the Local Solutions component of the USAID Forward Strategy, ECY is being implemented under a cooperative agreement by the Foundation for Integral Salvadoran Education (FEDISAL), which works closely with a partner network of six implementing partners (IPs): the Salvador del Mundo Foundation (FUSALMO), Salesian Association (AIS), Don Bosco University (UDB), Pro-Education Foundation of El Salvador (FUNPRES), and AGAPE Association of El Salvador.4 During the first three years of the activity (through May 2015), Family Health International 360 (FHI360) supported implementation and management of ECY through strengthening the institutional capacities of USAID local partners to implement this activity. 1.2 EVALUATION PURPOSE The purpose of the ECY mid-term evaluation is twofold: 1) determine whether ECY has been implemented as planned, identify and explain any deviations from the implementation plan, and identify any adjustments needed for the final phase of implementation; and 2) highlight lessons learned to inform future activity design or follow-ups and guide the design and implementation of future basic education-related activities under the new CDCS. The audience for the ECY mid-term evaluation is USAID/El Salvador (specifically 2 At-risk youth are those who: 1) attend school and live in poor communities or neighborhoods with high rates of crime and gang activity, 2) have learning disabilities, 3) have adult responsibilities at a young age (including teen mothers), or 4) are out of school and unemployed with little or no education, or are employed and need workforce readiness skills. 3 EDUCAME is a decentralized MINED initiative offered through the National Education Directorate that provides flexible educational services to young people and adults. 4 The Spanish Language acronyms for the ECY implementing organizations are used in the report, as those are the acronyms by which they are most commonly known. 20 the Office of Democracy and Governance), MINED, IPs, and other development organizations working in the same or similar space. (A copy of the evaluation Scope of Work (SOW) is presented in Annex 1.) 2.0 EVALUATION METHODOLOGY 2.1 EVALUATION QUESTIONS The ECY mid-term evaluation answers the six evaluation questions (EQs) and nine sub-questions found in Table 1 below. This list differs from the original list of evaluation questions included in the evaluation SOW (see Annex 2). The evaluation fieldwork quickly revealed it was not possible to answer the original evaluation questions as constructed, due to the fact that stakeholders tended to see ECY interventions as overlapping and contributing jointly to outcomes. The Evaluation Team (ET) thus made the decision to restructure the evaluation questions and sub-questions so that they align more closely with what was observed on the ground, while still covering the same questions/topics as the original set of questions. (See also Annex 3 for an Evaluation Matrix that includes the revised evaluation questions, data source, data collection method and data analysis method.) Table 1. Evaluation Questions and Sub-Questions Evaluation Questions Evaluation Sub-Questions EQ1: How do stakeholders perceive the Activity’s success in addressing the following key outcomes: a) Early gang infiltration? b) Critical drivers and protective factors of crime prevention? c) Safe school learning environment? d) Student dropout/retention? e) Student scholastic performance? f) OSY returning to school? EQ1.1: To what extent do stakeholders perceive that Activity interventions have been effective in achieving the following key outcomes: 1) Early gang infiltration? 2) Critical drivers and protective factors of crime prevention? 3) Safe school learning environment? 4) Student dropout/retention? 5) Student scholastic performance? 6) OSY returning to school? EQ2: In what ways has MINED involvement helped or hindered Activity implementation? NA EQ3: What have been the Activity’s outcomes with regards to gender and social inclusion? EQ3.1: How have Activity interventions addressed issues related to gender and social inclusion? EQ4: How sustainable are the Activity’s outcomes? EQ4.1: How have Activity interventions addressed issues related to sustainability? EQ5: What internal organizational capacities of the IPs have contributed to success or constraints/bottlenecks related to Activity outcomes? EQ5.1: To what extent did Activity implementation adhere to planned implementation? What factors internal and external to the activity have affected implementation? EQ5.2: What was the quality of general Activity implementation? What are its strengths and weaknesses? How could it be improved? EQ 5.3: To what extent are ECY interventions consistent with MINED’s policy priorities? EQ6: What are the internal and external challenges, resolved and unresolved, related to the development (e.g., baseline) and utilization of the monitoring system to track performance of the activity components for the activity’s main expected outcomes: student retention, achievement, school safety, and participation of OSY in educational opportunities? EQ6.1: How relevant/useful are the performance monitoring indicators being collected and reported (both internally and externally) by the activity? What, if anything, is missing? EQ6.2: What knowledge management processes have been established at USAID and within Activity IPs to generate internal and external learning, and how well are they functioning? EQ6.3: To what extent are USAID and IPs using Activity performance information to evaluate Activity performance, generate internal and external learning, and make activity adaptations? 21 2.2 EVALUATION SCOPE The evaluation scope covers 19 municipalities and 409 schools (Objective 1) and 11 youth training centers (Objective 2) located in the Western and Central Zones of El Salvador. The evaluation covers the time period of January 2013 (when ECY began) through December 2016. Thus, all findings and conclusions reported below are relevant to this time period only. Due to both time and resource constraints, it was necessary to limit the evaluation to a sample of municipalities and schools (for Objective 1) and a sample of youth training centers (for Objective 2). Overall, the ET visited 22 Component 1 schools in 13 municipalities belonging to following overlapping categories: 1) 7 Cohort 1 schools and 15 Cohort 2 schools; 2) 10 high-risk schools, 7 medium-risk schools, and 5 low risk-schools; 3) 16 urban schools and 6 rural schools; 4) 5 schools are participating in Plan El Salvador Seguro (PESS); and 5) 2 schools for the deaf. (See Annex 5 for a map of the schools visited during the fieldwork and Annex 12 for information on how school risk levels are determined.) The ET also visited six Component 2 OSY flexible modality groups in five municipalities. AIS facilitated five of the OSY groups, of which four participated in the sufficiency test modality and one in the semi face￾to-face modality. FUSALMO facilitated the remaining OSY group participating in the night modality. Grades attended by the OSY for the flexible modality courses included grades 7-9 in all six locations, and high school levels 1 and 2 in Santa Tecla and San Matías. 2.3 DATA COLLECTION METHODS The ET used purposive sampling methods in this evaluation. The main goal of purposive, or non￾probability, sampling is to focus on those people possessing particular population characteristics that will best enable evaluators to answer the evaluation questions. In this case, the ET worked closely with FEDISAL to identify schools and key informants within important stakeholder groups who were best positioned to provide informed answers to the evaluation questions. The purposive sampling method is appropriate when only a limited number of people can serve as primary data sources, and/or when random, representative sampling is not feasible for cost, logistical or other reasons, as is the case here (see Annex 5 for a more detailed explanation of the sampling methodology). The ET used a mixed-methods data collection approach consisting of qualitative and quantitative methods and drawing on primary and secondary sources. The data collection methods were selected based on what would achieve the highest quality and rigor in answering the evaluation questions, given the purpose of the evaluation and existing resource and time constraints. The methods used are described below. Document review: Prior to and concurrent with the evaluation fieldwork, the ET reviewed documents pertaining to the ECY Activity. A comprehensive bibliography of these documents is presented in Annex 4. Performance indicator review: ECY tracks 43 indicators that it uses to measure and report on performance, including 35 indicators in its Monitoring and Evaluation Plan (MEP) and either “process indicators” collected by IPs and reported internally. Key informant interviews (KIIs): KIIs are semi-structured individual or small group (2-3 persons) interviews conducted with key informants representing principal ECY stakeholder groups. The number of KIIs conducted and KII participants, broken down by stakeholder groups, is presented in Table 2 below. Table 2. Number of KIIs and People Interviewed by Stakeholder Group Stakeholder Group Number of KIIs Number of People Interviewed USAID 1 1 MINED National Officials 2 4 Technical Advisors (ATPs)5 7 7 5 ATP stands for Asesor Técnico Pedagógico, or Technical Advisor in English. ATPs are the MINED departmental 22 Stakeholder Group Number of KIIs Number of People Interviewed Other Development Programs 1 1 Objective 1 IPs 29 31 Directors 21 23 Objective 2 IPs 6 4 Teachers 3 3 Community leaders 5 5 Total 75 79 Focus Group Discussions (FGDs): FGDs are semi-structured and moderated group interviews (typically consisting of 4-8 people per FGD) and were conducted with teachers, students and parents at Component 1 schools and with OSY and their parents at Component 2 flexible modality courses. The number of FGDs conducted and FGD participants, broken down by stakeholder groups and the sex of participants, is presented in Table 3 below. (The KII and FGD discussion guides can be found in Annex 7.) Table 3. Number of KIIs and FGD Participants by Stakeholder Group Stakeholder Group Number of FGDs Number of Participants Total | Female | Male Objective 1 Teachers 25 93 53 40 Students 28 222 107 115 Parents6 14 56 39 17 Objective 2 OSY 6 37 18 19 Parents 2 20 13 7 Total 75 428 230 198 Mini-Surveys: Mini-surveys are short, closed-ended questionnaires embedded in the KII and FGD discussion guides for IPs, directors, teachers, students, and parents. All mini-survey questions used a 10- point Likert scale, with a 1 indicating “Disagree Completely” and a 10 indicating “Agree Completely.” Overall, the ET administered mini-surveys to 377 representatives from the stakeholder groups found in Table 4 below. (The mini-survey questionnaires can be found in Annex 8.) Table 4. Number of Mini-Surveys by Stakeholder Group Stakeholder Group Number of Mini-Surveys IPs 15 Directors 21 Teachers 85 Students 212 Parents 44 Total 377 field staff responsible for providing technical assistance directly to schools. 6 Many children live with grandparents or other adult relatives. Thus the category “parents” here includes these other adult caregivers as well. 23 2.4 DATA ANALYSIS Qualitative data analysis involved coding the data using Atlas.ti to identify themes and sub-themes in the data assessing their relative importance in answering the evaluation questions, supported by quotations and examples from the individual KIIs or FGDs. Quantitative data analysis method consisted of calculating summary statistics of ECY monitoring and evaluation (M&E) data and mini-survey results. 2.5 EVALUATION TEAM AND TIMELINE A description of the ET is presented in Annex 9 and the timeline for completing the ECY mid-term evaluation is presented in Annex 16. 2.6 METHODOLOGICAL LIMITATIONS The ECY evaluation design was subject to a set of methodological limitations. These limitations, and the approaches taken by the ET to address them, are described at length along with the evaluation sampling methodology in Annex 5. 3.0 FINDINGS AND CONCLUSIONS The presentation of the findings and conclusions is organized by the evaluation questions listed in Table 1 above. The quotes presented below to demonstrate the evaluation findings are not exhaustive of all quotes made by KII and FGD respondents, but are instead intended to be reasonably representative of the statements made and the common themes that emerged during the KIIs and FGDs. For the reader’s further reference, the findings and conclusions below refer to the set of ECY interventions found in Annex 15. 3.1 EVALUATION QUESTION 1 3.1.1 Findings Stakeholders perceive that ECY interventions have contributed to achieving one or more key outcomes in all sample schools Stakeholders uniformly agree that ECY interventions have improved schools in one or more of the following key outcomes: early youth gang infiltration, critical drivers and protective factors of crime prevention, safe school learning environment, student dropout/retention, student scholastic performance, and OSY returning to school.7 This qualitative finding is corroborated by two quantitative sources: the mini￾survey and the ECY performance monitoring system. The mini-survey results shown below in Table 5 confirm that stakeholders strongly agree that ECY interventions have, on the whole, positively affected youth early gang infiltration, safe school environment, student behavior, students completing 9th grade, student scholastic performance, and OSY returning to school. Table 5. Average Mini-Survey Scores on the Activity’s Perceived Effect on Key Outcomes Early Gang Insertion Safe School Environment Student Behavior Complete 9th Grade Student Scholastic Performance OSY Return to School Risk / Protective Factors 8.9 9.0 9.0 9.3 9.1 8.6 9.0 7 The ET did a disaggregated analysis of mini-survey results for all mini-survey questions by stakeholder group, sex, department, municipality, cohort, rural vs. urban schools, and risk level. In all cases, the scores tended toward the extreme range of the scale and did not change significantly enough to warrant including them in the tables. Ninth grade completion rate in target schools 81% 81% 92% 91% 92% 93% NA NA Table 6 below shows the actual results for the group of ECY-assisted schools for 2013-2016/17 for eight different MEP indicators. From the baseline through 2016/17, assisted schools reported higher ninth grade completion rates; improved student scholastic performance in math, science and language arts; increased school safety; and improved school inclusivity. The results in Table 6, however, do not confirm overall improvements in student drop-out rates and class repetition rates, albeit in the latter case, repetition rates did fall from their high of 5.7 percent in FY 2013 and FY 2014 to 5.2 percent by FY 2016, marginally higher than the baseline value of 5.0 percent. Some caution in interpreting the dropout rates, however, is in order. Stakeholders commonly noted that dropout rates are influenced in the short-term by factors largely outside of the Activity’s control (e.g., economic pressures, emigration, pregnancy, family conflicts, etc.). Thus more time would be required for ECY interventions to translate into noticeably improved dropout rates. As explained by a key informant at FHI360, “One must be careful about the dropout rate. There is not a single condition or situation that can influence that result. …The reasons for dropping out in lower secondary school involve many social issues.” Thus, “Up to three years would be needed to see a decline in the dropout rate, and to see the combined results of positive youth development.” Table 6. Trends in Key Activity Results from the ECY Performance Monitoring System Indicator ECY Total Target Baseline FY 2013 FY 2014 FY 2015 FY 2016 FY 2017 Cumu lative Female 93% 91% 92% 94% NA NA Male 90% 90% 91% 92% NA NA % students who say they feel safe in schools 80% 46% 46% 88% NA 86% NA NA Female NA 89% NA 87% NA NA Male NA 88% NA 86% NA NA Victimized and Aggression Index8 50% 60% 60% 75% NA 13% NA NA Female NA 73% NA 11% NA NA Male NA 77% NA 15% NA NA Drop-out rate in lower secondary education 3.5% 5.7% 6.3% 6.8% 7.2% 7.7% NA NA Female 5.9% 5.9% 6.7% 7.1% NA NA Male 6.8% 7.7% 7.7% 8.5% NA NA Repetition rate in lower secondary education 5.56% 5.0% 5.7% 5.7% 5.5% 5.2% NA NA Female 4.1% 4.2% 3.9% 3.7% NA NA Male 7.3% 7.2% 7.1% 6.7% NA NA % students who achieve good performance in mathematics, science and language-arts in lower secondary schools with USG support 65% NA NA NA NA 93.1% NA NA 8 The Victimized and Aggression Index measures the level of students’ sense of having been “victimized or attacked” in assisted schools. The index is calculated based on 100 percent from the victimization and aggression items of the school's assessment for each group of schools. 24 25 Indicator ECY Total Target Baseline FY 2013 FY 2014 FY 2015 FY 2016 FY 2017 Cumu lative # USG supported schools or learning spaces meeting criteria for safe schools program 665 0 0 0 119 334 142 595 # schools implementing the FTIS9 677 0 0 0 0 380 176 556 Stakeholders perceive that ECY interventions operate jointly to achieve key outcomes albeit each intervention through different causal mechanisms Stakeholders generally could not identify which ECY interventions contributed most to a particular outcome but rather tended to see interventions as operating jointly to produce observed outcomes. Although stakeholders largely perceived outcomes as the result of multiple ECY interventions, they were able to describe unique perceived “causal mechanisms” through which each intervention acted to produce outcomes. These perceived causal mechanisms are described below. ● Learn, Practice and Apply (APA) improves key outcomes by 1) making learning more interesting and fun, 2) teaching students to work independently and in groups, 3) encouraging students to invest more in their own learning, and 4) increasing students’ self-confidence. ● Sports activities improve key outcomes by 1) imparting respect for rules and encouraging cooperation among students, 2) providing direct incentives for students to remain at school as a condition for participation, and 3) offering students a safe space outside of the classroom. ● Extra-curricular activities improve key outcomes by 1) providing students a safe space where they can pursue interests or discover latent talents, 2) offering students constructive outlets for self￾expression and coping with stress, 3) offering students a safe space outside of school hours, and 4) enhancing students’ self-confidence and self-esteem.10 ● Creative conflict resolution (RCC) and psychological first aid (PAP) improve key outcomes by helping students, teachers and school administrators adopt healthier strategies for resolving conflicts, dealing with emotional crises and coping with stress. ● Coexistence and youth leadership activities improve key outcomes by 1) providing students opportunities to build self-confidence and enhance leadership skills and 2) making students more strongly connected to their school communities. ● Vocational orientation activities improve key outcomes by: 1) providing students with a greater sense of long-term perspective, 2) offering students the alternative to pursue positive life options, and 3) creating a sense of hope that such positive options are within their reach. 9 This indicator measures the number of supported schools meeting at least three out of the following five criteria related to the FTIS approach: inclusiveness, full-time, redesigning the classroom, democracy and participation, and improving teachers’ spaces and personal development. 10 Extra-curricular activities refer to activities falling generally under “arts and culture,” including theater, dance, music, etc. Mediation Committee as a Tool to Improve the School Learning Environment One school with a history of student conflict (Bernardo Villamariona) has created a “mediation committee” consisting of from 8-10 male and female lower secondary school students. Committee members are trained to mediate disputes between students so as to find a peaceful resolution, rather than just administer punishment as was done in the past. School administrators and students claim that the Mediation Committee has contributed reduced violence at the school and has further helped to create a safer and more harmonious learning environment. 26 Stakeholders consistently highlight APA, sports and coexistence committees as useful interventions in achieving key outcomes Stakeholders hold diverse opinions as to which ECY interventions have been most effective in improving student and school outcomes. Perceptions vary depending on, among other things, the stakeholder’s role in the education/learning process, the type of contact he/she has had with the Activity, and his/her own personal inclinations. During interviews, different groups of stakeholders tended to emphasize different ECY interventions as being the most effective in generating improved student and school outcomes. Table 7 shows which interventions different stakeholder groups tended to mention most during interviews.11 Stakeholders consistently identify APA, sports, and coexistence committees as particularly effective interventions in improving student and school outcomes generally. “If the teachers are effective in helping students see what matters, what qualities they possess, and what they can achieve, then when they leave the school they are not going to join gangs because they now have higher self-esteem. To do this, APA empowers teachers to improve teaching and to form better relationships with students.” –FUNPRES Technical Specialist “In the case of prevention, sports, coexistence activities, and vocational orientation are spaces where students can develop their leadership potential and develop their skills to have a successful life. There are cases where gangs are already affecting students, but in the school they participate in activities to help them become positive leaders and develop their potential to be something different. In this way, such activities help youth avoid joining gangs, converting their leadership into something positive.” – FUSALMO Technical Specialist. What follows immediately below describes the causal mechanisms by which stakeholders perceive ECY interventions to produce each of the above key outcomes. Table 7. Factors Perceived to Contribute Most to Improving Key Schooling Outcomes Stakeholder Mentioned Most Frequently Mentioned Often but Less Frequently Male students Sports, APA, ICT Clubs, Coexistence committee Female students APA, Coexistence committee Sports, Directed recreation, Clubs, RCC Parents APA, Sports, Clubs Coexistence Directors APA, RCC, Sports, Vocational orientation Coexistence committee, Youth leadership ATPs APA, Sports ICT Early Youth Gang Infiltration Stakeholders perceive that the most critical factor in preventing early youth gang infiltration is incentivizing youth to attend and remain at school during the day12 Put simply, youth who are at school are not “in the streets,” where they are most vulnerable to gang recruitment. This finding is consistent with the common theme expressed by stakeholders that schools offer students a critical refuge from the outside world and the pressure to participate in gangs or other criminal activities, or to engage in other behaviors that negatively affect their long-term schooling and life prospects. Keeping students at school and off the streets, however, requires that schools provide students more opportunities to participate in sports and extra-curricular activities outside of normal school hours. This is a specific aim of ECY’s sports monitoring, robotics, video games and art and culture interventions, 11 Senior officials at the MINED national office declined to offer an opinion as to the relative effectiveness of different ECY interventions. 12 Lower secondary school students in El Salvador attend classes either during a morning shift or afternoon shift, a policy necessitated by lack of physical space relative to the number of students. Thus the objective is to encourage morning shift students to remain at school during the afternoon and afternoon shift students to come to school in the morning. 27 consistent with MINED’s FTIS initiative. However, while all of the 22 schools visited were offering some combination of sports and other extra-curricular activities, relatively few were offering these activities during off school hours. Several reasons were identified for this gap. First, the schools lack the physical facilities to accommodate students during their off shift while other classes are in session. Second, teachers are not available to supervise the activities; they are either in the classroom teaching during their on shift, or they are not at school during their off shift (e.g., they are at home or are teaching at another school). “Robotics also keeps students busy in the afternoon. They stay at the computer lab instead of going to the street. It has helped the students not to keep bad company because they are now distracted and are always busy and barely walk in the street.” -Teacher Risk/Protective Factors of Youth Crime Prevention Stakeholders perceive that ECY interventions as a whole address numerous risk/protective factors of youth crime prevention via multiple causal mechanisms The table in Annex 11 shows how ECY initially mapped its interventions to risk/protective factors of youth crime prevention. In unprompted responses during KIIs and FGDs, ECY stakeholders mentioned that ECY interventions addressed each of the risk/protective factors in this table, albeit in different combinations depending on the stakeholder, thus confirming both ECY’s initial mapping linking its interventions to key risk/protective factors and the interventions’ effectiveness in addressing these factors. Safe School-Learning Environment Stakeholders perceive that ECY interventions as a whole contribute to a safer school-learning environment via multiple causal mechanisms Stakeholders generally agreed that ECY interventions have contributed to creating a safer school-learning environment. For example, APA has fostered a positive and safe learning environment with teacher-student relationships at their center. Sports and supervised recreation served to weaken or break down barriers to cooperation and create more harmonious coexistence. In the latter case, stakeholders cited examples where previous antagonists (e.g., gang members) have worked together to help each other learn or succeed in competitive events. Stakeholders also generally agreed that activities planned/organized under the aegis of the coexistence committee were effective in helping to create a safer and more harmonious school learning environment for students.13 Finally, RCC improves a school’s learning environment by providing students, teachers and administrators alternative methods and training for resolving conflicts that inevitably arise during school operations. “In a certain sense, the school is a bubble of protection in which students feel safe, and as such, the children perform better. Today we have a better environment for learning. Before, problem students would charge kids ‘rent’ or ‘a quarter [25¢]’ to leave them alone. Now these problem students see that everybody is busy having fun, involved in different activities, so they also want to participate in the activities.” –School Director. 13 Whether a coexistence committee existed and how it was functioning varied widely from school to school. According to one FUSALMO technical specialist, only about 15 of the 28 schools he/she is responsible for have an active coexistence committee. Reducing Gang Harassment of Students At Concepción de María, gang members used to hang out on the corner harassing students. To address this, the school decided to approach it as an opportunity rather than a problem. The teachers talked to the gang members and proposed that in place of hanging out and doing nothing, why don’t they participate in school activities? The school offered them a flexible schedule that suited their needs. Now they attend school on Tuesdays, Wednesdays and Thursdays and the incidence of gang harassment of students has fallen. 28 I like the coexistence committee because it shows us that we have the capacity to do what we want in our life. We are all equals. It has helped us get along with our fellow students, teachers, and students. When boys or girls have problems, the coexistence committee helped them.” –Student. “When there is a conflict, I now try to use mediation. In the past, I just suspended the kids for three or four days. Now [because of RCC] I work for a mutually agreeable solution.” –Teacher. Student Dropout/Retention Stakeholders generally agree that student dropout is largely affected by economic and social factors outside of ECY’s control There are examples, however, in which schools participating ECY inclusivity now make efforts to combat student dropout. For example, three schools (San Juan Buena Vista, Aldea de la Mercedes and Teresa Menendez Viuda de Escalante) report visiting the homes of students who have left school for various reasons, including pregnancy, to invite them back and explore options for doing so. Student Completion of Ninth Grade Stakeholders consistently highlight vocational orientation as useful for encouraging students to complete the ninth grade and attend high school Stakeholders widely agreed that ECY interventions have contributed to increased ninth grade completion rates. According to stakeholders, interventions that increase student interest and participation in learning and that improve the school-learning environment incentivize students to complete the ninth grade. However, the ECY intervention stakeholders credit most with increasing ninth grade completion rates is vocational orientation. In the vocational orientation intervention, ninth grade students take an aptitude test to determine their abilities and interests, receive counseling on schooling and career options, and complete a “life project” in which they define their career goals (i.e., what they want to be in life). “Many youth have lost faith that they can accomplish anything. They believe what they have been told, that they are no good and that because they are children, they have to remain at home doing domestic chores or unskilled work. Vocational orientation combats dropouts by motivating students, giving them information, and helping them explore themselves. The project has invested a great deal of time into developing a test for self-awareness and providing other resources so that youth have information about schools, careers, scholarships, and themselves in this context.” –School Director. Increasing the Ninth Grade Completion Rate with Vocational Orientation As part of its vocational orientation activities, one school (Republica de Nicaragua) participates in a fair with other schools hosted by its municipality Quezaltepeque in which high schools provide information about their schools so students understand their high school options. Each of the school’s 126 ninth grade students went on to high school in the most recent school year, compared to only about 30 percent in the previous year. Student Scholastic Performance Stakeholders consistently highlight APA as useful for improving the classroom-learning environment and increasing student scholastic performance According to stakeholders, APA has significantly improved the classroom-learning environment compared to the traditional “lecture and copy” teaching method, and this has contributed in turn to improved student scholastic performance in terms of increased student learning autonomy, student classroom attendance and participation, classwork Applying APA Using Practical Examples One school (Caserío Santa Paula) incorporates practical examples in its APA guides to teach students math concepts. For example, to help students understand and calculate dimensions, teachers take students outside to the school gardens where they learn how to measure length and width of the garden plots to calculate area. 29 and homework completion rates, and student subject matter mastery. APA provides teachers with the tools and capacity to plan and implement innovative pedagogical learning approaches focused not only on content knowledge, but also on competencies (communication, leadership, collaboration). “APA makes classes more dynamic, the quality of teaching and learning improved, and the guides helped kids to develop the habit of doing research to answer questions. Consequently, students are more engaged in learning and have improved their performance and grades.” –School Director. “APA is not only good, it is very good for improving students’ academic performance. We have not only implemented APA in the third cycle in this school, but from the first cycle through high school. We are now working on implementing APA [at other schools] in the entire integrated system.” –School Director. OSY Returning to School ECY has assisted over 15,000 OSY return to school via its flexible modality courses; however, over 90 percent of OSY who take the sufficiency test modality ultimately fail the sufficiency test Through 2016, ECY has supported 15,643 OSYs, including 5,804 (49.6 percent) females and 5,892 (50.4 percent) males, to return to school through one of its four flexible modality offerings. For OSY participating in the ECY flexible modality courses, their primary motivation is to earn a higher income. Men, and to a lesser extent women, cited the need to work as a primary reason for leaving school in the first place, while women commonly cited pregnancy as a primary reason for leaving. Under Objective 2, ECY has grafted its own flexible modality course onto MINED’s flexible modality courses offered as part of EDUCAME. For the distance, semi face-to-face, and night modalities, ECY has grafted a life skills module and a refresher course in math and language. Youth participating in these three flexible modalities meet once a week on the weekend over a 5-week “pause” in their GOES flexible modality course. According to AIS, the advantage of this approach is that it helps bring participants to a minimum competency level in math and language and gives them greater self-confidence. Nevertheless, no evidence exists yet as to how OSY participating in ECY’s five-week “pause” perform relative to their counterpart OSY who do not participate in it. For the sufficiency test modality, OSY in grades seven through second year high school attend an accelerated three-month, 96-hour course that includes 30 hours of math instruction, 18 hours of language, 16 hours of natural science, 16 hours of social studies, and 16 hours of life skills. Youth meet once a week on the weekend for four hours per session. The four-month length of the sufficiency test module acts as a powerful inducement for OSY to enroll in the course. At the conclusion of the sufficiency test modality, OSY in grades 7-9 take the sufficiency test, and OSY in high school take the Learning and Aptitude Test (PAES) for their respective grades. Passing either test certifies that the OSY has met the minimum requirements to advance to the next grade. Two factors, however, limit the effectiveness of the ECY sufficiency test modality: 1) low sufficiency test pass rate and 2) high OSY dropout rate. Low sufficiency test pass rate: 14 AIS estimates that, of the 1,126 youth who completed the sufficiency test modality between 2014 and 2016, only about 107 youth (9.5 percent) passed the sufficiency test, marginally better than the overall national pass rate of 7 percent for OSY taking the sufficiency test. (ECY has yet to receive data on the pass rate for the PAES among high school youth participating in its 5-week “pause.”) A number of factors contribute to this low pass rate: ● There is a wide disparity in abilities in math, natural science, language, and social sciences within the OSY taking the sufficiency test modality. The modality does not include a basic refresher/primer to bring participants to a minimum competency level before instruction begins. Instead, the instructional material and lesson plans are geared to the average ability of participants 14 The sufficient test pass rate is the percentage of OSY who take and pass the sufficiency test. Passing the sufficiency test indicates that the OSY has demonstrated the minimum level of knowledge in the four test subjects – math, language, natural science, and social science – to advance from one grade to the next. 30 based on an initial diagnostic administered at the beginning of the course. Thus many OSY begin the modality from a low competency base and struggle to learn the material. ● The compressed time period (three months) makes it difficult for youths to master the subject material at a level required to pass the sufficiency test. ● OSY spanning up to five grades are combined together in the same classroom with the same instructor. This leaves very little time for instructors to cover the subject matter for all grades, let alone to provide feedback on completed work, address all of the students’ questions, and/or provide extra individual assistance to those students who are struggling more than others. ● Many OSY have been outside of the formal education system for several years and lack good study habits. Moreover, many have jobs and/or family obligations that make it difficult for them to complete their homework. ● OSY lack sufficient support networks to motivate them or help them to study. Family, friends, and other members of their social network often fail to provide OSY with the emotional or tangible support that is vital to their educational success (and may actively discourage them in many cases). In addition, family and friends of OSY often lack the capacity to adequately support them and/or place demands on their time, which distract OSY from their academic focus. ● ECY flexible modality instructors receive only a short half-day immersion in the relevant flexible modality course they will be leading, and afterward receive little to no assistance in terms of instructional methods, course materials, and/or feedback on classroom performance. ECY also does not provide instructors with opportunities to meet together with their peers or technical staff to share experiences and lessons learned. ● Not all ECY instructors are certified by the GOES in flexible modality instruction. For example, the ET spoke to at least two instructors who were not certified. ● There are no practice tests, no question bank, and no on-line study resources. Thus, instructors cannot teach to the test and students do not know how to focus their study preparation. High OSY dropout rate: AIS estimates that about 30 percent of OSY who enroll in the sufficiency test module drop out. Key informants attribute the high dropout rate to work conflicts, security problems related to crossing gang boundaries or walking long distances, child care demands, transportation costs, difficulties learning the material, and fear of failure. “The teacher starts teaching seventh grade math, then eighth grade, and finally when he gets to ninth grade math, time is over. We barely have enough time to learn what we’re supposed to.” –OSY. “MINED is very apprehensive about its sufficiency test. We work based on the syllabus that EDUCAME provides us, based on these subjects, and we prepare the classes and learning materials. But when the students take the test, they say that questions appear that we have not covered in class. We communicated the youths’ concerns to MINED, but we have not obtained an answer.” –AIS Technical Specialist. 3.1.2 Conclusions EQ1: ECY interventions, taken as a whole, are effective strategies for achieving key outcomes. The interventions operate as a holistic package to achieve these outcomes, albeit in different combinations of interventions and with each intervention operating through different causal mechanisms. EQ1.1: While all ECY interventions contribute in one way or another to achieving key outcomes, APA, sports activities and coexistence committees are particularly effective interventions for improving a range of key outcomes. The key to reducing early youth gang infiltration is to incentivize youth to remain at school by providing them opportunities to participate in sports and extra-curricular activities outside the classroom and outside of school classroom hours. This requires strategies to address the human and physical resource constraints that hamper schools’ abilities to offer students these activities when they are not in class. 31 Each of the ECY interventions is effective in addressing multiple risk/protective factors related to youth crime prevention, while different interventions contribute in different ways to improving the safety of the school-learning environment. No particular intervention, or set of interventions, stands out from the other interventions in terms of addressing the risk/protective factors of youth crime prevention or creating a safer school-learning environment. ECY interventions as currently designed can be expected to achieve only marginal improvements in student dropout rates. Achieving greater improvements in dropout rates will require interventions that more directly address the social and economic factors contributing to the reasons why students drop out. In terms of students completing the ninth grade, vocational orientation is a particularly effective intervention at motivating students to complete the ninth grade and attend high school. It helps youth to envision a larger and better set of life options and provides them with the confidence and tools to achieve them. With regards to student scholastic performance, APA is an effective intervention that improves the classroom-learning environment and, in turn, students’ scholastic performance, particularly in contrast to traditional lecture and copy teaching methods. APA helps make learning more interesting and fun, teaches students to work independently and in groups, encourages students to invest more in their own learning, and increases their self-confidence. The ECY sufficiency test modality it is not effective as clearly evidenced by the low sufficiency test pass rate among the OSY participating in it. The low sufficiency test pass rate combined with the high dropout rate mean that a small fraction of OSY who initially enrolled in the modality are likely to achieve their goals of earning a higher income. This low success rate has the potential to set back ECY’s gains in this area, both by damaging the ECY brand and by discouraging OSY from continuing to pursue their formal education. The effectiveness of ECY’s five-week “pause” course, offered as part of its traditional distance, semi face-to-face, and night flexible modality courses, cannot be determined at this point. 3.1.3 Recommendations ● ECY should further provide schools guidance on how to manage their use of existing resources (e.g., money, human resources, and physical space) and to identify and forge links with external resources to support their full-time school extra-curricular activities. Possible external resources include the Mayor’s office, local police, health centers, local NGOs, civic clubs or organizations, etc. ● USAID should support ECY to continue testing different sufficiency test modalities without penalizing ECY for failure to achieve output targets (e.g., number of OSY returning to school); sufficiency test pass rates are more important than throughput. ● ECY should involve members of OSYs’ emotional and social support structures (e.g., family and friends) in an induction prior to beginning the flexible modality course, and/or involve them in activities at different points during the course. ● ECY should provide flexible modality instructors a more intensive induction (e.g., two days) to ensure their command of the subject matter and their capacity to teach within the flexible modality environment and equip them with tools and resources (to the extent possible) to prepare OSY to pass the sufficiency test. ● ECY should provide ongoing feedback to flexible modality instructors on their class performance and create formal opportunities for instructors to meet together, and with relevant IP staff, to share experiences and lessons learned. 3.2 EVALUATION QUESTION 2 3.2.1 Findings The loss of ECY’s counterpart at the MINED national office has led to a lack of oversight and accountability from the Ministry down to the departmental level 32 Toward the end of 2015, MINED abolished the ECY counterpart position at the Ministry, which was responsible for coordinating with ECY, without putting in place a replacement mechanism. This had three significant implications for the Ministry’s involvement with ECY at the national level. The first was the loss of institutional memory related to ECY. According to senior officials at the MINED national office, while they are generally familiar with ECY, they are not familiar with its work plan. The second has been the failure to integrate ECY and its interventions into the vision or planning either at the national Ministry level or in the national education system. Third, the failure to integrate ECY into the vision and planning at the national level has contributed directly to the failure to integrate ECY and its interventions into Ministry planning or activities at the departmental level.15 There is no consistent message about ECY from the national Ministry to the Ministry Departmental Directors, and this is accompanied by a lack of oversight and accountability from the national level to the departmental level. This, in turn, has trickled down from the Departmental Directors to the ATPs, who are similarly left without clear oversight and/or accountability. As a result, Departmental Directors and ATPs are largely left to their own discretion with regards to ECY. “The Ministry has abandoned us. What I do is of my own volition, as I do not have the means or resources to visit, and there is not good communication between the central level with the departmental or local levels. I never know what is happening or what will be done at the national level.” –ATP. “It is not the same at one place as another. Some ATPs are more active than others. It has to do with the Departmental Director and his leadership, as well as with the assigned work team. ATPs can be great, but there are others who do not want to do things.” –School Director. The level of engagement by Departmental Directors and ATPs with ECY depends on their own internal motivation The extent to which Departmental Directors and ATPs engage with ECY varies along a continuum from no engagement to active engagement. Thus, their role in ECY depends on where they are along this continuum, and where they are along this continuum depends largely on their own internal motivation. In the case of ATPs, their internal motivation is strongly influenced by the conditions under which they work. Each ATP is responsible for up to 30 schools, often without transportation assistance. Even for highly motivated ATPs, visiting all their schools, let alone engaging them on ECY, is difficult. ATPs also rotate frequently from one location to another, making it difficult for them to achieve any kind of continuity in terms of their engagement with schools on ECY. (One school visited by the ET, for example, has had three ATPs since Activity inception.) MINED’s primary contribution to ECY has been via its convening role at the departmental level MINED’s primary contribution to ECY has been via its role as convener at the department level responsible for: 1) inviting stakeholders to participate in ECY events, including trainings, educational workshops, teacher pedagogical networks; 2) permitting directors and teachers to participate in these events; and 3) lending visibility and credibility to ECY within the country’s educational establishment. MINED’s convening role. As a rule, MINED has played its convening role well thereby ensuring that Activity interventions have proceeded more or less apace consistent with their implementation plans. That MINED has played its convening role well has been critical to ECY implementation given ECY’s administrative structure that strictly limits the situations in which IPs can communicate directly with schools thus requiring them to use Departmental Directors as intermediaries to plan and implement events and ensure participation in those events by school directors, teachers, and students. 15 El Salvador is divided into 14 departments for administrative purposes. Under the MINED national office are 14 Departmental Directors responsible for implementing MINED policy within those departments. ATPs in turn work within a department under the supervision of the Departmental Directors. 33 There have been, however, exceptions to the above rule. Stakeholders, for example, cited a number of examples in which Departmental Directors failed to communicate information to them about an event, or failed to communicate it on time (e.g., day before the event, the day of event, or after the event took place). In other cases, Departmental Directors have refused to grant teachers time off to attend ECY training workshops, or have asked ECY to shorten training workshops because (presumably) they distract teachers from their main teaching duties. The aforementioned communication failures result in part from administrative rigidities introduced by the policy requiring IPs to work through Department Directors as intermediary and that also specify that only FEDISAL can communicate directly with MINED. While these administrative rigidities are not symptomatic of systemic communications failures within ECY, they do invariably introduce inefficiencies in the communications process that are manifested at times in failed or late communications to school directors and other stakeholders, whether due to lapses by MINED Departmental Directors or ECY IPs. MINED had initially proposed to play a larger role in ECY by training school directors in APA. However, they did not follow through on this plan, and ECY has subsequently taken on this role, resulting in a significant delay in school directors being trained in APA. This stands as one of the primary gaps in ECY implementation. “Some departmental directors are not conscientious and believe that teachers are wasting time in the trainings” –FEDISAL Technical Specialist. “For Cohort 1, the inclusion workshop was six days. For Cohort 2, the Ministry said six days was too much time for the teachers to be away, so it was cut to three days, which limited it a lot.” –FUNPRES Technical Specialist. 3.2.2 Conclusions EQ2: MINED Departmental Directors have played a critical role in ECY implementation, particularly with regards to their role in convening training workshops, meetings, and other events. Had the Ministry not played its convening role well, it is doubtful whether ECY would have been able to mobilize such a large number of schools (numbering into the hundreds) and directors, teachers, and students (numbering into the thousands) to participate in the Activity, particularly given existing administrative rigidities that require IPs to work through MINED Departmental Directors as intermediaries to plan and implement Activity events with school. The same administrative rigidities, however, have introduced inefficiencies in the communications process that have, at times, produced lapses in communication that have adversely affected Activity implementation. This, however, has been the exception rather than the rule. The MINED national office’s engagement with ECY has been minimal up to the point of the evaluation fieldwork. ECY has not influenced the Ministry’s vision or planning, and the Ministry has not influenced ECY implementation, resulting in an almost complete loss of oversight and accountability from the national Ministry to Departmental Directors and from Departmental Directors to ATPs. The lack of oversight and accountability from the national Ministry has left Departmental Directors and ATPs largely on their own, and according to their individual level of internal motivation, in deciding whether and how to engage with ECY, resulting in a wide variation in the levels and quality of engagement. 3.2.3 Recommendations ● ECY and MINED should negotiate more efficient arrangements for communicating information about ECY activities to Departmental Directors, ATPs and school directors. For example, MINED might agree to relax some communications restrictions and/or allow IPs to set up something akin to an alert system to remind ATPs, school directors and other stakeholders about upcoming events. ● MINED, in collaboration with USAID and ECY, should identify and assign an ECY counterpart within the Ministry with access to key Ministry decision-makers and who is responsible for working hand-in￾hand with ECY to plan and coordinate relevant details related to Activity strategy and implementation. 34 ● MINED should establish clear guidelines with clear lines of accountability for Departmental Directors and ATPs to engage with ECY. ● ECY should agree on a strategy with MINED to develop and implement a capacity building strategy for all levels of Activity implementation, particularly MINED national officials, Departmental Directors, ATPs, and school directors. 3.3 EVALUATION QUESTION 3 3.3.1 Findings ECY has taken a number of direct and indirect approaches to promote gender and social inclusion Direct Approaches Within the sample schools, the ET found a number of cases in which schools had implemented policies aimed explicitly at improving inclusivity as a result of their ECY inclusivity training (see text box). Indirect Approaches Other ECY interventions indirectly address inclusion in different ways. Some stakeholders believe that active teaching methods have helped facilitate the inclusion of students with physical and learning disabilities. According to the director of one of the deaf schools visited, for example, PAP has helped the school address the unique emotional needs of its deaf students that inhibit their participation in school activities. Sports, extracurricular activities, directed recreation, and coexistence activities have provided multiple avenues for girls and disabled students to more actively participate in school life and in leadership opportunities. The process of working with students, teachers and directors to resolve conflicts more creatively has further led some schools to place greater emphasis on zero tolerance for sexual abuse. Stakeholders strongly agree that ECY interventions include girls and disabled students As seen in Table 8, the stakeholders rated ECY favorably in terms of: 1) their inclusion of girls and disabled students and 2) the effectiveness of efforts to include girls and disabled students. Notably, there were no significant differences in the scores when disaggregating responses by sex. Table 8. Average Mini-Survey Scores on Activity Inclusivity Interventions Included Girls and Disabled Inclusion of Girls and Disabled was Effective 8.9 9.0 Schools have taken a gender or disability blind approach to inclusivity The ET found no evidence in any of the schools that girls or disabled students were being excluded from participation in school learning or extra-curricular activities. However, neither did the ET find evidence that the schools had made any attempt to identify and address specific barriers to inclusion related to sex or disability status. Instead, the schools have uniformly adopted a “gender blind” or “disability blind” approach, which does not include explicit strategies to promote inclusion or reflect an understanding of existing participation sex-based or disability-based barriers to inclusion. This finding suggests a gap in the Examples of School Inclusion Policies At Republica de Nicaragua, all students are allowed to participate in the school marching band, regardless of their musical ability. At Teresa Menendez, teachers visit the homes of pregnant girls to invite them back to school and assign tutors to students with physical or learning disabilities. At San Juan Buena Vista, students of both sexes can participate in all school recreational activities, but must also participate equally in all school chores (e.g., cleaning). Anita Alvarado holds classes for its students on how to treat their disabled peers, including one student in a wheelchair and two autistic students. Concepción de María emphasizes inclusion of its students suffering from cancer and is actively seeking to integrate gang members into school activities. Aldea de la Mercedes and Prudencia Ayala visit the homes of students absent from school to find out what is happening and invite them back to school. 35 ECY inclusivity training as reflected in this quote from the Gender Inclusion Report in Annex 13, “The activities that involved gender were limited to promoting equal participation. However, no analysis was made of gender gaps in participation, why they occurred, and whether gender equality was the best approach.” Challenges to inclusivity include lack of guidance from MINED and lack of capacity to help students with severe physical or learning disabilities A challenge in promoting inclusivity at schools is the lack of guidance from MINED. Under the FTIS framework, MINED has established the norm that schools are to be inclusive; however, it has provided little guidance on what inclusivity means or how it is to be implemented, leaving it to the discretion of Departmental Directors, ATPs, and schools (directors and teachers). A further challenge in promoting inclusivity is the lack of training and resources at schools to help children with special severe learning or psychological disabilities. According to one teacher, “We need help for children with learning disabilities to help them do better in school; we don’t have the tools to help them.” At the deaf schools visited, directors and teachers noted that ECY interventions and training curriculum failed to take into account the unique learning styles and emotional needs of deaf students. This issue arises also in the Objective 2 flexible modality settings where, according to one AIS Technical Specialist, “We do not have staff trained for this type of problem [students with learning disabilities].” Flexible modality courses do not address barriers to female participation and are disproportionately male Key informants note that 70-80 percent of “ni-ni” youth (neither in school or working) are female. Notwithstanding, female representation in Objective 2 flexible modality courses is disproportionately small at 49.6 percent. Barriers to female participation in the flexible modality courses include the lack of available childcare at meeting sites (remember that pregnancy is a primary cause of school dropout for females) and lack of reliable transportation (females face serious safety risks traveling long distances to attend flexible modality courses, especially if walking is their only option). Some ECY-supported OSY groups provide assistance with childcare and/or with transportation, but most do not.16 ECY provides insufficient evidence to support its inclusivity claims Finally, an assessment of ECY’s gender policy found that, despite numerous claims in ECY documents related to various gender-related activities, ECY has consistently failed to provide evidence supporting these claims, let alone the outcomes of such activities. Overall, ECY’s results with respect to gender and social inclusivity are opaque as evidenced by the lack of relevant information in the Activity’s performance monitoring system. 3.3.2 Conclusions EQ3: ECY interventions are open to all students, regardless of sex or disability, and there is no evidence of overt discrimination. Several schools, moreover, have adopted inclusivity policies as a result of their work with ECY. At the same time, however, ECY training does not help equip schools or OSY learning centers to adequately deal with students with severe physical or learning disabilities, including the unique learning issues faced by deaf students. Nor has ECY adequately supported its claims with regards to its inclusivity achievements. ECY’s flexible modality courses implicitly disadvantage female OSY by not addressing key factors that contribute to their lack of participation. The lack of childcare at OSY meeting sites and lack of safe and reliable transportation to the sites are important examples of practices that disproportionately disadvantage females and ultimately discourage their participation. Sub-EQ3.1: ECY has made a proactive effort, both directly and indirectly, to address issues related to gender and social inclusion. These efforts, however, do not include instruction to schools on the factors contributing to gender and disability exclusion and thus how to address them. As a result, schools address 16 The ET observed a number of cases of poorly supervised children loitering around the meeting sites. 36 gender and disability inclusion uniformly via so-called “gender blind” or “disability blind” policies that open school activities to all students regardless of sex or disability status. Such policies, however, may overlook structural constraints to female and disabled students’ participation, suggesting that inclusivity claims made by schools to the ET are likely overstated. 3.3.3 Recommendations ● ECY should investigate barriers to female participation in flexible modality courses and implement policies to address them, including, for example, assistance with transportation and childcare. ● ECY should include instruction about specific barriers to female and disabled participation in its inclusivity activities with specific guidance on how to address them at the school level. To develop the content for this instruction, it is further recommended that USAID and/or ECY conduct or sponsor research to identify barriers to inclusion within schools that goes far deeper than the ET was able to go during the mid-term evaluation. ● ECY should design interventions that explicitly take into account and focus on addressing the underlying causes of exclusion. ● ECY should adapt its training curricula and technical assistance to address the unique learning and emotional needs of students with severe physical and/or learning disabilities, including for deaf students, and/or equip schools better to meet those needs by linking them with the appropriate external resources. 3.4 EVALUATION QUESTION 4 3.4.1 Findings ECY has implemented four sustainability strategies with limited success ECY has sought to implement the following four sustainability strategies. First, ECY designed its interventions to be consistent with the MINED FTIS educational framework, support MINED’s educational policy, and integrate into Ministry planning. Second, ECY seeks to coordinate with MINED at the departmental level so that Departmental Directors and ATPs support its interventions and assume responsibility for them over the long-term. Third, ECY has sought to coordinate with school directors so that they support its interventions and assume responsibility for them over the long-term. Fourth, ECY encourages the replication and scaling-up of active teaching methods within and across schools via the creation of pedagogical teacher networks. The abolition of the ECY counterpart at MINED has made the first sustainability strategy difficult to implement. The departure of the ECY counterpart served to decouple the Activity from the national Ministry, leaving the Ministry rudderless in terms of coordinating with ECY at the national, departmental, and school levels. This, in turn, made it very difficult for ECY to implement the second strategy, given the absence of clear oversight or accountability mechanisms. To date, ECY has not been integrated into Ministry planning, and MINED Departmental Directors and ATPs have not assumed responsibility for ECY interventions over the long-term. ECY has had better success with the third strategy – school directors are aware of ECY, speak knowledgeably about its interventions, and uniformly express at least some level of support for the Activity. This, however, belies significant variation in the directors’ actual level of support. Some directors are more committed to ECY interventions than others, and this is reflected in the number and types of interventions implemented and their varying levels of success. Indeed, active and committed leadership by school directors emerged as perhaps the most critical factor for ECY success at the school level. The fourth strategy has met with mixed success. Teachers at a number of schools have replicated APA training to other lower secondary school teachers and, in some cases, to high school teachers as well. 37 Teachers in several of the schools, moreover, are participating in teacher pedagogical networks that appear to be functioning well and where mutual strengthening in active teaching methods is occurring. ECY is aware of the above deficiencies in its sustainability strategy and is in the process of implementing adaptations to the implementation strategy in order to address them. A particular focus of these adaptations is strengthening ECY’s relationship with MINED at both national and departmental levels so as to prepare them to assume responsibility for Activity interventions over the long-term. (See Annex 14 for more information about this and other implementation adaptations being planned or underway at ECY.) “In its final phase, the project needs to be integrated with FTIS at the central level in national planning, both in terms of sustainability and pedagogical aspects and at the local level in terms of integrated systems. The project also needs to focus on strengthening the departments’ institutional capacities for technical assistance, as well as at the central level. For the future, this is key.” –MINED National Official “One scenario project management didn’t consider was the transfer of knowledge and sustainability to the Ministry; this was never implemented in the project. It is important to do this so that it will continue to operate once the project goes. We want to have all the bricks in place when the project ends, so we are now planning to work together with the Ministry. The purpose is to transfer knowledge so the government is empowered about every single detail of the project.” –FEDISAL Senior Manager Stakeholders generally perceive ECY interventions to be sustainable; however, several factors threaten the sustainability of APA In the mini-surveys, stakeholders strongly agreed that Activity interventions are sustainable over the long￾term with an average score of 8.8 on the 10-point Likert Scale. At the same time, however, the ET visited a number of schools that have not implemented or replicated active teaching methods and/or where teachers are either not participating in teacher networks or are participating in poorly functioning teacher networks. A member of ECY’s implementing team estimates that only 30-40 percent of schools have replicated active teaching methods. Several factors help explain this outcome. ● Lack of administrative support by school directors. In at least one school, teachers who had received APA training had returned to their traditional methods due to a lack of support and supervision. ● Lack of support by MINED Departmental Directors and ATPs. ● Lack of follow-up technical assistance (TA) to directors, teachers, and teacher pedagogical networks. The lack of follow-up TA leaves directors and teachers on their own to figure out how to implement APA and, in the case of teacher pedagogical networks, how to make the networks function and deliver benefits to their members. ● Lack of formal accountability mechanisms rewarding or punishing schools or teachers for implementing or replicating APA. Teachers may or may not choose to train their colleagues in APA or to participate in a teacher network, with few personal repercussions in either case. ● Lack of incentives to apply or replicate APA. APA requires a significant investment of time and effort by teachers, for which they receive no special recognition. Some teachers have the internal motivation to invest the time and effort, while others do not. Thus, application of APA within a school is often inconsistent, even among teachers teaching the same subject matter and/or within the same grade. Where the implementation and replication of APA occurred, it was the result of the personal commitment by directors and teachers and (at times) ATPs. “This expansion is due to the enthusiasm around the APA guide, between the teachers themselves and the ATPs. The quality of the guides is the fuel for their expansion, but it is also the result of committed and conscientious teachers.” –School Director. ● Cost to reproduce APA guides. In the APA methodology, teachers create their own guides, which the students copy as a study aid. Guides can be several pages long, which at US$.02-.03 per page per class, add up to a relatively significant sum for students with limited financial means. Ostensibly for this reason, one school has opted not to implement APA. Teachers deal with this obstacle 38 different ways, such as copying all guides by hand, creating guides for groups of students so they can share the cost, or paying for the guides from their own pockets. ● Limiting the number of teachers trained within a school in only three subjects (math, language, and natural sciences). This limits the number of students within a school who benefit from APA. A related problem occurs when one or more of the few teachers trained in APA leave the school taking the institutional knowledge of with them or when a teacher trained in APA decides, for whatever reason, to quit using it. “Even though three have received APA training . . . there was neither collaboration among us to prepare APA guides, nor motivation from the principal to replicate the method with other grades.” –Teacher. “Teachers need more supervision to determine whether they have prepared and implemented the guides adequately.” –Teacher “MINED is no longer encouraging us to participate in learning circles.” –Teacher 3.4.2 Conclusions EQ4: ECY’s sustainability is threatened by a number of factors. Chief among them is its failure to date to integrate Activity interventions with MINED vision and planning at either the national or departmental levels. Other factors threatening sustainability include the lack of follow-up TA given to schools and teacher pedagogical networks, lack of oversight and accountability, and thus incentives, for implementing ECY interventions either at MINED or at schools, and (at times) lack of leadership by school directors. Committed, active leadership by school directors is absolutely critical to ECY’s success. As one of ECY’s key interventions, APA’s sustainability is also threatened by the same factors, plus some others that adversely affect its: 1) implementation within assisted schools, 2) replication within assisted schools, and 3) scaling-up across schools within integrated school networks. EQ4.1: ECY has initially put in place four mutually-supporting sustainability strategies, but these strategies have met with very limited to mixed success. Moreover, the failure of its strategy to integrate interventions into MINED vision and planning has had ripple effects on ECY’s other sustainability strategies. ECY’s plan to strengthen its relationship with MINED national and departmental officials, together with its adaptations to strengthen its follow-up TA, improve the Activity’s prospects for sustainability. 3.4.3 Recommendations ● ECY and MINED should jointly develop a formal, articulated structure for conducting integrated planning so as to ensure that Activity interventions are integrated into MINED priorities and policies at the national and departmental levels and vice versa. ● ECY should identify a counterpart within MINED specifically tasked to work with the Activity and who has access to key decision-makers within the Ministry. ● MINED should establish a set of clear guidelines with clear lines of accountability for Departmental Directors and ATPs to engage with ECY. ● Related to the previous recommendation, MINED should establish a set of positive or negative incentives for Departmental Directors, ATPs, school directors and teachers to engage actively with ECY. As the topic of incentive compensation is a complex one, particularly in a context of scarce budgets, identifying appropriate incentives for the El Salvador lower secondary school context requires additional research. However, absent some form positive or negative incentives for key stakeholders to engage with ECY and implement its interventions, the sustainability, replication, and scale-up of those interventions is doubtful. ● ECY (if possible with support from MINED) should provide schools with financial or other resources (e.g., electronic or hard copy templates) to create APA guides for students, at least in the short-term, while developing a long-term strategy to transition this responsibility to other education stakeholders. For example, implementing the recommendation for EQ1 to help link schools to external resources 39 could include funding for APA guides as an objective of this activity. The overall cost of creating APA guides is small in relative terms but with a potentially disproportionate return in terms of facilitating the implementation of active teaching methods within schools. ● ECY should develop and implements a sustainability strategy that addresses all identified threats to sustainability and prioritize the strategy over the remaining life of the Activity. ● ECY should direct follow-up TA to help school directors, teachers, and ATPs operate and manage the pedagogical teacher networks. This TA should focus in particular on how to manage learning networks and how to deliver value-added to network participants via such things as learning exchanges, learning products, timely and useful communication, and so forth. To function well, learning networks require a non-trivial time commitment by their members for which they are not compensated. Ultimately, member participation depends on the value-added they perceive relative to the costs of participation. 3.5 EVALUATION QUESTION 5 3.5.1 Findings Stakeholders are highly satisfied with the capacity of ECY IPs and the relevance, quality, and usefulness of ECY Interventions ECY’s IPs are widely recognized experts in youth education who have significant credibility among stakeholders. Both Component 1 and Component 2 stakeholders universally praised the experience and technical competency of IP technical staff and the quality and usefulness of the training or instruction they received. ECY has also been successful under Objective 2 in forging links with municipality officials, including the mayor’s office and municipal police forces. This has helped ECY to provide a safe space where OSY can meet, recruit OSY into the courses, and provide supervisory and other support during course delivery. This qualitative finding is corroborated by the mini-survey results, shown in Table 9 below. Overall, stakeholders strongly agree that: 1) interventions were well implemented and according to plan; 2) trainings were useful and well-delivered; 3) IPs have good technical capacity; 4) interventions responded to school needs and were appropriate; 5) students and parents participated in ECY interventions; 6) schools have a coexistence committee and a coexistence plan; 7) schools are promoting the replication of APA; and 8) interventions are consistent with MINED’s strategy. Table 9. Average Mini-Survey Scores on the Implementation of Activity Interventions Interventions Well Implemented Interventions Implemented According to Plan IPs Have Capacity to Implement the Interventions Training Useful and Well Delivered Interventions Respond to School Needs Interventions Appropriate 8.9 9.0 9.1 9.3 8.9 9.2 Students Participate in Interventions Parents Participate in Interventions Coexistence Committee Functions Well School Has Coexistence Plan School Promoting Replication of APA Interventions Consistent with MINED Strategy 9.1 8.6 8.8 9.0 8.9 9.1 Challenges integrating seven disparate IPs into a coherent institutional entity constituted significant constraints/bottlenecks to ECY implementation ECY was conceived as a taskforce consisting of a mixture of local and global institutions. The involvement of FHI 360 during the early years of the Activity was an important institutional arrangement that helped foster integration between the Activity IPs by bringing expertise in USAID project management initially absent among the local IPs and helped contribute to their growth in administrative and technical areas to 40 the point where they could function on their own. The diverse skills base coming from each local IP, moreover, allowed the Activity to offer a broader and more adaptable range of school interventions. These benefits, however, did not occur without some growing pains. Integrating seven IPs, each with a different organizational culture and technical expertise, proved to be a challenge that has adversely affected implementation efficiency and effectiveness. To begin with, integrating the seven IPs required significant time and effort to forge a common vision and to coordinate their work. In addition, some of the IPs lacked experience working with USAID, which led to conflicts about how money could be spent, among other things. Working through these and other related issues slowed down implementation early on in the Activity. These factors created tensions among the IPs as they had to figure out how to make the arrangement work. Moreover, ECY’s initial administrative structure – in which FHI360 was Chief of Party but only had technical responsibility for the Activity, while FEDISAL had administrative authority – introduced further administrative difficulties and contributed to contentious relationships among IPs that were only resolved once FHI360 left the Activity. The difficulty integrating ECY IPs not only existed at the institutional level but also at the school level, where ECY’s interventions are implemented within silos by IPs. The ET could not find evidence that IPs, or FEDISAL as the executing agency, attempted to coordinate the school-level interventions in an efficient manner. From the school’s perspective, the interventions appear ad hoc and disconnected. This in turn inhibits better resource planning (e.g., more effective allocation of teachers to trainings to match their skills and interests) and discourages integrated intervention planning (e.g., when to implement and how to sequence activities and/or match them to existing demand and resources) within schools. While the coexistence committee could serve as the coordinating body at the school to plan and integrate the ECY interventions, this rarely happens in practice. The current ECY administrative structure further lacks a central mechanism for quality assurance (QA), which is exacerbated by the fact that each organization has its own set of quality standards and its own set of QA processes. This decentralized structure has contributed to a disjointed and ineffective QA strategy, particularly with regards to providing follow-up TA to schools so as to assure that interventions are being properly implemented within the schools and are continued over the long-term. (See later in this section for more on issues related to providing follow-up TA.). The loss of ECY’s counterpart at the MINED national office significantly affected ECY implementation and results The implications of this external factor for ECY implementation and results are discussed at length in Sections 3.2.1 and 3.4.1. ECY implementation adhered closely to planned implementation The ET found no significant gaps between planned and actual implementation. Where the two did diverge, it was more as a result of ECY implementing adaptations to improve outcomes than it was due to a failure to adhere to existing work plans. In fact, ECY has demonstrated an admirable willingness and capacity to identify problems, learn from them, and make adaptations to its implementation strategy. ECY is closely aligned with the principles and practices found in MINED’s FTIS strategy ECY’s design and implementation is closely aligned with the following principles and practices included in MINED’s FTIS strategy: 1) active teaching methods are used in place of traditional lecture and copy methods; 2) learning process allows for timely decisions and adjustments so that students learn beyond what is needed for simple qualification; 3) adaptations are embodied in the curriculum to meet the needs of students with physical, sensory, or cognitive disabilities; 4) institutional organization guidelines are applied within the conceptual framework of capacity building throughout the entire student process; and 5) spaces are provided for students to allow them the enjoyment of expression, sports, reading, and other activities 41 that generate a positive link with free time and develop emotional intelligence.17 With that said, the ET found a number of issues with the implementation of ECY interventions that adversely affect their on-the￾ground alignment with the principles and practices of FTIS, in addition to the interventions’ efficiency, effectiveness and long-term sustainability. These are discussed below. ECY exhibits a number of implementation problems that adversely affect Activity results Weak Follow-Up TA The following quote by a teacher captures the most persistent finding across all stakeholders groups: “Follow up has been poor; there has been no follow-up monitoring or assistance.” The lack of follow-up TA has important implications for ECY’s effectiveness and sustainability. For example, due to the absence of further reinforcement or motivation, directors or teachers may decide not to implement an intervention and/or they may become discouraged and revert back to old practices. In the case of APA, the lack of follow-up TA has left many teachers feeling adrift in trying to apply or replicate APA, particularly among teachers that had received training in active teaching methods from UDB, which was shorter and less detailed than the APA training provided by FHI360. The UDB training was designed as a primer in active teaching methods under the assumption that those teachers who had received APA training from FHI360 would in turn train the other teachers in APA. This did not always occur, however, leaving UDB-trained teachers at a relative disadvantage.18 The lack of follow-up TA also extended to the teacher pedagogical networks. Teacher networks need technical assistance not only to bolster capacity in implementing active teaching methods, but also to help the teachers manage a learning network (e.g., coordination, communication, resource sharing, etc.) so that it functions well and delivers valuable results to its members. Concerns about the lack of follow-up TA was not limited to APA but extended to all ECY interventions; in no case did stakeholders generally feel that the level of follow-up TA was adequate. Factors explaining the lack of follow-up TA are described below: ● ECY strategy: ECY’s strategy does not appear to have contemplated the need to conduct intensive follow-up TA. Rather, for Cohorts 1-3, implementation focused almost wholly on training, with an apparent objective of maximizing training throughput. ● Budgeting and staffing: ECY did not budget for follow-up TA. IPs were allotted relatively few full￾time staff and often relied on contracted technical specialists. In 2013, for example, FUNPRES had four people working on ECY. In 2014, one person resigned and was not replaced, due to budget constraints, and in 2015 another person left the team and again was not replaced, for the same reason, leaving only two people on the FUNPRES technical team responsible for all 667 Cohort 1- 3 schools. Next, in July 2016, yet another person left, and in the following August, the sole remaining team member went on maternity leave. Later in the year, FUNPRES hired a new technical specialist, responsible for doing follow-up with PESS-supported schools, and a new technical coordinator, bringing the FUNPRES team back up to three people responsible for the 667 Cohort 1-3 schools. “The problem with the project is that the funds did not reach very far. For example, for active methodologies there was a maximum of two or three people per school. In the case of art and culture, there were four students. In art and culture there is only one position. In vocational guidance, only one person receives 17 MINED, USAID E. S. (2013). Elementos para el desarrollo de modelo pedagógico del sistema educativo nacional. El Salvador: MINED & USAID. 18 UDB provided training in active teaching methods more generally, but not specifically in APA. This outcome stems from a disagreement with FHI360, which provided the initial APA training for Cohorts 1-3, about the proper approach to active teaching methods. UDB is no longer involved in providing training on active teaching methods, while FUNPRES has taken over FHI360’s role in teaching APA and providing follow-up technical assistance. 42 everything. This is strategically correct [given the budget], but if handled poorly, it is counterproductive, . . . it only works, however, when it provides technical assistance and follow-up.” –FEDISAL Management. ● Weak coordination by Activity IPs: In theory, FUSALMO is responsible for following up with all schools participating in Objective 1 interventions. Assigning this role to FUSALMO made sense, as it already had a presence in ECY municipalities and schools and a large field staff. In practice, FUSALMO has struggled to play this role effectively, for a few reasons. 1. FUSALMO technical specialists are assigned an average of 25-30 schools spread out over multiple municipalities. As a result, its technical specialists are only able to visit each school about 2-3 times per year and often for only a few hours per visit. 2. FUSALMO technical specialists focus primarily on providing TA for FUSALMO-supported interventions, principally the coexistence committee. Thus, much of their time at the schools is spent working on coexistence issues, leaving them relatively little time to devote to other ECY interventions. 3. According to some FUSALMO technical specialists, the FUSALMO management team has created an environment in which technical specialists feel encumbered to adapt follow-up TA to the needs of the individual schools. Instead, the primary emphasis of follow-up TA is to achieve external reporting targets. 4. FUSALMO technical specialists lack expertise in other ECY intervention areas. On top of this, the TA they do provide may run counter to that preferred by the relevant IPs. 5. When FUSALMO technical specialists find issues at the schools they cannot address, there is no formal system for communicating this information to the relevant IPs. Lack of Resource Sharing by Schools For the robotics and art and culture interventions, schools belonging to an “integrated school system” (group of geographically proximate schools) are expected to share resources with each other. Robotics, for example, provides a kit to a host school and, to use the kit, students in other schools within the integrated system may have to cross gang boundaries, which subjects them to increased risk and demotivates them to participate. Once a school has a kit, moreover, it may be reluctant to share it with the other schools. “The vision for technical assistance at FUSALMO is very limited. The needs of the schools do not correspond to a standard agenda.” –FEDISAL Technical Specialist “I do not feel comfortable that FUSALMO understands all of the different components well enough to give the type of follow-up necessary.” –FEDISAL Technical Specialist “There are differences in the coordination provided by FUSALMO and FUNPRES. They have different ways of working that can cause problems. Each one does its own form of technical assistance.” –FUNPRES Technical Specialist. Limited Coverage of Teachers in Trainings To date, participation in the ECY training workshops has been limited to a small number of teachers and students from each school. This practice can have adverse effects on the implementation process for the following reasons: ● Certain teachers participating in the trainings inevitably leave the school, taking with them the relevant institutional knowledge and potentially leaving the school bereft of the internal capacity to continue the intervention. ● Teachers who participate in the trainings do not always share their knowledge with their colleagues, and/or their colleagues are not always interested in learning these skills, thus the relevant institutional knowledge remains limited within the school, a problem that is exacerbated if the teacher rotates from grade to grade or leaves the school. 43 ● Teachers who participate in trainings often rotate from one grade to the next, thus creating a lack of continuity in the implementation process within schools,19 particularly with regards to APA and vocational orientation. Selection of “Wrong” People to Participate in Trainings To ensure that the “right” people are sent to trainings, ECY shares a profile with school directors describing the characteristics participants should possess. Beyond this, however, ECY exerts little control over which teachers the directors choose to send. In some cases, directors send teachers who fit the profile; in other cases, they do not. According to FUNPRES, participants only fit the profile about 65 percent of the time - “The rest have no idea why they go to the workshops. It is difficult to control this.” With regards to active teaching methods, directors may send teachers to the training that lack expertise in the relevant subject matter. This occurred frequently enough that UDB designed its training approach to include two days of basic instruction in the relevant subject matter before moving on to instruction in active teaching methods. Excessive Number of Interventions at the School Level Objective 1 includes 21 interventions. Trying to implement such a large number of interventions places a significant burden on schools, and particularly on teachers, many of whom only work half-time at the school and have limited time to prepare APA guides, teach class, supervise student clubs, participate in coexistence committees, resolve disputes, provide psychological counseling, advise students on schooling and career decisions, monitor sports activities, and so forth. On top of this, teachers are expected to replicate and scale￾up the interventions both within their school and across other schools via teacher networks. Several stakeholders questioned whether this was a reasonable expectation. “The project is very ambitious. It is providing a large range of activities to the schools. With so many trainings, there is a lot of demand on teachers and schools in terms of time. The major challenge with such a large number of activities is that it sometimes distracts the teachers from being able to focus on anything.” –School Director Limited Involvement of Key Stakeholders in Activity Interventions For Cohorts 1-3, training workshops focused on teachers and students and largely excluded directors and parents. Directors thus often only know of ECY interventions second-hand from their teachers. As a result, directors are less likely to be fully invested in the interventions’ success, or to integrate the interventions into the school’s coexistence plan and annual school plan, and parents are less likely to support the school’s or their children’s participation in the interventions. Given the centrality of school directors in ensuring the Activity’s success at the school level, this is an important omission. Short Duration of Training Key informants within schools and among IP staff noted that the duration of the training workshops was too short in some cases to expect participants to have assimilated the information to the point where they could implement the interventions on their own, particularly absent follow-up TA. Low Test Pass Rates by OSY Taking the Sufficiency Test Modality This finding is addressed at length in Section 3.1.1. ECY is aware of its implementation issues and is implementing adaptations to address them Members of the ECY senior management team expressed a high level of awareness of the implementation issues described above and have already begun to formulate and implement adaptations to address them. These adaptations are described in Annex 14. 19 It is common, for example, for teachers to teach a cohort of students throughout the entire third cycle, beginning with them in seventh grade and then moving with them to eighth grade and then ninth grade, after which they rotate back to seventh grade to begin working with a new student cohort. 44 3.5.2 Conclusions EQ5: IPs are experts in their assigned fields, IP staff are highly qualified, and the quality of implementation by IPs has been consistently high. Stakeholders are, on average, very highly satisfied with IP service delivery. The primary implementation challenge ECY has encountered was the time and effort required to integrate its seven IPs into a coherent and coordinated institutional entity. This in turn adversely affected the efficiency, effectiveness, and sustainability of implementation and results, particularly in terms of delays in implementation, a lack of coordinating vision by IPs reflected in uncoordinated interventions at assisted schools, and a (to date) disjointed and ineffective QA process, including the lack of follow-up TA to assisted schools. While local expertise is important, it is also important to build capacity and foster integration among local organizations. International organizations/experts can be useful in this role, particularly if local IPs lack experience working within the USAID context. Similarly, working with multiple local IPs offers benefits in terms allowing the Activity to expand its experience/skill set and range of possible interventions. Both of these institutional arrangements, however, come with trade-offs that need to be acknowledged and planned for ahead of time. EQ5.1: ECY implementation adhered closely to its planned course; variations from planned implementation occurred mostly as result of adaptions made by ECY for the purpose of improving key outcomes. EQ5.2: Despite the expertise of IPs and IP staff and the overall high quality of implementation, there exist a number of serious deficiencies with ECY implementation that adversely affect its efficiency, effectiveness, and sustainability, including: 1) weak follow-up TA, 2) lack of resource sharing by schools, 3) limited coverage of teachers in trainings, 4) selection of the “wrong” people to participate in trainings, 5) excessive number of interventions at assisted schools, 6) limited involvement by key stakeholders in Activity interventions, 7) short duration of training, and 8) low sufficiency test pass rates. A number of these deficiencies, moreover, affect different aspects of Activity implementation discussed above. For example, weak follow-up TA, limited coverage of teachers and other key stakeholders, and short training duration adversely affect the degree and quality of implementation within schools, the horizontal sharing of knowledge and practice within and across schools, and intervention sustainability. At the same time, the large number of interventions within schools strains available resources making it more difficult to achieve extensive coverage for other, perhaps priority, interventions and arguably contributes to the diminishing marginal quality of teachers being selected to participate in training activities. The assumption that teachers trained in a particular intervention would in turn pass their knowledge on to their colleagues, whether within their school or to other schools via teacher networks, absent follow-up TA and a set of clear incentives to do so, has proven true in some cases but has not proven true in a sufficient number of cases to justify basing intervention strategy on this assumption. This is seen particularly with regards to APA where teachers not receiving APA training, or who received training from UDB in active teaching methodologies more generally, either were not implementing APA or were struggling to translate a general understanding of active teaching methods into the specific practice of APA methodologies. This highlights the need to: 1) provide follow-up TA to ensure that the desired within and across school horizontal knowledge/skills transfer is taking place and becomes embedded within the schools; 2) extend the same level of training to all training participants; and/or 3) include a larger cohort of teachers within a school in the relevant training course. ECY demonstrates both a substantial capacity for recognizing and learning from its weaknesses and devising adaptations to address them. ECY is in the process of implementing a number of significant revisions to improve its Component 1 and Component 2 interventions, to strengthen its working relationship with MINED, and to strengthen its process for providing follow-up TA to assisted schools. 45 EQ5.3: ECY designed Activity interventions to be consistent with and support MINED’s FTIS strategy. Where deviations occur, they are primarily the result of on-the-ground implementation issues occurring, as opposed to strategic or design misalignments. 3.5.3 Recommendations ● ECY should create a single team responsible for conducting QA and follow-up TA and bring it under the FEDISAL umbrella. This would free up IPs to focus on training, create clearer lines of accountability, and promote greater coordination and consistency in QA and follow-up TA. ● ECY should tailor the number and types of interventions it implements in a given school consistent with each school’s interests, resources, and capacity. These interventions should include, at a minimum, APA, sports activities and coexistence committees. Limiting school-level interventions to a smaller set of “strategic interventions” should further place less pressure on school directors to select teachers with diminishing marginal quality to participate in trainings. ● If feasible, ECY should include all lower secondary school teachers in APA training workshops while also providing all participating teachers the same level of training. If not feasible, it should include as many as possible given resource, time and other constraints. This would presumably ensure greater consistency in applying APA within schools. ● ECY should include school directors in APA training workshops and other training workshops to the extent possible. ● ECY should develop a formal set of processes by which IPs can communicate with other IPs when they find issues in the field relevant to the interventions supported by the other IPs. ● ECY should include ATPs and parents more often in training workshops and on-the-ground activities. ● ECY should extend training duration where relevant and feasible. One way to do this is to stagger training sessions and provide follow-up TA in-between training sessions. This approach would also help ensure that stakeholders are not left on their own to figure things out, reduces the need for additional follow-up TA, and provides stakeholders the opportunity to share their experiences with peers via cross-learning opportunities. ● ECY should exercise greater discretion over who participates in training activities. This might include, for example, requiring schools to provide information about teachers selected for trainings ahead of time or administering screening tools to teachers at the commencement of training and then empowering IP to reject unqualified/unmotivated teachers from the training. ● USAID and ECY should budget for and conduct rolling assessments using either Activity staff or external researchers so as to: 1) determine the relative effectiveness of different interventions, including training, follow-up TA, and school-level implementation; 2) identify key drivers of and barriers to intervention effectiveness; and 3) recommend mid-course Activity adaptations. Such assessments could take a number of forms depending on the purpose, resources, and timeframe, for example formal probability-based surveys, mini-surveys using purposive sampling methods, rapid assessments, or lot quality assurance surveys. Once (if) ECY turns Activity interventions over to MINED, the interventions should be as fine-tuned as possible given resource constraints and the time remaining in the Activity. ● Related to the previous recommendation, ECY should conduct formal assessments of each major adaptation of pilot test it implements so as to determine its relative effectiveness. Again, these assessments can take any number of methodological approaches depending on the situation. ● For schools that are unable to participate in robotics or art and culture interventions due to security concerns, ECY should involve them in a different integrated sharing network, provide resources directly to them, exempt them from participation, or offer them alternative methods to receive similar benefits. ● For future Activities involving a team of international and local IPs and/or a team of local IPs, the Activity design should include explicit strategies to facilitate the hand-off to local IPs and to integrate systems across IPs to the extent feasible so as to avoid a “silo effect.” 46 3.6 EVALUATION QUESTION 6 3.6.1 Findings ECY’s two primary reporting frameworks (MEP and process indicators) are well designed and credible, and the current M&E staff is capable and effective The ECY M&E system has two reporting frameworks for collecting and reporting information about Activity performance: MEP and “process indicators.” The MEP consists of 35 performance indicators that ECY reports to USAID/El Salvador on a quarterly or yearly basis. Process indicators consist of eight internal monitoring indicators collected annually by field staff that measure pedagogical processes and coexistence at ECY-assisted schools. The purposes for collecting the process indicators are: 1) to inform management decision-making and plan school assistance; 2) to identify factors that contribute to successful interventions; 3) to measure the extent to which results are aligned with the minimum MINED FTIS performance criteria; and 4) to serve as the primary database for ECY’s new signal system. (See Annex 14 for more on ECY’s new signal system). The ECY M&E staff is clearly capable and, with limited resources, generates a relatively substantial volume of information about Activity performance and related topics. This information includes two paper series launched during 2017, one a monthly briefing paper series and the other a bi-annual policy paper series posted on the ECY website. The monthly brief paper summarizes information collected via ECY’s M&E activities and is disseminated via email to Activity IPs. Its purpose is to highlight performance or other information of interest to the ECY implementing team. Policy papers are more formal research papers addressing issues related to ECY and targeted to internal and external audiences. A review of briefing and policy papers finds them well written and informative. However, the M&E department concedes that neither the briefing papers nor policy papers are widely read among IP staff, stating, “IP staff do not tend to read what is sent to them. In order to get them to pay attention you have to do a presentation.” IP staff generally perceives that the ECY M&E system generates useful information that is disseminated and acted on As seen in Table 10, IP staff strongly agreed that the ECY M&E system generates useful information, that the ECY has in place sufficient processes to disseminate M&E information, and that ECY uses M&E information to make informed decisions. Table 10. Average Mini-Survey Scores by IP Staff on the ECY M&E System M&E System Generates Useful Information ECY Has Internal Processes to Disseminate M&E Information ECY Uses M&E Information 8.6 8.6 8.8 ECY lacks a formal knowledge management system to disseminate performance information to IP staff, MINED, or schools or used to inform Activity strategy or implementation Despite the IPs favorable perceptions of the ECY M&E system found in Table 12, the ET found no formal knowledge management processes within ECY by which it routinely shares either explicit or tacit information on Activity performance with IP staff, MINED national office, Departmental Directors, ATPs, or schools.20 The majority of stakeholders in all stakeholder groups interviewed by the ET said that they had not received information about Activity performance, nor had they used M&E information for planning or decision-making. IP technical staff further noted that there is no formal process for sharing experiences and lessons learned across IPs, despite the potential learning that such cross-learning activities would generate. The M&E Department concedes, “There is not a culture of knowledge management so far at the 20 Explicit information is easy to communicate, store, and distribute and is the knowledge found in reports, numbers, and other visual and oral means. Tacit information is information held by IP field staff grounded in their emotions, experiences, insights, intuition, and observations. 47 project. This finding stands in contrast to the finding under EQ5 that ECY management has shown an openness and capacity to identify and learn from Activity weaknesses. ECY is working to integrate more knowledge management features in its M&E system At the beginning of calendar year 2016, FEDISAL hired a new M&E Specialist who is working toward transitioning the M&E function from a “focus on responding to information requests and distributing reports” to function that places a greater focus on knowledge management and using M&E information as a management tool to inform on-the-ground strategy, particularly related to providing follow-up TA. One element of this transition is the adoption of the signal system described above and at greater length in Annex 14. Another element will be increased emphasis on field visits by members of the ECY management and M&E teams. Beginning in 2017, the M&E team intends to participate in trainings and conduct school visits, while the ECY management team intends to visit ATPs, join them for school visits, and share information with them. Still missing from the Activity’s M&E transition plan, however, are processes for collecting tacit information from its IP technical staff and for the sharing of experiences or lessons learned across Activity IPs for the purpose of guiding Activity planning and decision-making. 3.6.2 Conclusions EQ6: The performance monitoring indicators being collected and reported by ECY are, for the most part, both relevant and useful. There are no notable omissions from either. Overall, M&E staff is highly qualified, the ECY M&E function is well managed, and the M&E team generates a large volume of credible information about Activity performance and other topics. Sub-EQ6.1: The performance monitoring indicators being collected and reported by ECY are, for the most part, both relevant and useful. There are no notable omissions from either. Sub-EQ6.2: While the ECY M&E system generates a large volume of information about Activity performance and other topics, this information is generally not accessed or used by IPs or other stakeholders for any purpose. This results both from an absence of formal KM processes within ECY for the disseminating of performance information to IP staff, national or departmental MINED officials or schools and a corresponding lack of a knowledge management or learning culture within the organization. In contrast with this conclusion, ECY senior management is well informed about Activity weaknesses and results, and it has demonstrated both a capacity and willingness to identify and address Activity weaknesses. Thus, while a learning culture does not exist globally across ECY IPs, there is evidence that a learning culture (or at least a commitment to learning and adaptation) exists at the upper levels of ECY management. A challenge for ECY moving forward is how to generalize what is happening at senior management levels and transform that into an organization-wide set of knowledge management processes that serve to create a similar commitment to learning and adaption across the Activity IPs. The change in management of the ECY M&E function has brought with it a commitment to introduce more KM features into function. While current plans to do so are encouraging, they still lack formal design features to collect both explicit and tacit information and disseminate this information, along with lessons learned, among IP staff and other key stakeholders for the purpose of Activity planning and decision￾making. Sub-EQ6.3: IP staff and other key stakeholders (outside of senior ECY managers) are not well informed about Activity performance, nor do they generally use information about Activity performance for planning or decision-making. 3.6.3 Recommendations ● ECY should develop and implement KM processes aimed at creating a learning culture across Activity IPs. This KM strategy should include processes for collecting, disseminating, and using explicit and tacit information across IPs for the purpose of informing planning and decision-making. 48 ● For schools falling in the red category in ECY’s new signal system, ECY should distinguish between schools that either have not had time to implement the Activity interventions and schools that are struggling to implement them, but are committed to trying, from those schools that lack the motivation or commitment to implement the interventions, and thus where additional follow-up TA is unlikely to yield much improvement. ● In planning any follow-on Activity, USAID should require that that the Activity designs and implements a formal KM system that establishes clear processes for generating, disseminating, and using both explicit and tacit performance along with clear lines of responsibility and accountability for management and staff members of the IP team. This system should complement but also go beyond the set of performance indicators found in the MEP and reported routinely to USAID, while the Activity should further be encouraged to utilize a wide range of qualitative and quantitative data collection methods focused more on generating information useful for internal learning and action and less on external accountability to USAID or other stakeholders. ANNEX 1: ECY MID-TERM PERFORMANCE EVALUATION STATEMENT OF WORK STATEMENT OF WORK (SOW) Mid-Term Performance Evaluation of the USAID's Education for Children and Youth (ECY) activity A. Evaluation Purpose The purpose of this mid-term performance evaluation of USAID/El Salvador’s Education for Children and Youth activity (herein “ECY”), covering the period January 2013 through December 2016, is: ● To determine whether the ECY activity has been implemented as planned, any deviations from the implementation plan, and identify any adjustments needed for final phase of implementation in order to achieve objectives; and ● To highlight lessons learned to inform future activity designs or follow-ons. The audience of the evaluation report will be USAID/El Salvador, specifically the Office of Democracy and Governance; El Salvador’s Ministry of Education (MOE); and the Implementing Partner. USAID and MOE will use the findings and recommendations of the evaluation to make necessary changes in the implementation of the activity. USAID/El Salvador will also use the evaluation to guide the design and implementation of future basic education related activities under the new Country Development Cooperation Strategy (CDCS). This performance evaluation is focused only on objectives 1 and 2 of the activity because they are more aligned with USAID’s Education Strategy and the existing CDCS. ● Objective 1. Sustaining improved educational outcomes for lower secondary school students ● Objective 2. Increase access to educational opportunities for out-of-school youth This performance evaluation will not consider objectives 3 and 4: ● Objective 3: Procurement and distribution of educational supplies to 32 schools damaged by Tropical Storm Ida. ● Objective 4: Rapid Response Fund. 49 B. Background Identifying Information The ECY activity seeks to reduce educational gaps in El Salvador through increasing and improving educational opportunities to vulnerable and/or disadvantaged lower secondary students and out-of-school, at-risk youth ages 12-24 years living in selected municipalities with high crime rates. For the purpose of this activity, at-risk youth are those who: 1) attend school and live in poor communities or neighborhoods with high rates of crime and gang activity, 2) have learning disabilities, 3) have adult responsibilities at a young age (including teen mothers), or 4) are out of school and unemployed with little or no education or are employed and need workforce readiness skills. The ECY activity supports and coordinates programs with the MOE in responding to the needs to improve the quality of education through the practice of inclusive, pertinent, and safe learning environments. The ECY activity will help create safer communities by preparing youth for productive alternatives to criminal gang activity and addressing issues of inequitable access that may be a consequence of violence among youth (due to income, geographic location, exclusion of certain groups, among others), including environments affected by lawlessness, violence, and crime and gang activity. The ECY activity supports out of school youth by providing complementary learning activities designed to reintegrate youth back into a formal school setting or provide an equivalent primary or secondary school diploma or skills training linked with the local labor market. USAID’s ECY activity supports basic education to improve student transition to the critical, yet overlooked, area of lower secondary education, teacher development, and expanding the Full Time Inclusive School (FTIS) approach (“Escuela de Inclusiva de Tiempo Pleno”), and broader violence prevention in public schools. The development hypothesis of the ECY activity is that increased equitable access to education for children and youth living in areas of high violence and crime can be accomplished through establishing safe learning environments and improved quality education for students at the critical lower secondary school level (grades 7 to 9), and by assisting community-based organizations and local authorities to provide out of school youth with vocational skills, training for skills needed by the local job market or the opportunity to return to formal education and job placement services. Increasing equitable access to education for girls and boys, young women, and young men in target municipalities contributes to the joint GOES–U.S. Government efforts to create safer communities in El Salvador by preparing youth for productive alternatives to criminal gang activity. While the ultimate goal of the activity is to improve educational outcomes, the reduction of conflict and violence, and increased economic growth can be two positive externalities associated with keeping youth in school and getting out-of-school youth back into general education or vocational training. Enhancing educational opportunities for Salvadoran children and youth is critical at this moment of gang-related crisis and conflict and goes hand-in-hand with economic growth activities.1 ----------------------------------------------------------------------------- 1 The Evaluation Team will have access to the Cooperative Agreement, which includes the activity logical framework in the annex, before the preparation of the Evaluation Plan. The ECY activity contributes to Development Objective (DO) 1 of the CDCS 2013 2017, Citizen Security and Rule of Law in Targeted Areas Improved, and Intermediate Result (IR)1.2, Crime and Violence in Targeted Municipalities Reduced. Furthermore, ECY complements USAID’s Citizen Safety for El Salvador: Crime and Violence Prevention (CVPP) activity as well as other USAID citizen 50 security-related activities implemented under the Central American Regional Security Initiative (CARSI). The ECY activity contributes to the Prosperity pillar of the U.S. Strategy for Engagement in Central America (CEN strategy) and the line of action on human capital under the Alliance for Prosperity Plan in the Northern Triangle (A4P). It also supports the Quinquennial Development Plan of the Government of El Salvador (GOES) 2014-2019 and the GOES’ Plan El Salvador Seguro aimed to reduce crime in the 50 most violent municipalities of El Salvador: The list of 107 schools prioritized under the ECY activity in the 10 municipalities of the Phase I of Plan El Salvador Seguro can be found in the Annex I. In light of the current high levels of extreme violence and citizen insecurity in El Salvador, largely due to the gang crisis, USAID basic education funding (including ECY activity) focuses on Goal 3 of the Agency‘s Education Strategy 2011-2015 “Increased equitable access to education in crisis and conflict environments.”2 As part of the Local Solutions component of the USAID Forward Strategy, ECY is implemented under a cooperative agreement by the Foundation for Integral Salvadoran Education (FEDISAL, by its acronym in Spanish) , which works closely with its partner network comprised of five organizations: Salvador del Mundo Foundation, Salesian Association, Don Bosco University, Pro-Education Foundation of El Salvador, and an international partner. During the first three years of the activity, Family Health International 360 (FHI 360) supported implementation and management of ECY in the context of strengthening institutional capacities of USAID local partners to implement this activity. 51 C. Evaluation Questions The following evaluation questions, in priority order, have been identified by USAID/El Salvador. They should be answered by the Evaluation Team and clearly presented in the final report in terms of how they relate to the evaluation purpose. In addition, for each of the questions below, the Evaluation Team should incorporate, to the extent feasible, analysis of possible differences associated with gender or social groups (particularly historically excluded groups), and they should be reported separately for men and women. 1. On the basis on stakeholder perceptions (e.g., parents, students, teachers, school staff, and community members) what aspects of the activity’s focus on children and youth in the 7- 9th grade age group in FTISs most and least contribute to prevention of early gang infiltration? 2. How do stakeholders (e.g., parents, students, teachers, school staff, and community members) perceive the activity’s success in addressing critical drivers3 4 and protective factors5 of crime prevention? 3. What internal organizational capacities of the implementing partner have contributed to success or constraints/bottlenecks related to activity outcomes? Note that the definition of activity-related success is improved educational outcomes through: a) keeping youth in school and b) getting out-of-school youth back into general education or vocational training. 3.1 In what ways has MOE involvement helped or hindered ECY implementation? Why? 4. To what extent have specific interventions contributed to establishing safe learning environments in assisted schools? 5. What are the internal and external challenges, resolved and unresolved, related to the development (e.g., baseline) and utilization of the monitoring system to track performance of the activity components for the activity’s main expected outcomes: student retention, achievement, school safety, and participation of out of school youth in educational opportunities? ----------------------------------------------------------------------------- 2 http://pdf.usaid.gov/pdf_docs/Pdacq946.pdf 3 According to Olate, Salas-Wright, & Vaughn (2012), “poverty, social exclusion, practices of the civil war, access to illicit guns, drug trafficking routes (cocaine from South America), organized crime, weak institutions, corruption, and deported gang members from the US are often factors cited as the main causes of the level of violence and crime. The first step in violence prevention is to identify the particular individual, social, and environmental factors that place youth at risk for taking part in violent activity (Farrington, 2000; Olate et al., 2010). Risk factors for violence have been defined by a variety of scholars (Hawkins et al., 1992; Loeber and Farrington, 1998), but the general consensus is that a risk factor is any individual, social, or environmental factor that increases the likelihood of a negative outcome, in this case of violent behavior. Researchers generally categorize risk factors for violence and delinquency into one of five particular domains of risk: individual, family, school, peer, and community (Esbensen et al., 2009; Hawkins et al., 1998; Herrenkohl et al., 2001)” 4 Olate, R., Salas-Wright, C., & Vaughn, M. G. (2012). Predictors of violence and delinquency among high risk youth and youth gang members in San Salvador, El Salvador. International Social Work, 55(3), 383-401. 5 “Empathy, anger management, perspective-taking, creative generation of alternatives, considering consequences, active listening, assertiveness, and critical questioning of beliefs” Chaux, E. (2008). Classrooms in Peace: Pedagogical Strategies. Inter-American Journal of Education for Democracy, 1(2), 123-145. Also see Chaux, E. (2009). Citizenship competencies in the midst of a violent political conflict: The Colombian. 6. How do stakeholders view the relative value of different methods (e.g., active, parental engagement, leadership, coexistence committees) in terms of their contributions to helping students complete the ninth grade? 52 D. Evaluation Method(s), Data Collection, and Analysis For the ECY performance evaluation, a non-experimental mixed-methods design that combines a comprehensive, rigorous analysis of existing quantitative data with customized qualitative techniques designed to elicit primary data from a wide range of counterparts, partners, beneficiaries, and stakeholders is recommended. This approach allows for triangulation of complementary data to elucidate linkages between activity inputs, outputs, and outcomes. The Evaluation Team should consider a range of possible methods and approaches for collecting and analyzing the data to fully answer each of the evaluation questions thoroughly. The use of participatory methods will enhance collaboration and dialogue among counterparts is required. Further, data collection and analysis methods should be sensitive to possible differences related to sex and/or social status, and should ensure safety and confidentiality of all individuals providing data or information for the purposes of the evaluation. In addition, these methods, to the extent feasible, should be capable of identifying both positive and negative unintended consequences for women or girls. The finalized evaluation method(s) and approaches, data collection plan and analysis will be included in the Evaluation Plan submitted to USAID for revision and approval before field visits and data collection begin (see Deliverables section below). The method(s)proposed should comply with the USAID Evaluation Policy (http://www.usaid.gov/evaluation/policy). The data collection plan for this evaluation will include, at a minimum: a desk review of relevant documents such as cooperative agreements and work plans; review of activity performance monitoring data and macro data tracked by both, the Implementing Partner and the Ministry of Education of El Salvador; key informant interviews and/or focus groups discussions promoting equal participation of women and men; and direct observations through site visits. USAID expects both qualitative and quantitative data to be collected; and the results will be coded, triangulated, and analyzed for content. The Evaluation Team is encouraged to propose additional/alternate data collection and analysis methods in the Evaluation Plan that they consider can yield stimulating, robust evidence in answering each of the evaluation’s questions and capturing USAID contribution to increase the access to educational opportunities for out-of-school youth in El Salvador. Data collection shall be systematic and data must comply with the five data quality standards of validity, integrity, precision, reliability, and timeliness.6 Specific interview, survey and/or focus groups protocols will be appended to the Evaluation Plan and finalized with approval from USAID; the questions should be used to answer each of the evaluation questions listed in this document and addresses the purpose of this evaluation. All data collected in response to the evaluation questions must have as much disaggregation as possible. At minimum, per USAID Gender Equality and Female Empowerment Policy,7 all data must be disaggregated and analyzed by sex, as well as analyzed for any difference between the effects on girls and boys and men and women or male and female participation.Where possible, data should be disaggregated by educational region or municipality. ----------------------------------------------------------------------------- 6 http://pdf.usaid.gov/pdf_docs/Pnadw118.pdf 7 USAID Gender Equality and Female Empowerment Policy (https://www.usaid.gov/sites/default/files/documents/1865/GenderEqualityPolicy_0.pdf) ● Desk review of relevant documents and deliverables USAID will provide the Evaluation Team with all relevant, available strategy and activity-specific documents for ECY, such as cooperative agreements with expected results, performance reports stating the results achieved, any prior assessments, if applicable, etc. The Evaluation Team must review these documents and other existing literature to be provided by USAID and others in preparation for the initial planning meeting and before meeting with local stakeholders for interviews. The Evaluation Team is expected to review these, make their own contextual literature research and review, and create a Review Matrix to be delivered to USAID as part of the final Evaluation Report, which indicates how key information extracted from reviewed documents and other methodologies used link to each evaluation 53 question. At minimum, the following documents relevant to the ECY mid-term performance evaluation shall be reviewed by the Evaluation Team: o Country Development Cooperation Strategy (CDCS 2013-2017); o U.S. Strategy for Engagement in Central America (CEN strategy); o Alliance for Prosperity Plan in the Northern Triangle (A4P); o Country Assistance Strategy FY 2013-FY 2018; o ECY cooperative agreement and its amendment(s); o Draft revised SOW for any agreement amendments (if the agreement amendment is incomplete); o Monitoring and Evaluation Plan for ECY; o ECY annual and quarterly progress reports; o ECY annual work plans; o USAID Annual Portfolio Reviews; o Key activity report deliverables as specified in the cooperative agreement; Reports not otherwise required regarding initiatives, participation, support, diagnostics, trainings; o Samples of deliverables such as diagnostic instruments and framework or instructional manuals/guides; o Relevant GOES Education and Security related strategic plans (Quinquennial) o Development Plan, Plan El Salvador Seguro, Plan El Salvador Educado, etc.); o Relevant FHI360 and Full-time Inclusive School reports. These documents will be provided to the Evaluation Team by USAID. Many of them are publicly available on USAID’s Development Experience Clearinghouse (DEC): http://dec.usaid.gov ● Key Informant Interviews, surveys, focus groups discussions The Evaluation Team will interview stakeholders and conduct thorough key informant interviews, group interviews, short surveys, and/or focus groups. The Evaluation Team will include both men and women in the stakeholders’ consultation process. USAID or its Implementing Partner will provide a general list of stakeholders and key informant contact information once the evaluation begins. At minimum, the Evaluation Team will interview USAID staff, including ECY’s AOR; personnel from FEDISAL and its partners; technical and executive staff from MOE; and direct and indirect beneficiaries of ECY. It is encouraged that the Evaluation Team conducts interviews with randomly selected samples of the Trainers of Teachers, Resource Teachers, Trainers of Principals, and Principals trained through ECY and relevant MOE personnel. A sampling plan describing the selection process (such as purposeful, random, or a combination of approaches) for organizations and stakeholders for key informant interviews, surveys, and focus groups discussions (including sex disaggregation) is expected to be included in the Evaluation Plan and Final Evaluation Report. ● Site visits, direct observation and case studies The Evaluation Team, with inputs from USAID and FEDISAL, will select relevant site visits based on a sampling plan developed for the Evaluation Plan and included in the Final Report. Geographical areas of Quezaltepeque, where schools have received support from the Municipal Government, and Sonsonate, where there were pilots to track key indicators related to dropout rates, offer the best sampling. Schools located near Salvador del Mundo Foundation could provide key insights regarding organization and after school programs. Finally, visits to Catholic schools could provide key insights on interventions in faith￾based environments. The Evaluation Team will conduct in-depth case studies of 1-2 “most improved” or “best performing”8 schools in the ECY, and 1-2 “low performing” ECY schools, to compare and contrast how ECY has been implemented in each and to review additional factors that may be impact their performance. Participant observation of randomly selected group of teachers will be conducted to determine whether and how they are using new training methodologies. The Evaluation Team may attend events hosted or organized by ECY during the fieldwork period of the evaluation to conduct direct observation. The 54 Evaluation Team can use these events to talk with stakeholders, conduct interviews and collect additional data as evidence to answer the evaluation questions. USAID and FEDISAL will provide the Evaluation Team with a list of events once the Evaluation begins. ● Review of performance indicators ECY has an activity-specific Monitoring and Evaluation Plan and has collected data on a number of standard and custom indicators during its implementation. This monitoring data collected for USAID Annual and Quarterly reports and the implementing partner will provide one source of data on progress toward objectives and outcomes. The Evaluation Team may use monitoring data on performance indicators as part of the evaluation analysis and should report on it in the Final Evaluation Report as much as it relates to the evaluation questions stated above and satisfies relevant data quality standards. In addition, MOE collects data at the national level related to education access and quality such as grade level scores (e.g. grades or scores), enrollment and dropout rates, etc. This data should be analyzed and included to the maximum extent possible when answering the evaluation questions. Team planning meetings An initial team-planning/kick-off meeting will be held in El Salvador between USAID and the Evaluation Team before the submission of the Evaluation Plan so that USAID can clarify any questions from the Evaluation Team, expectations, and guidelines. The expected results of this meeting are to: ● Clarify each team member’s role and responsibilities; ● Confirm the anticipated timeline and deliverables; ----------------------------------------------------------------------------- 8 In this context, “best performing” is understood to mean a program with the lowest dropout rates and highest retention rates that can most reasonably be attributed to activity interventions. ● Discuss data collection tools and methodologies by evaluation question to be presented in the Evaluation Plan; and ● Identify communications logistics and how the Evaluation Team, USAID, and the implementing partner will communicate with each other. Additional meetings may be held as deemed necessary by USAID and/or the Evaluation Team. E. Deliverables It is estimated that not more than 90 working days of services from the starting date of the evaluation will be needed to complete a high-quality performance evaluation as required under this Statement of Work. During that timeframe, the Evaluation Team shall submit the following deliverables: 1. An Evaluation Plan, in Word, font Times New Roman, size 12, to be completed by the Evaluation Team after the Team Planning Meetings, no later than 15 working days after the start of the evaluation. USAID will receive the Evaluation Plan via electronic mail and review it to provide comments no later than five working days after receiving the document. The Evaluation Plan will provide details of how the various deliverables, tasks, and activities will be undertaken. It must include at least: ● ECY activity description and logic (theory of change/development hypothesis); ● Evaluation design,9 and the justification of why one design or mix of designs is selected, its limitations, and how these limitations will be addressed; ● A Matrix10 summarizing the following information per each Evaluation question: ♦ Method(s) for data collection, sources, the justification of why one method or mix of methods is selected, its limitations and the ways to address them; ♦ Technique(s) for data analysis11, the justification of why one analysis or mix of analyses techniques is selected, its limitations and the ways to address them; ● Timeline and/or Milestone Plan, including tentative starting time and duration of each activity conducted under the evaluation; 55 ● Drafts of data collection protocols, such as questionnaires, focus group moderator guide(s), interview scripts, consent forms, etc.;12 ● Evaluation Team composition and roles; and ● Location for the evaluation and Site visit plan. ----------------------------------------------------------------------------- 9 Some examples of designs of performance evaluations are snapshot design, before-and-after design, time series designs, cross-sectorial design, case study design, panel design, etc. 10 Some examples of matrix formats are available in https://usaidlearninglab.org/library/evaluation-design-matrix￾templates 11 Some examples of data analysis techniques are parallel, conversion, sequential, multilevel, data synthesis, etc. 12 If underage persons (less than 18 years old) will participate in this evaluation, the Evaluation Team must make sure to comply with the Law for the Integral Protection of Children (LEPINA by its initials in Spanish): http://www.asamblea.gob.sv/eparlamento/indice-legislativo/buscador-de-documentos-legislativos/ley-de-proteccion￾integral-de-la-ninez-y-adolecencia If the Evaluation design includes key informant interviews, surveys, and/or focus group discussions, the Evaluation Plan should include the following information:13 ● How the interviews/surveys will help to answer the evaluation questions ● Who will conduct the interviews/surveys and why they are qualified to do so ● What the rationale and methods are for deciding the number, timing, and location of the interviews/surveys ● How the participants will be selected and recruited ● How the interviews/surveys will be recorded ● How the interview/survey data will be analyzed and presented The Evaluation Plan, particularly the data collection and analysis protocols, as well as interview and focus group guides must be approved by USAID prior to the start of data collection and the field work. All interview protocols must be submitted in English and Spanish. The Evaluation Team will have another ten working days to address USAID observations and make any changes. Once the Evaluation Plan is approved, the Evaluation Team will submit to USAID an electronic copy of it in PDF. Any subsequent change to the Evaluation Plan must be approved by USAID. The Evaluation Team shall provide USAID and the Implementing Partner with a preliminary briefing on the Evaluation Plan prior to the beginning of data collection. 2. Brief weekly bullet reports of activities, submitted the COR of this Evaluation by electronic mail due every Monday by the close of business. 3. A draft of the Final Report in Word, font Times New Roman size 12, submitted for review, due no later than 40 working days after the approval of the evaluation design plan via electronic mail. USAID will be responsible for distributing it to the implementing partner and other stakeholders, for comments. USAID will consolidate all comments and send the draft back to the Evaluation Team within 10 working days. At a minimum, and in accordance with the USAID Evaluation Policy and Automated Directive System (ADS) 201, the final report and its draft version must include the following sections: ● Executive Summary of the purpose, background, evaluation questions, method(s), findings, conclusions, and recommendations; ● Evaluation purpose and questions; ● Through description of the evaluation design and any challenges/limitations, with emphasis on the timelines and methods for data collection and data14 analysis; ● Relevant data analysis tables; ● Findings and conclusions drawn from the analysis of the findings; ● Action-oriented, practical and specific recommendations with defined responsibility for the action; 56 ● A dissemination plan of findings, conclusions and recommendations to intended users of the evaluation; and ----------------------------------------------------------------------------- 13 For additional information on Focus Group Interviews, see “USAID’s Technical Note: Focus Groups Interviews” (http://usaidlearninglab.org/sites/default/files/resource/files/Focus_Group_Interviews_Tech_Note_FINAL_2013_11 19.pdf ) 14 The Evaluation Team must identify: a) steps taken to mitigate any limitations, and b) how/whether the limitations affect any particular findings, conclusions, or recommendations. ● Appendices: a. Original SOW, annotated with any changes approved by USAID b. Evaluation and data collection team composition and roles, with conflict of interest disclosures for all real or perceived conflicts of interest, if necessary c. Data collection protocols and instruments including questionnaires and checklists d. Review matrix of documents consulted e. Meetings Notes f. Complete schedule of evaluation activities, meetings, and interviews g. List of individuals and organizations contacted and sites visited 4. A Draft Report Briefing for USAID and other stakeholders that USAID consider necessary on the contents, findings, conclusions, and recommendations included in the draft final report. According to the audience, the draft report briefing may be conducted in English or Spanish. Only the Team Leader needs to be present for this briefing; however, local/regional Evaluation Team members may also attend. The Draft Report Briefing will be used by the Evaluation Team as a feedback exercise to adjust the Final Report. 5. A Final Report in PDF, font Times New Roman size 12, no longer than 40 pages in its body, excluding the cover page; Table of Contents; List of Acronyms; tables, graphs, charts, or pictures taking during site visits; and/or any material deemed important and included as appendices. The approved Final Report must adhere to USAID’s Evaluation Policy and Automated Directives System (ADS)15 must be submitted in English and Spanish and have incorporated USAID's comments, as appropriate. The electronic version and five high-quality printed, bound, copies in English and Spanish of the Final Evaluation Report must be submitted to USAID within 10 working days after the Evaluation Team receives comments on the draft report. 6. A One-Page summary of the evaluation purpose, findings, conclusions and recommendations. The One￾Page summary will be prepared in English and Spanish in PDF. 7. Any raw data (qualitative or quantitative) collected in electronic form (DVD or flash drive, in original format of Word, Excel, etc.) is due no later than 90 working days after the starting date of the evaluation. As per Automated Directives System 540, the Evaluation Team must submit to the Development Data Library (DDL) at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any datasets that are used (or of sufficient quality) to produce an Intellectual Work. 8. A Final Presentation with PowerPoint slides to USAID (and potentially to any other stakeholders that USAID considers relevant) in English and Spanish as the Final Report is being finalized no later than 90 working days after the reception of this Statement of Work. ----------------------------------------------------------------------------- 15 In addition to the Evaluation Policy and most recent version of ADS (both available online at www.usaid.gov) please refer to the Evaluation Toolkit on USAID’s Learning Lab website, https://usaidlearninglab.org/evaluation specifically “Managing an Evaluation” and “From Draft to Final Report.” 57 Only the Team Leader needs to be present for the final presentation; however, local/regional Evaluation Team members may also attend. The Evaluation Team will upload the final presentation to the Development Experience Clearinghouse (DEC) and submit an electronic copy of the final presentation to the COR of this Task Order. 9. Other deliverables as identified during the Team Meeting and agreed to by USAID and the Evaluation Team. All reports and papers will be considered draft versions until they are approved by USAID. These draft documents must be labeled with the word “DRAFT” in watermark. Findings must be presented as analyzed facts, strong qualitative and quantitative evidence, and data, and not based on anecdotes, hearsay, or the compilation of people’s opinions. To ensure unbiased findings, there is no guarantee that findings will be modified based on USAID suggestions. The Evaluation Team will research, investigate, and corroborate as objective any suggestion before it is incorporated in the findings, and the change will be noted in the Draft document so as to have a record of the change. All submitted reports and presentations must be thoughtful, well-researched, and well-organized documents, and objectively answer the evaluation questions. When writing the report, the Evaluation Team must remember the different audiences. The style of writing should be easy to understand and concise, while making sure to address the evaluation questions and issues with accurate and data-driven findings, justifiable conclusions and practical recommendations. 16 The Evaluation Team should clearly disclose in the report any biases or limitations that exist during both data collection and analysis (selection bias, recall bias, unobservable differences between comparator groups, etc.). In addition, all real or perceived conflicts of interest must be disclosed by each member of the Evaluation Team in writing. When quoting an individual in any report, the Evaluation Team must always give the context or circumstances of the quote. Correcting a grammatical error in the quote may be valid, but not rewording an entire phrase. When translating quotes from one language to another, the Evaluation Team should do so in an idiomatic way and care must be taken to ensure that the tone of the translation is equivalent to the tone of the original. Quotes should be presented in their original language in report texts. All reports must comply with the USAID Graphic Standards Manual (http://pdf.usaid.gov/pdf_docs/PNADB334.pdf) and the ADS Style and Format Guide ( https://www.usaid.gov/sites/default/files/documents/1868/501mac.pdf).17 Once a Final Report has been approved by USAID, the Evaluation Team will make it compliant and submit it to the DEC.18 ----------------------------------------------------------------------------- 16 For additional information on the criteria to ensure the quality of Evaluation Reports, see the USAID’s How-To Note “Preparing Evaluation Reports” (https://www.usaid.gov/sites/default/files/documents/1870/How-to￾Note_Preparing-Evaluation-Reports.pdf), and Annex 1 of the USAID Evaluation Policy (https://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf) 17 Evaluation Team may use the Evaluation Report Template available in 18 Per Automated Directives System 540, documents and development assistance projects materials produced or funded by USAID must be submitted for inclusion in the DEC: https://dec.usaid.gov/dec/content/Create.aspx?ctID=ODVhZjk4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDB mY2Uy http://usaidlearninglab.org/library/evaluation-report-template The Evaluation Team will send by electronic mail to USAID the DEC link where the evaluation reports are available. USAID will assess the quality of all evaluation reports using the Evaluation Report and Review Template (http://usaidlearninglab.org/sites/default/files/resource/files/template￾evaluation_report_checklist_and_review_aug2015.pdf). USAID may attach a Statement of Differences as an Annex to any Final Report if any differences remain in the final version. 58 F. Evaluation Management Evaluation Team This performance evaluation will use a combination of multidisciplinary international, regional, and local experts. The Evaluation Team must include at a minimum: Team Leader (Key Personnel) Minimum qualifications Education: Master’s Degree in development or related development fields, such as Political Sciences, Public Administration, International Relations, Social Sciences, or other disciplines related to development assistance is required. Ph.D. is a plus. Formal training in monitoring and evaluation is preferred. Language Proficiency: Spanish Level III and English Level IV. Work Experience: At least 10 years of relevant prior experience conducting rigorous external evaluations using both quantitative and qualitative methods for development objectives and monitoring projects and programs, preferably in Latin America. Experience in Central America is a plus. At least eight years of project management experience. Experience with management of multidisciplinary teams is required. Some familiarity with USAID’s objectives, approaches, operations, and policies, particularly as they relate to evaluations is a plus. Position Description: The Evaluation Team Leader will be responsible for overseeing and coordinating all activities related to this performance evaluation and for ensuring the production and completion of quality deliverables in a professional manner, in conformance with this Statement of Work. Education Specialist (Key Personnel) Minimum qualifications Education: Master’s degree in education administration, planning, economics, of education or similar field. Ph.D. is preferred. Formal training in monitoring and evaluation is a plus. Language Proficiency: Spanish Level III and English Level IV. Work Experience: At least eight years of significant experience working with educational projects and youth development in developing countries, preferably in Central America. Some familiarity with USAID’s objectives, approaches, operations, and policies, particularly as they relate to evaluations is a plus. Position Description: The Education Specialist will provide technical support to the Evaluation Team regarding issues of access and quality for the lower secondary level (grades 7 to 9). Research Specialist (Key Personnel) Minimum qualifications Education: Master’s degree in Educational Research/Statistics, Social Sciences, or other related fields. Ph.D. is a plus. Formal training in monitoring and evaluation is preferred. Language Proficiency: Spanish Level III and English Level IV. Work Experience: At least eight years of significant experience working with educational projects and youth development in developing countries, preferably in Central America. At least five years of experience in evaluating educational programs, conducting social researches, and conducting cost-effectiveness analysis. Some familiarity with USAID’s objectives, approaches, operations, and policies, particularly as they relate to evaluations is a plus. Position Description: The Research Specialist will provide technical to the Evaluation Team on conducting statistical analyses, cost-effectiveness, and qualitative analysis based on in-depth interviews or focus groups 59 Logistics Coordinator (Non-Key Personnel) Minimum qualifications Education: High school or technical degree in administration or a related development field, such as Economics, Political Science, Public Administration, Business Administration, or other related discipline related to development assistance is required. Language Proficiency: Spanish Level IV and English Level II. Work Experience: At least five years of relevant work experience. Participation in or knowledge of evaluation of development projects is a plus. Position Description: The Logistics Coordinator will be responsible for coordinating with USAID, the implementing partner, and the Evaluation Team to schedule meetings, transportation, and other administrative logistics. S/he will be involved in planning and may help with data collection and analysis. The combined Evaluation Team must have expertise in performance evaluation methodologies, qualitative and quantitative data collection and analysis, primary level curriculum development, classroom instruction, and teacher training. Key personnel must have excellent written and oral presentation skills and the ability to conceptualize and write clearly and concisely. Understanding of the Latin American context is necessary, with a preference for personnel with work experience specifically in Central America, and particularly in El Salvador, or the Northern Triangle countries (Guatemala, Honduras, and El Salvador). At least one staff member shall have in-depth local knowledge of El Salvador’s educational system. All Team members will be required to provide to USAID a signed statement indicating any conflict of interest. The Team Leader must be someone external to USAID. Anyone who has worked directly with USAID/El Salvador or its implementing partner in the last five years must not be considered as part of the Evaluation Team. No staff members shall have been directly involved in the implementation of ECY. In addition, one or more team members should have experience in integrating gender in evaluation methods and knowledge of gender issues in the education sector. Logistics The Evaluator will be responsible for all logistics support under this Statement of Work, including field office administration, all travel arrangements (with required USAID clearances), team planning facilitation and appointment scheduling, coordination with all partners and stakeholders involved, administrative services (computer support, printing and copying), report editing and dissemination, and for complying with provisions set forth in this Statement of Work. USAID will provide limited support to the Evaluation Team. This support, if needed, may include assistance in arranging high level meetings; access to the U.S. Embassy compound as necessary; and access to all reports, data, and other relevant documents created by ECY.USAID will organize all internal meetings. 60 Annex 1: List of 107 schools prioritized in the 10 municipalities of the Phase I of Plan El Salvador Seguro. 61 62 63 ANNEX 2: ORIGINAL EVALUATION QUESTIONS AND SUB-QUESTIONS Evaluation Questions Evaluation Sub-Questions EQ1: On the basis of stakeholder perceptions (e.g., parents, students, teachers, school staff, and community members) what aspects of the activity’s focus on children and youth in the 7-9th grade age group in FTISs most and least contribute to prevention of early gang infiltration? EQ1.1: How successful has the activity been in addressing early gang infiltration among children and youth in the 7-9th grade group? EQ1.2: To what extent do activity results related to early gang infiltration differ between boys and girls? EQ1.3. How sustainable are activity results related to early gang infiltration? Which factors will determine how sustainable they are? EQ2: How do stakeholders (e.g., parents, students, teachers, school staff, and community members) perceive the activity’s success in addressing critical drivers and protective factors of crime prevention? EQ2.1: Which aspects of ECY’s activities most and least contribute to addressing critical drivers and protective factors of crime prevention? EQ 2.2: To what extent do activity results related to the critical drivers and protective factors of crime prevention differ between boys and girls? EQ2.3: How sustainable are activity results in terms of addressing the critical drivers and protective factors of crime prevention? Which factors will determine how sustainable they are? EQ3: What internal organizational capacities of the implementing partner have contributed to success or constraints/bottlenecks related to activity outcomes? EQ3.1: To what extent did actual activity implementation adhere to planned implementation? What factors internal and external to the activity affected implementation? EQ3.2: What was the quality of general activity implementation? What are its strengths and weaknesses? How could it be improved? EQ3.3: To what extent did activity strategy and implementation address issues related to gender and social inclusion? EQ3.4: In what ways has MINED involvement helped or hindered ECY implementation? EQ 3.5: To what extent are ECY activities and their implementation consist with its objectives and with the GOES’ policy priorities? EQ4: To what extent have specific interventions contributed to establishing safe learning environments in assisted schools? EQ4.1: How successful has the activity been in establishing safe learning environments in assisted schools? EQ4.2: To what extent do activity results related to creating safe school learning environments differ between boys and girls in the 7-9th grade group? EQ4.3: How sustainable are activity results related to establishing safe learning environments in assisted schools? Which factors will determine how sustainable it is? EQ5: What are the internal and external challenges, resolved and unresolved, related to the development (e.g., baseline) and utilization of the monitoring system to track performance of the activity components for the activity’s main expected outcomes: student retention, achievement, school safety and participation of OSY in educational opportunities? EQ5.1: How relevant/useful are the performance monitoring indicators being collected and reported (both internally and externally) by the activity? What, if anything, is missing? EQ5.2: What knowledge management processes have been established at USAID and within activity 64 Evaluation Questions Evaluation Sub-Questions implementing partner to generate internal and external learning and how well are they functioning? EQ5.3: To what extent are USAID and implementing partners using activity performance information to evaluate activity performance, generate internal and external learning, and make activity adaptations? EQ6: How do stakeholders view the relative value of different methods (e.g., active, parental engagement, leadership, coexistence committees) in terms of their contributions to helping students complete the ninth grade? EQ6.1: How successful has the activity been in helping children and youth complete ninth grade? EQ6.2: To what extent do activity results related to the completion of ninth grade differ between boys and girls? EQ6.3: How sustainable are activity results related to the completion of night grade? Which factors will determine how sustainable it is? ANNEX 3: EVALUATION MATRIX Evaluation Question & Sub￾Question Data Source Data Collection Method Data Analysis Methods EQ1: How do stakeholders (e.g., parents, students, teachers, school staff, community members, implementing partners, government officials, etc.) perceive the Activity’s success in addressing the following key outcomes: ● Early gang infiltration? ● Critical drivers and protective factors of crime prevention? ● Safe school learning environment? ● Student dropout/retention (students finishing 9th grade)? ● Student scholastic performance Out of school youth returning to school USAID IPs MINED-National ATPs Directors Teachers Students Parents Document review KIIs FGDs Mini-Surveys Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources EQ1.1: To what extent do stakeholders perceive that Activity interventions have been effective in achieving the following key outcomes: ● Early gang infiltration? ● Critical drivers and protective factors of crime prevention? ● Safe school learning environment? ● Student dropout/retention (students finishing 9th grade)? ● Student scholastic performance? Out of school youth returning to school? USAID IPs MINED-National ATPs Directors Teachers Students Parents Document review KIIs FGDs Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources 65 Evaluation Question & Sub￾Question Data Source Data Collection Method Data Analysis Methods EQ2: In what ways has MINED USAID Document review Identification of themes involvement helped or hindered IPs KIIs and sub-themes via Activity implementation? MINED-Central ATPs Directors Teachers Other programs FGDs content analysis Summary statistics Triangulation of findings from multiple data sources EQ3: What have been the Activity’s outcomes with regards to gender and social inclusion? USAID IPs Directors Teachers Students Parents Document review PI Review KIIs FGDs Mini-Surveys Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources EQ3.1: How have Activity interventions addressed issues related to gender and social inclusion? USAID IPs Directors Teachers Students Parents Document review KIIs FGDs Mini-Surveys Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources EQ4: How sustainable are the USAID Document review Identification of themes Activity’s outcomes? IPs MINED-Central ATPs Directors Teachers Students Parents Other programs KIIs FGDs Mini-Surveys and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources EQ4.1: How have Activity USAID Document Review Identification of themes interventions addressed issues IPs KIIs and sub-themes via related to sustainability? MINED-Central ATPs Directors Teachers Students Parents FGDs content analysis Summary statistics Triangulation of findings from multiple data sources EQ5: What internal USAID Document review Identification of themes organizational capacities of the IPs KIIs and sub-themes via implementing partner have MINED-Central FGDs content analysis contributed to success or ATPs Mini-Surveys Summary statistics constraints/bottlenecks related Directors Triangulation of findings to Activity outcomes? Teachers Students Parents from multiple data sources 66 Evaluation Question & Sub￾Question Data Source Data Collection Method Data Analysis Methods EQ5.1: To what extent did USAID Document review Identification of themes Activity implementation adhere IPs KIIs and sub-themes via to planned implementation? MINED-Central FGDs content analysis What factors internal and ATPs Mini-Surveys Summary statistics external to the activity have Directors Triangulation of findings affected implementation? Teachers Students Parents from multiple data source EQ5.2: What was the quality of USAID Document review Identification of themes general Activity IPs KIIs and sub-themes via implementation? What are its MINED-Central FGDs content analysis strengths and weaknesses? How ATPs Mini-Surveys Summary statistics could it be improved? Directors Teachers Students Parents Triangulation of findings from multiple data source EQ 5.3: EQ 5.3: To what extent USAID Document review Identification of themes are ECY interventions consist IPs KIIs and sub-themes via with MINED’s policy MINED-Central FGDs content analysis priorities? ATPs Mini-Surveys Summary statistics Triangulation of findings from multiple data source EQ6: What are the internal and external challenges, resolved and unresolved, related to the development (e.g., baseline) and utilization of the monitoring system to track performance of the activity components for the activity’s main expected outcomes: student retention, achievement, school safety, and participation of out of school youth in educational opportunities? USAID IPs MINED-District Directors Teachers Students Document Review KIIs FGDs Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources EQ6.1: How relevant/useful are USAID Document Review Identification of themes the performance monitoring IPs KIIs and sub-themes via indicators being collected and MINED-Central FGDs content analysis reported (both internally and ATPs Mini-Surveys Summary statistics externally) by the activity? Directors Triangulation of findings What, if anything, is missing? from multiple data sources 67 Evaluation Question & Sub￾Question Data Source Data Collection Method Data Analysis Methods EQ6.2: What knowledge USAID Document review Identification of themes management processes have IPs KIIs and sub-themes via been established at USAID and FGDs content analysis within activity implementing Mini-Surveys Summary statistics partners to generate internal and Triangulation of findings external learning, and how well from multiple data are they functioning? sources EQ6.3: To what extent are USAID and implementing partners using activity performance information to evaluate activity performance, generate internal and external learning, and make activity adaptations? USAID IPs Document review KIIs FGDs Mini-Surveys Identification of themes and sub-themes via content analysis Summary statistics Triangulation of findings from multiple data sources 68 69 ANNEX 4: BIBLIOGRAPHY Arévalo, J. F. (2015). Education for children and youth project: Quarterly report (fourth quarter-July 1st to September 30th, 2016) (Quarterly Report No. 519-A-13-0000-1). El Salvador: USAID. Arévalo, J. F. (2017). Education for children and youth project: Quarterly report (first quarter FY2017- October 1st to December 31st 2016) (Quarterly Report No. 519-A-13-00001). El Salvador: USAID. Arévalo, J. F. (2017). Proyecto educación para la niñez y juventud: Reporte trimestral (primer trimestre FY2017-del 01de octubre al 31 de diciembre 2016) (Reporte Trimestral No. 519-A-13-00001). El Salvador: USAID. Consejo Nacional de Educación: Gobierno de El Salvador. (2016). Plan El Salvador educado: Por el derecho a una educación de calidad (Plan El Salvador Educado). San Salvador, El Salvador: Ministerio de Educación (MINED). El Salvador, Guatemala, and Honduras. (2014). Plan of the alliance for prosperity in the northern triangle: A road map (Plan of Alliance Road Map.) Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: Aprendizaje social y emocional (Fascículo Aprendizaje Social Emocional). El Salvador: USAID, FEDISAL y sus asociados. Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: El rol del personal directivo en la orientación vocacional (Fascículo del rol del personal directivo en la orientación vocacional. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: Entorno social del estudiante (Fascículo Entorno Social. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: Fascículo de presentación programa de orientación vocacional (Fascículo de presentación programa de orientación vocacional. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: Plan de acción anual de orientación vocacional (Plan de Acción Anual. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educación para la niñez y juventud: Proyecto pedagógico de aula y orientación vocacional (Proyecto Pedagógico y Orientación. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educaciónal para la niñez y juventud: Autopercepción aprendo a conocerme (Fascículo Autopercepción. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2015). Proyecto educaciónal para la niñez y juventud: Oportunidades educativas y productivas (Fascículo Oportunidades Educativas y Productivas. El Salvador: USAID, FEDISAL y sus asociados.) Equipo FEDISAL. (2016). Proyecto educación para la niñez y juventud: Sistematización de la implementación de la metodología activa: Aprendo, practico y aplico en El Salvador (Sistematización de la implementación de la metodología activa. El Salvador: USAID, MINED.) FEDISAL, E. S. (2013). Annual portfolio review: October 1st, 2012 to September 30th, 2013 (Annual Portfolio Review. El Salvador: USAID.) FEDISAL, E. S. (2015). Annual portfolio review: October 1st, 2014 to September 30th, 2015 (Annual Portfolio Review. El Salvador: USAID. (ECYA) 70 FEDISAL, E. S. (2016). Annual portfolio review: October 1st, 2015 to September 30th, 2016 (Annual Portfolio Review. El Salvador: USAID.) FEDISAL, P. N. (2013). Proyecto educación para la niñez y juventud: Plan de trabajo anual 2013 (Plan de Trabajo Anual No. 519A1300001). El Salvador: USAID, MINED. FEDISAL, P. N. (2014). Plan de trabajo anual 2014 (Plan de Trabajo Anual No. 519-A-13-00001). El Salvador: USAID, MINED. FEDISAL, P. N. (2015). Education for children and youth project: Work plan 2015 (Work Plan No. 519- A-13-00001). El Salvador: USAID. FEDISAL, P. N. (2015). Proyecto educación para la niñez y juventud: Plan de trabajo anual 2015 (Plan de Trabajo Anual No. 519-A-13-00001). El Salvador: USAID, MINED. FEDISAL, P. N. (2016). ECYP: Annual work plan 2016: English final version-April (Annual Work Plan No. 519-A-13-00001). El Salvador: USAID, MINED. FEDISAL, P. N. (2016). Education for children and youth project: Annual work plan 2016 English final version-April (Annual Work Plan No. 19-A-13-00001). El Salvador: USAID, MINED. FEDISAL, P. N. (December 2016). Education for children and youth project: Annual work plan 2017 (Annual Work Plan No. 519-A-13-00001). El Salvador: USAID, MINED. FEDISAL, P. N. (Enmienda (Septiembre) 2016). Proyecto educación para la niñez y juventud: Plan de trabajo anual 2016 (Plan de Trabajo Annual. No. 519-A-13-00001). El Salvador: USAID, MINED. FEDISAL: And Partners. (2013). Education for children and youth project: Monitoring plan 2013-2018 5th draft (Monitoring Plan No. 519-A-13-00001). El Salvador: USAID. FEDISAL: And Partners. (2013). Proyecto educación para la niñez y juventud: Plan de monitoreo 2013- 2018 5to borrador (Plan de Monitoreo No. 519-A-13-00001). El Salvador: USAID. FEDISAL: And Partners. (2015). Education for children and youth project: Monitoring plan updated 2015-2018 (Monitoring Plan No. 519-A-13-00001). El Salvador: USAID. Flóres, Ana: Moore, Dr. Audrey: Field, Dr. Samuel: Kluve, Dr. Jochen. (2015). Evaluating the impact of closing a community managed schools programme in El Salvador (Grantee Final Report. El Salvador: International Initiative for Impact Education). Guerrero González, R. (2015). El salvador: Productivo, Educado y Seguro: Plan Quinquenal de Desarrollo (Plan Quinquenal). Santa Tecla, La Libertad: Imprenta Ricaldone. MINED, USAID E. S. (2013). Elementos para el desarrollo de modelo pedagógico del sistema educativo nacional. El Salvador: MINED & USAID. Mogollón, O. S., Marina. (2011). Active schools: Our conviction for improving the quality of education (Expository Report FHI360). Taddei, D. A. (2016). Modelo de escuela inclusiva de tiempo pleno en El Salvador: Documento de sistematización. El Salvador: MINED. U.S. Department of State: Diplomacy in Action. (2016). U.S. strategy for engagement in Central America. Retrieved Accessed, April 2017, 2017, from https://www.state.gov/p/wha/rt/strat/index.htm USAID, E.S. (2015). Annual Portfolio Review. El Salvador: USAID. USAID, E. S. (2012). Cooperative agreement: Education for children and youth project (FEDISAL Cooperative Agreement No. 519-A-13-000001). El Salvador: USAID. 71 USAID, E. S. (2013). Annual portfolio review: October 1st, 2015 to September 30th, 2016 (Annual Portfolio Review). El Salvador: USAID. USAID, E. S. (2016). Country development cooperation strategy: 2013-2017 (Cooperation Strategy USAID). USAID, E. S. (2017). Statement of work (SOW) mid-term performance evaluation of the USAID's education for children and youth (ECY) activity. USAID. (2011). Education, Opportunity through Learning: USAID Education Strategy. Washington, DC: USAID. USAID: Equip2. (2012). Strengthening basic education program: Equip2, El Salvador (Final Report FHI 360 No. 519-A-00-06-00009-00). El Salvador: USAID. ANNEX 5: EVALUATION METHODOLOGY Sampling Methodology The large number of schools assisted under Objectives 1 and 2 through 2016, combined with evaluation resource and time constraints, made a statistically representative sample of activity schools impractical. Thus, in lieu of a fully representative, random sample, the evaluation employed the following set of purposive selection criteria. 1. Activity interventions should be far enough along at the school that results have had a reasonable time to emerge and useful lessons can be learned. Under Objective 1, ECY implementation involved three phases, each one lasting approximately one year. Thus, to be eligible for selection, a school must have completed, or nearly completed, all three implementation phases. This criterion effectively limited the choice to the set of schools belonging to implementation Cohort 1 (2013- 2015) and Cohort 2 (2014-2016), and ruled out schools belonging to Cohort 3 (2015-2017) and Cohort 4 (2016-2018). Next, within Cohorts 1 and 2, FEDISAL identified 290 eligible schools in 15 eligible municipalities that it judged to be sufficiently advanced in the implementation process that useful lessons could be learned. 2. The sample of schools should include a mixture of schools judged to have a low, medium, and high risk for violence. 3. The sample of schools should include schools for the deaf that have received activity support. To date, ECY has assisted two schools providing education for deaf students. Inclusion of these schools in the sample was intended to allow the Evaluation Team to investigate the evaluation’s crosscutting theme of social inclusion. 4. The sample of schools should include schools that have received support from the GOES under PESS. 5. The sample of schools should be spread out equally among the 15 eligible municipalities so as to achieve reasonable geographic representation of assisted municipalities and schools. 6. Within the 15 municipalities visited during the fieldwork, schools visited were selected in consultation with FEDISAL so as to ensure that schools fulfilling other selection criteria (e.g., schools for the deaf and PESS schools) were included. The above sampling methodology produced the sample of schools shown in Table A5.1 Figure A5.1. 72 Table A5.1. Schools Visited during ECY Mid-Term Evaluation Fieldwork Dept.* Municipality School Name Cohort PESS Deaf Risk** Zone*** CU Cojutepeque Para Sordos De Cojutepeque 2 X X L U CU Cojutepeque Anita de Alvarado 2 X M U CU S. Pedro Perulapán San Pedro Perulapán 2 M U LL San Juan Opico Cantón San Nicolás La Encarnación 2 L U LL San Juan Opico Cantón El Ángel 1 M R LL Zaragoza José María Cáceres 1 H U SO Sonsonate Prudencia Ayala 1 H U SO Sonsonate Carlos S. Langenegger 1 X L U SO S. Antonio d. Monte República de Líbano 2 X M U SO S. Antonio d. Monte Victoria Teresa Méndez Viuda de Escalante 1 H R SS Nejapa Cantón Aldea las Mercedes 1 H U SS Nejapa José Matías Delgado 1 L U SS Panchimalco Prof. Bernardino Villamariona 2 M U SS Panchimalco Goldtree Liebes 2 H R LL Quezaltepeque José Dolores Larreynaga 2 X H U LL Quezaltepeque República de Nicaragua 2 X M U SS Rosario de Mora Oblatas Al Divino Amor 2 M U SV Apastepeque Caserío Santa Paula 2 L R SV San Sebastián Cantón el Porvenir Aguacayo 2 H R SV San Sebastián San Sebastián 2 H U SV San Vicente Concepción de María 2 H U SV San Vicente Hacienda San Juan Buena Vista Canto El Rebelde 2 H R * CU=Cuscatlán; LL=La Libertad; SO=Sonsonate; SS=San Salvador; SV=San Vicente. **L=Low-risk; M=Medium-risk; H=High-risk ***U=Urban; R=Rural 73 Figure A5.1: Schools Visited for the ECY Evaluation For OSY flexible modality groups, the ET employed a purposive sampling approach in consultation with FEDISAL focusing on those Objective 2 flexible modality groups that were meeting on either Saturday or Sunday during the period of the evaluation fieldwork with some representation of different flexible modality approaches, to the extent possible. This approach yielded the sample of OSY flexible modality groups shown in Table A5.2. 74 Table A5.2. OSY Groups Visited during ECY Mid-Term Evaluation Fieldwork Department Municipality Training Center IP Flexible Modality Grades Attended* La Libertad Comasagua Casa Comunal de Comasagua AIS Sufficiency Test 7, 8, 9 La Libertad Santa Tecla Casa Comunal Comunidad El Progreso, El Boquerón AIS Sufficiency Test 7, 8, 9, HS1, HS2 La Libertad Santa Tecla Casa de la Juventud, Cafetalón AIS Sufficiency Test 7, 8, 9, HS1, HS2 La Libertad San Matías Centro Escolar Natalia López AIS Sufficiency Test 7, 8, 9, HS1, HS2 La Libertad Zaragoza C.E. José María Cáceres AIS Semi Face-to-Face 7,8,9 Santa Ana Santa Ana Centro Escolar Colonia Santa Eleonor FUSALMO Night 7, 8, 9 Methodological Limitations The ECY evaluation design was subject to a set of methodological limitations. These limitations, and the approaches taken by the ET to address them, are described at length along with the evaluation sampling methodology. Non-representative school sample: Due to time and resource constraints, the evaluation covered only 22 of 409 total schools and 290 of eligible Cohort 1 and 2 schools supported by ECY through December 2016. The limited representation of the school sample thus potentially limits the evaluation’s generalizability across the Activity. With that said, certain themes emerged with a sufficiently high level of consistency across schools and stakeholder groups to give the ET confidence that they represent important, underlying issues with the Activity and assisted schools. Subjective perceptions of stakeholders: The ET was not able to verify all perceptions and anecdotes cited by interviewed stakeholders. To mitigate this limitation, the ET has been careful to report only those findings that recurred with relative frequency across multiple stakeholder interviews. Difficulty assessing sustainability: The evaluation was implemented while many examined interventions were ongoing, which made it difficult for the ET to determine whether reported/observed outcomes were likely to continue once ECY support ends. In light of this, the Findings and Conclusions Section below discusses at length the Activity’s prospects for long-term sustainability and factors that are likely to affect it, for better or worse. Inability to attribute outcomes to activity interventions: Due to the absence of a counterfactual, the evaluation methodology is not able to generate sufficient evidence to attribute observed results to ECY interventions in a statistically valid manner. This limitation was exacerbated by the inability of stakeholders to link outcomes to specific ECY interventions (see above) meaning that it is next to impossible for the data analysis to tease out the unique contribution of the different interventions to reported or observed outcomes. Potential for respondent or interviewer bias: The ET cannot discount the possibility of respondent bias in the qualitative and quantitative data. The desire to attract follow-up programming among certain types of respondents is one such source of bias. Another potential source of respondent bias stems from certain respondents’ desire to cast themselves, their school, or their organization in a positive light. The evaluation design sought to compensate for the above limitations through a variety of methods: 1. Triangulated evidence from different qualitative and quantitative data sources, which served to increase the credibility of findings that were validated via multiple data sources. 2. Focused on those schools that have completed, or are nearing completion, of all three Objective 1 implementation phases, which, owing to the longer passage of time, increased the likelihood that changes and evidence of sustainability would have emerged by the time of the fieldwork; 75 3. Conducted cross-team validation exercises during the fieldwork, which allowed the ET to validate findings while in the field so as to validate or invalidate different lines of inquiry.21 4. Conducted systemic data analysis using well-established analysis methods supported by advanced data analysis software, which added rigor to the data analysis process and increased the credibility of the subsequent findings. 21 During the evaluation fieldwork, the entire Evaluation Team met once each week to review and corroborate findings, make adjustments to the fieldwork plan and areas of focus, and plan the fieldwork over the following week. 76 ANNEX 6: LIST OF KEY INFORMANTS INTERVIEWED Name Position FEDISAL Hector Quiteño Executive Director Jorge Arévalo Chief of Party ECY Hilda Rosales Program Manager Jenniffer Morán Sub-Director, Objective 1 Rodrigo Cuadra Sub-Director, Objective 2 Jhoana Castenada Manager, Monitoring & Evaluation Johana Mejía Coordinator, Vocational Training Alexander Alvarenga Technical Assistant, Vocational Training Donaldo Ramírez Technical Assistant, Vocational Training Aída Herrera Technical Assistant, Vocational Training Merlín Velasco Technical Assistant, Vocational Training Arlette Escobar Technical Assistant Mayra Pérez Technical Assistant Alfonso Viquez Technical Assistant, Youth Specialist FUNPRES Delia de Avila Executive Director Cristina León Coordinator, Art & Culture Brenda Boyat Technical Assistant, RCC, PAP, Acosa Escolar AIS Father Salvador Cafarelli Executive Director Mario Argueta Coordinator, Objective 2 Ibell Araniva Assistant to Coordinator, Objective 2 Roberto Stanley Technical Assistant, Flexible Modalities Walter Castillo Technical Assistant, Flexible Modalities Fátima Ivania Technical Assistant, Flexible Modalities Tony Martínez Technical Assistant, Flexible Modalities Mario Argueta Operations Coordinator FUSALMO Nelson Cruz Executive Director Herbert Martínez Coordinator, Sports Monitoring Carlos Escalante Technical Advisor, Education Ivette Henríquez Technical Assistant Abraham Ramírez Technical Assistant AGAPE Oscar García Director, Center of Professional Training Marvin de León Coordinator, Special Projects 77 Name Position UDB Fabián Antonio Bruno Director, Department of Education Programs FHI360 Antonieta Harwood Former COP ECY Ana Florez Director for US Program MINED Central Renzo Valencia Manager, National Education Management María Estela Ávila Orozco Angélica Paniagua National Program Director for Youth and Adults Gloria Evelyn Hernández Manager of Educational Flexible Modalities ATPs Rosalba Renderos ATP, Quezaltepeque Ada Guadalupe Castro ATP, Panchimalco Mauricio Humberto Navarrete ATP, San Pedro Perulapán Teresa de Jesús González Bonilla ATP, Cojutepeque Pedro Castellón ATP, Rosario de Mora Nelson Orlando Guillén ATP, San Antonio del Monte Rafael Antonio Alfaro ATP, San Juan Opico Community Leaders José Ricardo Reyes Rosales Mayor, Comasagua Fanny Estefanía Díaz Romero Councilor, San Matías Irma Elizabeth López Technical Specialist, Children and Youth, San Matías Martha Dalmau Representative, Santa Tecla María Tomasa Saravia de Juárez Representative, Santa Tecla Objective 2 Teachers Yul Engelbert García Linares Natural Sciences, Casa de la Juventud, Cafetalón Clara Luz Gonzales Math, C.E. José María Cáceres Celina Castellanos Math, Casa Comunal Comunidad El Progreso, El Boquerón Other Development Programs Mauricio Esteben Cáceres Alvarenga Technical Specialist, Prevention of Juvenile Violence in Central America, GIZ 78 ANNEX 7: DISCUSSION GUIDES GUIDE FOR STUDENTS Municipality School Sex Male: Female: Grade 7th: 8th: 9th: INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. Now, let’s talk about how the project activities have affected you and your classmate in the following areas: 1. Which project activities did you like the most that have helped you to learn and that have created a better learning environment in your school? 2. How have the activities helped you to learn better? For example, how have the activities made the school more interesting or improved your learning experience? 3. How have project activities reduced verbal or emotional aggressions or mistreatment among your classmates? 4. Have you witnessed some form of abuse in the school by your classmates, teachers or others? 5. Which project activities, and how, have helped you avoid: 1) walking with bad companions, 2) bad conduct with your companions, 3) illegal activities, and 4) dropping out of school? 6. What has been your participation in different project activities, for example, committees, clubs, or leadership positions? 7. In what way has the school formed or taken advantage of linkages with actors outside the school? (For example, municipality, municipal police, or other members of the community.) 79 GUIDE FOR PARENTS Municipality School Sex Male: Female: INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. Now, let’s talk about how the project activities have affected you children in the following areas: 1. Which project activities de you like the most, that have helped your children to learn and that have created a better learning environment in the school? 2. How have the activities helped your children to learn better? For example, how have the activities made the school more interesting or improved your children’s learning experience? 3. How have project activities reduced verbal or emotional aggressions or mistreatment among students? 4. Have you witnessed some form of abuse in the school by your classmates, teachers or others? 5. Which project activities, and how, have helped your children avoid: 1) walking with bad companions, 2) bad conduct with their companions, 3) illegal activities, and 4) dropping out of school? 6. What has been your participation in different project activities, for example, committees, clubs, or leadership positions? 7. In what way has the school formed or taken advantage of linkages with actors outside the school? (For example, municipality, municipal police, or other members of the community.) 80 GUIDE FOR TEACHERS AND DIRECTORS Municipality School Name Sex Telephone E-mail INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. With which of the project activities are you familiar? Please describe them as best you can. Now, we are going to talk about how project activities have affected different aspects of the life and schooling of children and youth in 7th to 9th grade. 1. How do project activities affect your students in the following areas: 1) academic achievement, 2) classroom behavior, 3) dropout, 4) safe school learning environment, 5) bad behavior and bad influences, and 6) inclusion of girls and persons with disabilities. Do you believe these results will last over the long-term? 3. Which of the activities have been the most effective in obtaining the above results? How? 4. Describe the training your school has received from the project, including who received the training and in what subjects, and the quality of the training and the training organizations. Which methods have shown to be most effective? What type of follow-up have you received from the training organizations? 5. How have you sought to replicate these activities? (For example, are you participating in learning circles or teacher networks?) 6. In what way as the Ministry of Education, including ATPs, participated in the project? How have they affected the project in terms of implementation and results? To what extent do you believe that the project activities are consistent with the Ministry of Education’s priorities? 8. In what way has the school formed or taken advantage of linkages with actors outside the school? (For example, municipality, municipal police, or other members of the community.) 7. What are the strengths and weakness of the project? 8. What do you recommend to improve the project and its results? 81 GUIDE FOR IMPLEMENTING PARTNERS AND USAID STAFF Name Position Sex Telephone E-mail INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. 1. What is your role in your organization and within the project? 2. How do project activities affect the following areas: 1) academic achievement, 2) classroom behavior, 3) dropout, 4) safe school learning environment, 5) bad behavior and bad influences, and 6) inclusion of girls and persons with disabilities. Do you believe these results will last over the long-term? 3. Which of the activities have been the most effective in obtaining the above results? How? 4. In what way as the Ministry of Education, including ATPs, participated in the project? How have they affected the project in terms of implementation and results? Do you believe that the project activities are consistent with the Ministry of Education’s objectives? 5. To what extent do you believe that the project has addressed topics that are relevant to lower secondary school in El Salvador? To what extent do you believe that the project activities are consistent with the Ministry of Education’s priorities? 6. What implementation changes or adaptations has your organization made in the project? Why? 7. What are the strengths and weakness of the project? 8. What do you recommend to improve the project and its results? 9. How useful do you think the monitoring indicators are that the project collects and reports? 10. What systems exist within the project and USAID to generate and use information to make decisions and how do they function? 11. What examples can you cite where USAID or the project have used performance information from the project to make decisions? 12. What are the principal challenges that USAID and the project face regarding the development and use of project performance information? 13. What do you recommend to improve the project monitoring and evaluation system? 82 GUIDE FOR MINED STAFF Municipality Department Sex Name Telephone E-mail INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. 1. What do you know of the project? With which of its activities are you familiar? 2. How are you and the Ministry involved in the project? (For example, what training have you received or provided as part of the project?) 3. How do project activities affect the following areas: 1) academic achievement, 2) classroom behavior, 3) dropout, 4) safe school learning environment, 5) bad behavior and bad influences, and 6) inclusion of girls and persons with disabilities. Do you believe these results will last over the long-term? 4. Which of the activities have been the most effective in obtaining the above results? How? 5. How has the Ministry affected the project and its activities and results? What things, including those inside the Ministry, affect your ability to contribute to the project? 6. To what extent do you believe that the project has addressed topics that are relevant to lower secondary school in El Salvador? To what extent do you believe that the project activities are consistent with the Ministry of Education’s priorities? 7. What are the strengths and weakness of the project? 8. What do you recommend to improve the project and its results? 83 GUIDE FOR OTHER PROGRAM STAFF Organization/Program Sex Name Telephone E-mail INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. 1. What do you know of the project? With which of its activities are you familiar? 2. How do project activities affect the following areas: 1) academic achievement, 2) classroom behavior, 3) dropout, 4) safe school learning environment, 5) bad behavior and bad influences, and 6) inclusion of girls and persons with disabilities. Do you believe these results will last over the long-term? 3. Which of the activities have been the most effective in obtaining the above results? How? 4. To what extent do you believe that the project has addressed topics that are relevant to lower secondary school in El Salvador? To what extent do you believe that the project activities are consistent with the Ministry of Education’s priorities? 5. What are the strengths and weakness of the project? 6. What do you recommend to improve the project and its results? 84 GUIDE FOR OBJECTIVE 2 Organization/Program Sex Name Telephone E-mail INTRODUCTION ● We are evaluating the Education for Children and Youth Project that has implemented activities in your school, and we are seeking your opinion on how the project can be approved. ● The information that you provided will be kept strictly confidential. ● If you don’t understand something, please feel free to ask. 1. What do youth leave school? How come they are now returning to school in this course? What benefits do they expect to receive from their participation in the class? 2. What are the challenges that confront youth in this type of modality? What are the challenges that confront teachers? For example, the method of instruction, class length, course length, etc. 3. What type of support or feedback do students receive from teachers? What about teachers from the project? Is this sufficient? Why? 4. What type of support do the municipality, Ministry of Education, or other actors give to youth or your instructors? For example, recruitment, provision of classrooms, security, transport, or food? 5. How do you think the classes prepare youth for the sufficiency test and PAES? What more is needed to prepare and achieve passing marks? 6. What are the strengths and weaknesses of the project? 7. What do you recommend to improve classes and obtain better results? 85 ANNEX 8: MINI-SURVEYS MINI-SURVEY FOR PARENTS AND STUDENT TYPE OF RESPONDENT (Parents or Students): _____________________________________ SEX: ________________________________________ SCHOOL: __________________________________________________ GRADE: _______________________________________________________ Please indicate in the appropriate space the extent to which you agree with the following statements using a 10-point scale, where 1=Completely disagree and 10=Completely agree. Please respond the best you can even if you are not 100 percent certain. However, if you feel you cannot answer a question, leave it blank. Do you have any questions? STATMENT POINTS Project activities have helped improve the behavior of lower secondary school students. Project activities have helped lower secondary schools students to avoid walking in bad company. Project activities have helped establish a safe school-learning environment for lower secondary school students. Project activities have helped lower secondary school students to complete the 9th grade. Project activities have helped improve classroom performance among lower secondary school students. Results of project activities are likely to last over the long-term. Project activities include strategies aimed specifically at including girls or students with disabilities. Project activities have been well implemented. Project activities have addressed important school needs. Students have taken active part in project activities. Parents have taken active part in project activities. 86 MINI-SURVEY FOR DIRECTORS AND TEACHERS TYPE OF RESPONDENT (Director or Teacher): ________________________________________ SEX: ________________________________________ SCHOOL: __________________________________________________ Please indicate in the appropriate space the extent to which you agree with the following statements using a 10-point scale, where 1=Completely disagree and 10=Completely agree. Please respond the best you can even if you are not 100 percent certain. However, if you feel you cannot answer a question, leave it blank. Do you have any questions? STATMENT POINTS School activities have helped to reduce early youth gang infiltration. School activities have helped address the factors that protect lower secondary school youths from becoming involved in crime. Project activities have helped establish a safe school-learning environment for lower secondary school students. Project activities have helped lower secondary school students to complete the 9th grade. Project activities have helped out-of-school youth to return to school. Results of project activities are likely to last over the long-term. Project activities include strategies aimed specifically at including girls or students with disabilities. Project activities aimed at improving inclusion of girls or disabled students have been effective. Project activities were implemented according to plan. Implementing partners implementing project activities have the necessary capacity The training you received was useful to help you do your job better. The Coexistence Committee exists and functions well. The school has a coexistence plan, and it is being implemented. You are participating in activities to promote the replication of project activities to other schools, directors, teachers, etc. Project activities have been well implemented. Project activities have addressed the topics that are important for lower secondary school education in El Salvador. Project activities have been appropriate to achieve project objectives. 87 MINI-SURVEY FOR IMPLEMENTING PARTNER AND USAID STAFF SEX: ________________________________________ ORGANIZATION: __________________________________________________ Please indicate in the appropriate space the extent to which you agree with the following statements using a 10-point scale, where 1=Completely disagree and 10=Completely agree. Please respond the best you can even if you are not 100 percent certain. However, if you feel you cannot answer a question, leave it blank. Do you have any questions? STATMENT POINTS School activities have helped to reduce early youth gang infiltration. School activities have helped address the factors that protect lower secondary school youths from becoming involved in crime. Project activities have helped establish a safe school-learning environment for lower secondary school students. Project activities have helped lower secondary school students to complete the 9th grade. Project activities have helped out-of-school youth to return to school. Results of project activities are likely to last over the long-term. Project activities include strategies aimed specifically at including girls or students with disabilities. Project activities aimed at improving inclusion of girls or disabled students have been effective. Project activities were implemented according to plan. Implementing partners implementing project activities have the necessary capacity Project activities have addressed the topics that are important for lower secondary school education in El Salvador. Project activities have been well implemented. Project activities have been appropriate to achieve project objectives. The projects M&E system collects useful information related to project performance. There exist within the project and project implementing partners good systems to disseminate and use information to improve the project. The project uses performance information from its M&E system to make project adaptations. 88 ANNEX 9: EVALUATION TEAM The Evaluation Team was comprised of seven individuals: Dr. Gary Woller, Team Leader; Cristina Accioly, Education Specialist; Carlos Umaña, Research Specialist; Margarita Sanchez, Gender Specialist; Amalia Alejandro, Research Specialist; and Evelyn Jacir de Lovo, Policy Advisor. The Team Leader was responsible for overseeing and coordinating all activities related to the evaluation, and for ensuring the production and completion of deliverables in a timely and professional manner and in compliance with the evaluation Scope of Work. Two sub-teams, consisting of one host country consultant and one international consultant (Dr. Woller with Ms. Alejandro, and Mr. Umaña with Ms. Accioly) conducted all school and OSY group site visits in addition to KIIs with MINED officials, ATPs, and selected IP staff. Ms. Sanchez was responsible primarily for conducting interviews with USAID, selected IP staff, and other development organizations. Ms. Jacir de Lovo acted in an advisory role to the ET, providing higher-level guidance on education policy issues in El Salvador relevant to the ECY evaluation and accompanying the ET to the field on two occasions). Dr. Gary Woller is a Senior Evaluation Specialist at Mendez England & Associates. Prior to that he was an independent consultant for 15 years specializing in M&E, a Professor of Public Management at Brigham Young University, and Research Director at FINCA International. He has a Ph.D. in Political Science from the University of Rochester. Carlos Umaña is a sociologist and economist with extensive experience in research, territorial planning and in the elaboration of evaluation and monitoring systems. In 2010, he participated in the initial design of the territorial strategy of the GOES full-time integrated school intiative. Cristina Accioly is a senior education specialist with substantial experience conducting program evaluations for USAID and other donors, including the Inter-American Devleopment Bank, Caribbean Development Bank, and World Bank. Ms. Accioly has an Ed.M. in International Education Policy from Harvard University. Evelyn Jacir de Lovo has over 25 years of experience working in public policy at a national, regional, and hemispheric level. As a technical level government officer (10 years), she worked as an advisor to ministers and directors. At higher level government positions (10 years, she served as Minister of education, Coordinating Minister of the Social Cabinet, and President of the Consumer Protection Agency. At the hemispheric level (7 years), she as director of Social Development and Employment Department, Advisor to the Secretary General in Social Development and Inclusion at the Organization of American States. Amalia Alejandro has worked as a consultant with MGR & Associates, United Nations Development Programme, World Food Programme and Women’s Institute. Prior to that she was a journalist with 22 years of experience in communication, magazine design, and radio program management. She has experience in research, elaboration, and production of social and formative content reports. 89 ANNEX 10: GOLDTREE LIEBES CASE STUDY INTRODUCTION This case study documents the transformative power of the ECY activities and their connections to deeper learning and student resilience at Goldtree Liebes School in Panchimalco, El Salvador. The school, which had close to four years of experience forging these connections, offers an extraordinary opportunity to observe educators, students, and the director embracing the ECY Activity as a strategy for turning around years of lawlessness, crime, and gang activities. Goldtree Liebes’ experiences offer an instructive example of how interventions, like those implemented by ECY, can profoundly affect a school’s learning environment and improve student/youth outcomes, and what the factors are that contribute to these results. LITERATURE REVIEW The relationship between education and conflict presents society with a conundrum. “Schools are almost always complicit in conflict. They reproduce the skills, values, attitudes, and social relations of dominant groups in society; accordingly, they are usually a contributory factor in conflict. Simultaneously reconstructing and reforming education is increasingly viewed as a critical element in the strategy to reduce the risk of conflict or relapse into conflict” (World Bank 2005). Fortunately, as this case study demonstrates, schools are surprisingly resilient, and the disruption caused by conflict offers them opportunities to open their doors to innovative programs with alternative solutions not only for students’ academic performance but also for their social and emotional wellbeing. Over the last two decades a growing body of research has attested to the positive impact of Social Emotional Learning (SEL) in the contexts of violence and conflict. In this context, where children and youth are disproportionately and uniquely affected, their resilience and social emotional well-being are essential to any post-conflict long-term reconstruction. The negative effect of armed conflict and prevalent violence on the mental as well as physical wellbeing of children and youth has been broadly documented (Apfel and Simon 1996, Bernard van Leer Foundation 2005, Machel 1996, Summerfield 1991). Such hardships, and the associated impairments they cause, affect not only well-being but also learning outcomes (Kostelny and Wessells 2010). Schools are often seen as a means of ensuring children’s safety and wellbeing (Alexander, Boothby, and Wessells 2012). SEL involves processes through which children and adults acquire and effectively apply the knowledge, attitudes and skills necessary to understand and manage emotions, set and achieve positive goals, feel and show empathy for others, establish and maintain positive relationships, and make responsible decisions (CASEL 2013). Education programs that incorporate SEL can play a crucial role in developing protective factors in children and youth, which mitigate the negative developmental and behavioral effects of exposure to conflict. This is achieved through building intrapersonal and interpersonal skills that are necessary for managing emotions and building healthy relationships. SEL strengthens the healing and coping mechanisms needed to deal with adversity, violence and suffering, which are essential for healthy development (Edgerton and Wessells 2008) and contribute to academic performance at school. Evidence-based SEL programs – such as conflict resolution, life skills, character education, violence prevention, or peace education – are designed to empower children and youth to have improved academic, social, and emotional learning outcomes. These programs play critical parts in the education ecology, but their potential is increased when integrated into daily instruction in a systemic approach. Similarly, supporting evidence suggests that when teachers implement interventions such as proactive classroom management, interactive teaching methods, cooperative learning techniques, and interpersonal skills instruction, students’ positive attitudes and commitment to school significantly increase (Hawkins, et al. 2001). This type of teacher support increases students’ desire and ability to engage in learning, 90 participate in class, and complete homework (Murdock 2001). The Durlak et al. (2011) meta-analysis of 213 school-based universal SEL programs clearly demonstrates the significant role of SEL in promoting the healthy development and academic achievement of all students. It also shows that leadership (director, district-level) support is a critical factor in high-quality implementation. The study, utilizing data from three intervention schools (13 classrooms and 164 students), suggested that two factors contributed to the success of SEL implementation: 1) adequate support from school directors, and 2) a high degree of classroom implementation by teachers (Kam, Greenberg, Walls, 2003). THE CASE STUDY After visiting 22 schools in 11 municipalities in El Salvador, the Evaluation Team selected the Goldtree Liebes school as the subject of analysis for two main reasons: 1) its strong leadership and engagement with the Activity’s interventions, while paving the grounds for sustainability, and 2) the way ECY interventions permeate and transform the school’s culture. The school’s efforts and achievements are even more remarkable when considering its challenging environment prior to Activity implementation. Goldtree Liebes represents the prototype of high violence and crime schools in El Salvador. It is located in a neighborhood with high rates of crime and gang activities. According to the school’s director, “In 2011, when I accepted the position, students were coming to school armed with 9mm pistols and selling 1 ½ lbs. of marijuana a day. Students were either engaged in or were the targets of bullying that contribute to the climate of fear and violence at school.” There were specific school areas where students bought and sold drugs. When the director started making changes to improve safety at school, such as installing cameras in places where gang members trafficked drugs, she received several death threats from gang leaders. As the director notes, “Gangs were providing a ‘negative supply’ of affection and basic necessities to our students who were orphans or had absent parents. The only protection factor they had was the school.” This case study draws on the SEL literature cited above in crisis and conflict contexts, key informant interviews, and focus group discussions with stakeholders from 22 schools in El Salvador, and SEL program evaluations from several urban middle and high schools in the United States. DISCUSSION AND FINDINGS Strong Leadership and Engagement Although the implementation of ECY interventions at the Goldtree Liebes was successful overall, there were areas of resistance as typified by teachers who felt threatened by the changes or were apathetic towards the interventions. The director said, “We need to know how to be the teachers’ captain to work effectively with them. When I started implementing the interventions only two teachers wanted to work with me and 35 percent were not in agreement with changes and left the school. They could not comprehend the future benefits.” A number of factors help explain how the school addressed these challenges and effectively implemented ECY activities. These are described below. School leadership. The director supported a constructive and trusting workplace environment that valued innovation. She perceived the Activity as a comprehensive tool for academic and behavioral school change. On one hand, APA offered an innovative approach to improve students’ cognitive skills, active engagement, and learning outcomes. On the other hand, non-cognitive activities promoted social cohesion, a peaceful environment, and resilience. She said, “Positive leaders and RCC had a strong and positive impact on students’ behavior and school environment. They helped change the school fabric, which used to be a threatening context of violence, drugs, and bullying.” School sponsorship. The Activity had full support of leadership, including the director and ATP. The director perceived the Activity’s interventions as “powerful instruments to make the necessary changes to increase student academic success and decrease gang infiltration at school.” The role of the director was 91 crucial in supporting and catalyzing the implementation process, especially in getting the support of key teachers who later became the champions and leaders in the APA Specialist Network and Learning Circles. Teachers as champions. The solid ownership of the APA methodology propelled teachers to actively participate in: 1) learning circles at school, and 2) specialist networks as chairs of math, science, and language subjects. Teachers reported, “We identify where students have difficulties and together we create strategies to improve learning.” The ATP said, “I extensively participated in the specialist network and was able to witness a strong synergy among teachers. They work together and help each other build their own teaching instrument. They not only elaborate the guides but also validate them twice; amongst themselves and with the students.” Pedagogical model replication. After seeing the impact of APA on students’ performance and engagement, the director replicated the APA methodology in the entire school (1- 12 grades). In addition, the school was unifying the APA guides in order to replicate them with its integrated school system. Goldtree Liebes is the head of a school cluster/system (Sistema Integrado) of nine schools. An Intentional Community A primary indication that Goldtree Liebes made effective use of the interventions is the perceived culture of community and social cohesion. A focus on values, student leadership, cooperation, and conflict￾resolution seemed to develop a sense of belonging and agency among the students. The environment now offers students social ties outside the gangs. Some of the outcomes include: Group activities provided student with social and emotional support. The Activity helped students to increase participation in school activities through RCC, coexistence committee, arts & culture, and sports monitoring. Students learned and practiced a range of social skills in and outside class: theater, music, recycling, directed recreation, and sports. Teachers said, “The activities’ value added is that they foster values while students have fun. Sports monitors are trained to teach younger students discipline while they play their favorite sport.” Fun surpassed anti-social habits. According to teachers, “The ECY activities have transformed the school! Students now are involved in many activities they enjoy. Consequently, gang activities such as charging students rent has decreased at school because gang members perceived that their peers were having fun and they were not. They end up giving up gang duties to join their peers in fun activities.” Culture of peace. Teachers perceived that activities helped the school to create a better learning environment. Teachers believe that APA, RCC, the school peer mediation committee, and the coexistence committee were interconnected and helped students improve their behavior. They said, “The activities helped the school to create a culture of peace. The classroom environment is more harmonious; students get along with each other, there is less bullying, and academic achievement is better.” Frequent assemblies and events brought students, teachers, and parents together. Teachers said, “Coexistence committee and RCC activities taught students about values and how to respect each other while preventing them to go into wrong paths.” Students perceived more solidarity amongst themselves. According to students, “Talks and presentations on violence and drugs, alcohol, sexual abuse, and self￾esteem helped us to choose the right path.” Connections and Engagement While Learning With lessons built on one another in a sequence of active learning activities, APA has enhanced students’ understanding of subjects through guided practice. Goldtree Liebes linked engagement and scholarship in ways that mattered to students. APA was the most popular activity among the girls. Outcomes included: Academic achievement. Students reported that APA helped them to improve academically. “APA is an easy and better way to learn,” reported the girls, “it is very practical and we can also support each other in the process of learning.” Teachers reported, “The methodology benefits not only us who attended the training, but also the institution [school]. We replicated the method to the entire school, from primary to 92 high school.” Strong and purposeful student-teacher relationships were the norm. With the APA active learning methodology, teachers viewed their role as that of coaches and facilitators of learning, with teacher-student relationships at their center. Teachers said, “Students now trust and count on us. When students have problems they come to share their concerns with us.” Again and again, students spoke movingly about how much their teachers cared. Helping students push past fear. Students said, “APA is an easy way to learn, even subjects such as math which was so threatening in the past.” Likewise, the methodology has helped students to develop communication and collaboration skills, which are used in all classes and outside school projects, such as robotics competitions. APA collaborative practice reinforced a harmonious environment at school. Students reported, “APA helped us to develop better relationships with peers while fostering collaboration and teamwork in the classroom. With APA we support each other in the process of learning even when the teacher is not present.” Focus on Developing Student Agency The activities enabled active engagement of children and youth, where their agency was embraced and leadership was offered. Some ways this came across include the following: Peer mediation was created. Forging constructive disciplinary policies, third cycle students who received training in RCC created a mediation committee to solve students’ problems, as well as a brochure to market their initiative. Teachers said, “Students now work together to find solutions to their own problems. When problems are too complex they work in collaboration with the school’s psychologist to find alternative solutions to the problems.” Students were encouraged to find their voice: Students were encouraged to express themselves and to find their voice in class discussions and in personal writing on issues they cared about, when they felt something was unfair, or when they did not understand something. CONCLUSION Goldtree Liebes school administration and teachers were effective in their efforts to institutionalize and catalyze the Activity’s interventions by engaging in a balanced deployment and leveraging of three key drivers of change: people, instruments (activities), and culture. In other words, it succeeded in creating the proper support and culture for teachers to implement the interventions. This was a process that was empowered and catalyzed by a drive from the top of the school in the form of the director. Key factors contributing to the successful implementation of Activity interventions in the school include those described below. Leadership support and top champions. Strong commitment and direction by the director was absolutely critical, perhaps the most critical factor in the school’s success. The director was absolutely committed to transforming the school into a refuge of safety for the students and was, as a result, able to improve the school’s environment considerably. The director said, “In 2011, 11 students disappeared or were killed. In 2015 the number decreased to seven, in 2016 four students were killed, and in 2017 we have zero students dead. That means, today our school offers a safe and peaceful environment for our kids.” Identifying and overcoming resistance to Activity implementation. The two main impediments to implementation were identified: a high violence and crime environment at school and teachers’ resistance to change. The school purposively set out to address these impediments and managed to do so, leading to effective Activity implementation. APA specialist network and learning circle. The analysis of the KIIs and FGDs indicated that the core group of teachers trained in APA was essential to sustain and replicate the active learning methods 93 throughout the school. The methodology application was reinforced and sustained in learning circles at the school level, as well as in the specialist network at the municipal level. REFERENCES Alexander, J., N. Boothby, and M. Wessells. 2010. “Protecting Education from Attack: A State-of-the-Art Review” In Education and Protection of Children and Youth Affected by Armed Conflict: An Essential Link. United Nations Educational, Scientific and Cultural Organization Apfel, R.J., and S. Bennett S. (eds.). 1996. Minefields in their Hearts: The Mental Health of Children in War and Communal Violence. New Haven: Yale University Press. Bernard van Leer Foundation. 2005. “Responding to young children in post-emergency situations.” Early Childhood Matters. The Hague: Bernard van Leer Foundation. CASEL, Collaborative for Academic, Social, and Emotional Learning. 2013. What is Social and Emotional Learning? Chicago: CASEL. http://www.casel.org/social-andemotional-learning Durlak, J.A., R.P. Weissberg, A.B. Dymnicki, R.D. Taylor, and K.B. Schellinger. 2011. “The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions.” Child Development 82(1): 405-432. Greenberg, M.T., Weissberg, R.P., O’Brien, M.U., Zins, J.E., Fredericks, L., Resnik, H., & Elias, M.J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58, 466–474. Hawkins J.D., J. Guo, K.G. Hill, S. Battin-Pearson, and R.D. Abbott. 2001. “Long term effects of the Seattle social development intervention on school bonding trajectories.” Applied Development Science 5:225–236. Humphrey, N. 2013. Social and Emotional Learning: A Critical Appraisal. London: Sage. Kam, C.M., Greenberg, M.T., & Walls, C.T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Kostelny, K., and M. Wessels. 2010. Psychosocial assessment of Education in Gaza and Recommendations for Response. United Nations Educational, Scientific and Cultural Organization. Machel, G. 1996. Promotion and Protection of the Rights of Children: The Impact of Armed Conflict on Children. United Nations International Children's Emergency Fund. Murdock, T.B., N.M. Hale, and M.J. Weber. 2001. “Predictors of cheating among early adolescents: Academic and social motivations.” Contemporary Educational Psychology 26: 96-115. Summerfield, D. 1991. “The psychosocial effects of conflict in the third world”. Development in Practice 1(3):159-173. Wessells, M. and A. Edgerton. 2008. “Concepts and practices to support war-affected children.” The Journal of Developmental Processes 3(2): 2-12. World Bank, The. 2005. Reshaping the Future: Education and Postconflict Reconstruction. Washington, DC: The World Bank. Zins, J.E., R.P. Weissberg, M.C. Wang, and H.J. Walberg. 2004. Building Academic Success on Social and Emotional Learning (SEL): What Does the Research Say? Columbia University: Teachers College Press. 94 ANNEX 11: RISK/PROTECTIVE FACTORS ADDRESSED BY ECY Table A11.1. Risk/Protective Factors Addressed by ECY Intervention Risk/Protective Factor Addressed APA ● Low student motivation towards learning caused by the exclusive use of traditional teaching methodologies ● Teacher use of textbooks as sole teaching resource ● Authority-subordinate relationship between teachers and students ● Non-existence of student leadership and participation in the learning process RCC ● Inadequate interpersonal relationships between teachers, between teachers and students, and between students ● Low capacity and self-esteem of teachers ● Lack of collaborative learning ● Lack of effective vertical and horizontal communication ● Existence of bullying and physical altercations ● Lack of information and capacity to make decisions related to prevention, detection, and attention to bullying Inclusive Education ● Educational and social exclusion of disabled and over-age youth and pregnant girls ● Application of rigid and punitive criteria for determining academic failure without taking into account the needs or conditions of students ● Traditional vision for teaching and learning by teachers with limited access to resources PAP ● Lack of support, including emotional and physical space, for student emotional relief and crisis care ● Lack of orientation and capacity for addressing emotional crises or for linking students to specialized psychological services Vocational Orientation ● Lack of long-term perspective or planning for future by students ● Low personal expectations among students about life prospects ● Lack of information/knowledge about schooling and career options ● Lack of parental/familial support for students to pursue additional schooling and career beyond current circumstances Sports Activities / Directed Recreation ● Lack of variety in sports and recreation activities ● Channeling of energy into non-productive or illegal activities ● Failure to take advantage of sports and recreation as a strategy for teaching values, respect for rules, and cooperation ● Inappropriate student conduct in sports and recreation activities (e.g., language, aggression, and physical violence) ● Exclusion of female students from traditionally male-dominated sports and recreation activities ● Exclusion of physically or mentally disabled students from sports and recreation activities ● Lack of guidelines for appropriate use of student leisure time Robotics / Video Games ● Low student motivation towards learning caused by the exclusive use of traditional teaching methodologies ● Teacher use of textbooks as sole teaching resource ● Authority-subordinate relationship between teachers and students ● Non-existence of student leadership and participation in the learning process ● Low student motivation caused by the lack of alternative learning options to stimulate student curiosity and help them discover/develop new talents ● Lack of guidelines for appropriate use of student leisure time Digital Literacy ● Limited access to technology and connectivity in the school, principally in rural and high-risk locations ● Apathy towards the use of technology by directors and teachers 95 Intervention Risk/Protective Factor Addressed ● Lack of knowledge related to the general use of technology and its application to the school/classroom setting Coexistence Committee ● Absence of a formal mechanism in the school for promoting collegiality and fostering cooperation, through which different stakeholders (directors, teachers, students, parents, and community members) jointly contribute to creating a harmonious learning environment ● Lack of a formal and coordinated planning process for school activities School Management ● Leadership: Difficulty among school administrators in articulating school needs and generating strategies for addressing them ● Pedagogical: Administrative burden placed on school administrators limits their ability to take actions to improve student learning ● Management: Lack of tools to promote a more harmonious and safe learning environment Art & Culture ● Scarce or inadequate opportunities for student artistic and cultural expression ● Lack of guidelines for appropriate use of student leisure time ● Disabled or pregnant students excluded from extra-curricular activities Source: Own elaboration based on ECY documents 96 ANNEX 12: DETERMINANTS OF SCHOOL RISK RATING No Risk Situations Score 1 Gangs have prohibited persons or institutions outside the community and group of teachers from reaching the area or school (direct threats received by the school or by community leaders) 2 Project personnel have received direct threats that prohibit them from reaching the school for a certain period of time. 3 The school authorities communicate directly to project staff that it is not advisable to arrive at school for a certain time. 4 The occurrence of violence within the school or with people directly related to the school shows a high risk of arriving at school for a certain time. 5 Presence of people who appear to be gang members or criminals on the road controlling and even detaining (they stop to ask for information and / or to "pester"). 6 In the area there is a curfew imposed by criminals or gangs. 7 It is necessary to enter accompanied by someone of the school both in the first time as in the following. 8 If the visit is not notified in advance, it is very likely that they will assault or stop it along the way. 9 The gangs seem to control the school: presence of "listener" and leader. They seem to be present in some of the structures of participation and decision making of the school (PTA or others). 10 The access road to the school is solitary in long stretches and there are danger points of assault 11 There are indications that the teachers or the principal (or both) are extorted. The extortion also happens with students. 12 Gangs seem to control information from the school: presence of "listeners.”They have strategically placed people who listen and pass information. Risk Classification 97 ANNEX 13: GENDER INCLUSION REPORT FINDINGS Regarding the application of the ADS Chapter 205 “Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle,” the findings were as follows: PROJECT DESIGN: No documents were available concerning a request for proposal or the initial technical proposal submitted for this project. Thus it is not possible to ascertain if it complied with the Gender and Female Empowerment Policy and other inclusion policies of USAID. PROJECT CONTRACTING: The agreement and sub agreements with implementing partners in the consortium were reviewed. USAID complied with ADS 205 in the primary agreement with FEDISAL, since it contains specific clauses describing expectations in terms of Gender and Disability inclusion (articles A17 and A20 respectively). However, the sub agreements between FEDISAL and its partners (between 2013-mid 2016, before Agape joined) did not include the original wording of clauses A17 and A20. What the sub-agreements included were: ● That data collected had to be disaggregated by “sex, gender and disabled”. ● It obligated them to comply with the Standard Provisions for non US Non-Governmental Organizations that include a NONDISCRIMINATION (M.18) and a DISABILITY INCLUSION (M.19) standard clause that are not as specific as articles A17 and A20 of the original agreement. PROJECT EVALUATION: The terms of reference include gender and historically excluded groups. PROJECT IMPLEMENTATION: This part of the project cycle did not follow ADS 205. The objectives of the Gender and Female Empowerment Policy of closing gender gaps in equality, in gender based violence, and in empowerment were not included in the Logical Framework, nor amongst the indicators of the Project Management Plan until 2015 when an indicator was developed but never reported in 2015 and 2016. As was the case in Gender, the Lesbian Gay Bisexual Trans and Intersex population was not made visible in the project. Disability Inclusion was a distinct component of the project implemented by FUNPRES, and it was periodically reported as a cross sector theme, as well as activities. The Youth Policy was complied with, since the project is an integral part of the USAID El Salvador program and the project specifically targets youth and aims to provide them with increased educational opportunities. CONCLUSIONS The preliminary findings show that ADS 205 was not adhered to during the project implementation part of the program cycle. The clauses stated in the cooperative agreement USAID-FEDISAL (A-17 and A-20) were not in evidence in a satisfactory manner in the First Annual Work Plan, and what was stated in that Work Plan was not reported in the Quarterly Reports. Standard Gender Indicators (for participation and for empowerment) could have been included since training was a major part of the project, but were not included in the MEL Plan. The single gender indicator devised was included only in 2015 and was never reported. The activities that involved gender were limited to promoting equal participation. However, no analysis was made of gender gaps in participation, why they occurred, and whether gender equality was the best 98 approach. There were equity measures promoted for young mothers to be able to continue their education whether in the public school system (by allowing them to continue and take time to breastfeed their babies) and in the out of school youth component. In addition, the Vocational Training Component and the Life Skills training for school principals reportedly included a gender module. EVIDENCE IN FINDINGS The findings of the documentary review and institutional interviews show that Gender related actions were as follows: ● Project Monitoring Plan: The 2015-2018 Project Monitoring Plan included a Gender Indicator, No. 11b: GR-1.4G “Number of principals, teachers and MOE administrators enrolled in any gender educational activity to promote equality and women empowerment”. It showed targets for 2015-2017 but no baseline nor participants trained. The Quarterly Reports for 2015 up to December 2016 did not report on this indicator.22 ● First Year Work Plan: The 2013 Annual Work Plan refers to gender in Section 1.1.1, Result 1A￾Activity 3. Implement the expansion of the EITP, sub activity 3.4 Design and implement a management training process for lower secondary school principals, stating: “between May and July and there will be another training module on skills for a life and culture of peace, gender equity and inclusive education, work competencies in accordance with the local department vision that will take place between the months of August and September…Don Bosco University will support this process which is validated and certified by the MINED. ● The Quarterly Report for the last quarter of 2013 reported that 363 women and 262 men (total 625 teachers and principals) were trained in Life Skills. The narrative does not mention any gender content in the training nor results. ● Activity 3.5. Design and implement a pedagogic training process for lower secondary school teachers… Includes a statement that “There will also be a consultancy on gender. The consultant will be on charge of creating a methodological proposal for the design, creation and revision of gender approach educational materials; designing a training plan for the technical and field teams and a monitoring and assessment proposal with a gender approach. The Project will be responsible of creating and implementing a strategy that will guarantee a gender approach in all the components of the project.” ● The Quarterly Reports for 2013 to 2016 did not include a report that this activity took place. The interview with personnel from the members of the consortium did not reveal that such a consultancy, report, or strategy ever took place. ● Result 1c Activity 2 in the work plan of the first year, stated that “For monitoring activities, all project indicators will be taken into account which will be disaggregated by gender, age, urban, rural, in absolute numbers and percentages to facilitate comparisons and gender-specific analysis. The information for these indicators will be collected systematically and periodically to identify behavioral tendencies in school attendance and dropout rates by gender, to design activities that meet the specific needs of men and women to complete lower secondary school.” ● Quarterly Reports provide information on trainees by gender but no analysis is made of the differences between them or the implication of gender gaps in the reports. ● 9th and 10th Quarterly Reports (Jan – June 2015) in Section 1.14 Result 1D- Develop a fully Operational Monitoring Process, mention in the 9th report that “the next briefing paper is being prepared for discussion at the gender table, and the corresponding report is being prepared; and in the 10th report, 22 Source: ME Plan Final ECYP English_17 Nov 2015. 99 that Briefing paper two is being prepared for discussion at the gender table, and the corresponding report is being prepared. 11th quarterly report July-Sept 2014 is NOT AVAILABLE, and the 12th report does not mention this anymore. The report is not available in the documentation collected. Work plan activity 2.1 Training processes on coexistence to principals and teacher teams, ATP, fathers, mothers, and volunteers under MOE guidelines considering the gender approach. Implemented by FUSALMO, FUNPRES. ● The 12th Quarterly Report states: Section C. Advance in technical assistance for FTIS: “Teachers, parents, principals and students are highly participating in 408 Coexistence Committees (as APT´s)…. Female as well as men participation an involvement in those Committees are promoted as a gender equity strategy to narrow the gap of male participation in educational decision-making process at local schools.” ● The 12th Quarterly Report states: In regards to the third crosscutting theme, ECYP promotes the participation of women and girls in all its actions. Similarly, the training processes include content for the empowerment of women and a more equitable participation. ECYP also creates and produces educational material with inclusive language with the purpose of breaking gender barriers. However, these generalities are not supported by a list of specific activities as evidence, as is the case when reporting DISABILITY INCLUSION. In addition, Indicator 11b is neither mentioned nor measured in this report. ● 13th Quarterly Report (Jan-Mar 2016) states once again, “In regards to the third cross-cutting theme, the Project promotes the participation of girls and women in every action. At the same time, the training process includes contents for women’s empowerment and a more equitable participation. In the elaboration of educational materials, created and reproduced by the Project, it is used an inclusive language to break up gender barriers.” Again, no evidence stating the names of the documents is provided. ● The 14th Quarterly Report states: B. Technical assistance provided at the 677 selected schools… “The Project promotes five strategic proposals oriented to the educational improvement of lower secondary students and the expansion of FTIS… The strategies are also accompanied by a gender equality and sustainability approach.” No evidence is provided beyond this descriptive paragraph. In addition, it states once again “Regarding crosscutting themes…On the other hand, the Project promotes participation from girls and women in every action developed. At the same time, the training process includes topics to empower women and have an equal participation. During the elaboration of educational materials created and reproduced by the Project, an inclusive language is used in order to break up gender barriers.” The Indicators did not include 11b, nor its measure. ● 15th Quarterly Report July-Sept 2016: For the first time, gender gaps are analyzed in Section 3 INTEGRATION OF CROSSCUTTING ISSUES AND USAID FORWARD PRIORITIES, 3.1 G Gender Equality and Female Empowerment. Analyses of schools “which adopt different activities advocated by the Project and aimed to promote gender equality in schools,” with 80 percent of schools in that period with student organizations showing gender equity in number of roles, 75 percent Extra￾curricular activities showing equal representation between males and females, and 65 percent of the schools catering to pregnant students. Even so, the MEL Plan does not report indicator 11b. Interviews with AIS and FUSALMO regarding Out of School Youth mentioned that among the incentives for pregnant girls or young mothers to return to school were the provision of daycare service during the sessions attended on weekends, and time permitted to breast-feed their children. 100 EVIDENCE PROVIDED OF DISABILITY INCLUSION MEASURES ● 12TH Quarterly Report says: “In regards to the second crosscutting theme, ECYP and FUNPRES keep promoting inclusiveness and providing assistance to youth with special educational needs to ensure the inclusion strategy in the learning process of students who have learning difficulties associated with a disability. This is done with the use of differentiated instruction strategies in the classroom.” ● To date, ECYP is benefiting 8,594 lower secondary level students who have some kind of physical disability such as blindness, deafness, intellectual disability, motor skill disability, hearing loss, among others. Furthermore, there is attention to needs not associated with physical disability such as dyslexia, attention deficit disorder and hyperactivity, among others. ● 13th Quarterly Report (Jan-March 2016): “In this matter, FUNPRES facilitates the learning process of teachers to assist these cases and jointly works with MINED to adapt the curricula to the specific needs of these students in their learning process. According to figures, it is estimated that a total of 8595 lower secondary school students who have any type of physical disability, are benefited. These type of disabilities can be blindness, deafness, intellectual and motor disability, hypoacousis and some others, as well as assistance to other needs not related to a physical disability such as: students with some kind of dyslexia, attention deficit, hyperactivity and others.” ● 15th Quarterly Report: Section 3.2 Disability and Inclusive Development. “Through technical assistance, training on inclusive education, and curricular adjustments, ECY supported 859 lower￾secondary students who have some type of physical disabilities such as blindness, hearing loss, deafness, mental retardation, and motor impairment, among others. In addition, ECY also supports special needs that are not associated with physical disabilities. For example, students have some form of dyslexia, attention deficit, or hyperactivity disorder. Thus, ECY promotes the active participation of disability in school. This quarter, 35 disability students participated in art clubs, which breaks down paradigms about the integration of disabilities in schools. ” 101 ANNEX 14: ECY IMPLEMENTATION ADAPTATIONS Adaptation to the Flexible Modality Approach: The original proposal for the Objective 2 flexible modality classes only included youth in lower secondary school, not youth in high school. On further investigation, however, it was determined that the largest demand for schools existed among high school youth. As a result, ECY decided to open up enrollment to include high school youth as well. Also with regards to Objective 2, ECY conducted a review of the flexible modalities approach and concluded that it was, according to FEDISAL management, “a completely collapsed or failed approach. Objective 2 followed the model from the Ministry exactly, but it is a broken model. The project followed it for the first four years and did what the Ministry wanted. Then the project decided to change the way it was done.” The Activity has proposed the following three experimental modalities, each to be implemented by one of its IPs: ● ACCEDO: Accelerated sufficiency test modality implemented by AIS in Zacatecoluca for 53 youth aged 12-24, covering 402 hours (compared to 700 hours for MINED) over eight months. Instruction is personalized for each youth based on an initial diagnostic test. Includes life skills course and student mentoring. As youth are drawn from coastal areas, instruction will reflect the coastal work and lifestyle, with the objective being that the youth remain in the area after course completion. ● AVANCO: Accelerated sufficiency test modality implemented by AGAPE at AGAPE training facilities in Sonsonate for 80 youth, covering 80 hours over four months. Admission is screened based on an assessment of the likelihood that the youth can pass the sufficiency test in an accelerated timeframe. Instruction is personalized for each youth based on an initial diagnostic test and also includes a mid-term and final diagnostic to track youth progress over time. Links youths to the private sector through internships and other means. ● CONSTRUYO: Job placement modality implemented by FUSALMO in Soyapango, in partnership with Wal-Mart and Winglass, for 33 youth who are participating in one of four MINED flexible modalities: virtual, distance, accelerated, or semi-presence. The objective is to forge linkages between youth and private sector actors to help them find jobs. Additionally, ECY asked AGAPE to design a pilot strategy for improving the sufficiency test modality. For this pilot, AGAPE decided to increase the hours of instruction to 108 hours, add a leveling module aimed at bringing students to a basic competency level before beginning the intensive instruction, and to meet at different days and times during the week instead of a single time on the weekend. During the pilot, 16 youth in Jujutla met from 8-12 am and 1-3 pm on Monday, Wednesday, and Thursday, where they received 12 hours of basic competency instruction, 20 hours of intensive instruction in each of the four subjects, and 16 hours of life skills training. For 2017, AGAPE is continuing the pilot with two new groups of 26 and 24 youth in the community of La Bolsona in Sonsonate. To help the youth defray the cost of taking the sufficiency test, AGAPE took their photographs and prepared the documentation on their behalf to present to MINED. Adaptation to Follow-Up Technical Assistance Approach: During interviews with member of the ECY management team, they were well aware of the problems related to the lack of follow-up technical assistance. In response, ECY had crafted two complementary strategies to strengthen its follow-up TA. First, FUSALMO will no longer be primarily responsible for follow-up TA at the schools. Rather, from now on each IP will be responsible for conducting follow-up TA related to its own interventions, using a team of technical specialists dedicated to this task. The process of providing follow-up TA has begun, or will begin, for each IP during 2017. Second, in November 2016, ECY launched a new outcomes-based approach to TA (as opposed to the previous output-based approach) focused on developing the 102 competencies of Activity-assisted schools. The goals of this new outcomes-based approach are to: a) leave a legacy at the school level, b) create technical portfolios, and c) provide TA led by specialists. ECY began in 2013 with nine interventions; it now implements 21 separate interventions, each based on the specialties of the technical specialists at the different implementing partners. To help run this effort, ECY inaugurated two new management units in March 2017: Covenant Fulfillment (Cumplimiento de Convenios) and Technical Quality Audit (Auditoría de la Calidad Técnica). Under their direction, ECY will conduct a statistically representative survey of Activity-supported schools to measure technical competency levels at the schools. Based on the findings, ECY will develop a TA package to address the average competencies, or average competency deficiencies, at the schools, to be delivered by technical specialists at the Activity IPs. Second, ECY has created a so-called signal system, which is a color coded system based on the information generated through the collection of ECY’s eight process indicators that indicates to what extent schools have implemented ECY interventions. The colors in the signal system and their interpretation are as follows: ● Green: Schools that have implemented or institutionalized the majority of ECY interventions (67 percent-100 percent of process indicators achieved). ● Yellow: Schools that have implemented ECY interventions, but the interventions have not been included in the School Plan or the Annual Operational Plan (34 percent-66 percent of process indicators achieved). ● Red: Schools that have just started with the interventions or that have not started with the interventions yet and need ECY technical assistance (0 percent-33 percent of process indicators achieved). The signal system acts as a triage sorting mechanism that ECY will use to identify and prioritize schools most in need of follow-up TA and then to guide the types of TA provided to those schools. Adaptation to ECY’s Relationship with MINED: ECY also concedes that the Activity’s relationship with MINED has indeed been weak, that the hoped-for transfer of knowledge and sustainability to the Ministry has not occurred, and, moreover, that if it does not occur, then the Activity’s prospects for sustainability are dim. Thus, a top priority for ECY in its later stages is to “transfer to the government all that we have learned during the project,” reflecting the realization that “the Ministry is not going to become empowered unless the project works directly with it” (FEDISAL Management). USAID/El Salvador has approved a plan for ECY to form a collective team with five senior MINED officials – National Director of Schools, National Director of Technology, National Director of OSY and Flexible Modalities, National Director of Secondary Schools, and National Director of Prevention and Social Programs – with the objective of transferring knowledge so that the government is empowered regarding every single detail of the project. According to FEDISAL management, the purpose of the collective ECY and MINED team “serves one purpose: to transfer ECY to MINED.” M&E: In the area of M&E, ECY is seeking to transition from an emphasis on responding to information requests and distributing reports, to a greater focus on KM, while also increasing the field presence of its M&E and management teams (see Section 6). ECY plans to launch its Internet portal shortly, with information and resources for assisted schools and IP staff. (The launch of the portal has been delayed due to the time required to satisfy USAID security requirements.) Pilot Test in Zacatecoluca: For Activity Cohort 4, ECY is implementing a pilot test with 13 schools in Zacatecoluca. The pilot test incorporates a number of adaptations designed to address several of the issues raised above, in addition to others. To begin with, ECY concedes that with Cohorts 1-3 it made a mistake in that it did not address the subject of family. According to ECY, the violence in the country has created a structural break between the schools and families. To address this structural break, ECY will work with 103 Miles de Manos in Zacatecoluca to try to forge closer relationships between schools, teachers, and families, so as to integrate the latter more firmly into the schools’ fabric, and vice versa.23 With regard to the vocational orientation intervention, ECY will administer the aptitude test only to those schools that are actively implementing the intervention, with the rationale that the aptitude test only makes sense when students have more information about school and career opportunities. In those schools implementing the intervention, and where ECY has administered the aptitude test, it will provide school and career mentoring for students. Mentoring will take the form of an adult from the school or community who will provide one-on-one counseling to the students. In Cohort 4, ECY will put more emphasis on training directors alongside teachers and seek to increase the number of teachers and students trained when possible. A major change in this regard is that for the APA training, all lower secondary teachers and directors will be invited to attend. Similarly, for sports monitoring and art and culture training, schools will be invited to send more teachers and students to the training workshops, where they will have more flexibility to select the areas in which they are most interested. In the area of coexistence, Cohort 4 will seek to increase the presence of parents on the coexistence committee and strengthen their role there, while also providing more hours of training to strengthen committee functioning. 23 Miles de Manos is a component of the GIZ program PREVENIR aimed at preventing youth violence via formal school and out-of-school education. 104 ANNEX 15: ECY INTERVENTIONS Intervention Implementing Partner Implementation Timeframe Active Teaching Methods (APA) FHI360 2014- April 2016 Active Teaching Methods (APA) and Innovation by Specialty UDB 2013-2015 Creative Conflict Resolution (RCC) y Prevention of Bullying FUNPRES 2013-2016 Inclusive Education FUNPRES 2013-2015 Psychological First Aid (PAP) FUNPRES 2013-2017 Vocational Orientation FEDISAL 2014-2017 Sports Monitoring FUSALMO 2014-2015 Directed Recreation FUSALMO 2014-2015 Robotics FUSALMO 2015-2016 Educational Video Games FUSALMO 2014-2016 Digital Literacy FUSALMO 2014-2015 Virtual Communities FUSALMO 2014-2017 Coexistence Committee FUSALMO 2014-2017 School Management FHI360, FEDISAL, FUSALMO, FUNPRES 2015-2017 Art and Culture FUNPRES, FUSALMO 2015-2017 Objective 2 Life Skills AIS, FUSALMO, AGAPE 2013-2017 Workforce AIS, FUSALMO, AGAPE 2013-2017 Induction Course AIS, FUSALMO, AGAPE 2015-2017 Tutoring for Sufficiency Test AIS, FUSALMO, AGAPE 2014-2017 105 ANNEX 16: EVALUATION TIMELINE March 2017 Sun Mon Tue Wed Thu Fri Sat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 ET arrives in San Salvador for Inception Visit 27 Inception Visit 28 Inception Visit 29 Inception Visit 30 Inception Visit 31 Inception Visit 106 April 2017 Sun Mon Tue Wed Thu Fri Sat 1 ET departs San Salvador 2 3 ET prepares draft Evaluation Design 4 ET prepares draft Evaluation Design 5 ET prepares draft Evaluation Design 6 ET prepares draft Evaluation Design 7 ET prepares draft Evaluation Design 8 9 10 ET prepares draft Evaluation Design 11 ET prepares draft Evaluation Design 12 ET submits draft Evaluation Plan for internal review 13 ET reviews and comments on draft Evaluation Plan 14 ET revises draft Evaluation Plan and submits it to USAID for review and comment 15 16 17 USAID reviews comments on draft Evaluation Plan 18 USAID reviews comments on draft Evaluation Plan 19 USAID reviews comments on draft Evaluation Plan 20 ET revises draft Evaluation Plan based on USAID comments 21 ET revises draft Evaluation Plan based on USAID comments 22 23 ET travels to San Salvador for fieldwork 24 ET meets with USAID and FEDISAL to finalize evaluation plan and fieldwork logistics 25 ET meets with USAID and FEDISAL to finalize evaluation plan and fieldwork logistics 26 ET begins fieldwork Objective 1 27 Fieldwork Objective 1 28 Fieldwork Objective 1 29 Fieldwork Objective 2 30 107 May 2017 Sun Mon Tue Wed Thu Fri Sat 1 ET enters interview notes into Google Drive 2 Fieldwork Objective 1 3 Fieldwork Objective 1 4 Fieldwork Objective 1 5 Fieldwork Objective 1 6 Fieldwork Objective 2 7 8 ET enters interview notes into Google Drive 9 Fieldwork Objective 1 10 Fieldwork Objective 1 11 Fieldwork Objective 1 12 Fieldwork Objective 1 13 Fieldwork Objective 2 14 15 ET enters interview notes into Google Drive 16 Fieldwork Objective 1 17 Fieldwork Objective 1 18 Fieldwork Objective 1 19 ET prepares Out￾Brief for USAID ET presents Out￾Brief for USAID 20 Fieldwork Objective 2 21 ET departs San Salvador 22 ET prepares draft Final Report 23 ET prepares draft Final Report 24 ET prepares draft Final Report 25 ET prepares draft Final Report 26 ET prepares draft Final Report 27 28 29 ET prepares draft Final Report 30 ET prepares draft Final Report 31 ET prepares draft Final Report 108 June 2017 Sun Mon Tue Wed Thu Fri Sat 1 ET prepares draft Final Report 2 ET prepares draft Final Report 3 4 5 ET prepares draft Final Report 6 ET prepares draft Final Report 7 ET prepares draft Final Report 8 ET prepares draft Final Report 9 ET prepares draft Final Report 10 11 12 ET prepares draft Final Report 13 ET submits draft Final Report for internal review 14 ET reviews and comments on draft Final Report 15 ET reviews and comments on draft Final Report 16 ET revises draft Final Report and submits it to USAID for review and comment 17 18 19 USAID reviews and comments on draft Final Report 20 USAID reviews and comments on draft Final Report 21 USAID reviews and comments on draft Final Report 22 USAID reviews and comments on draft Final Report 23 USAID reviews and comments on draft Final Report 24 25 26 USAID reviews and comments on draft Final Report 27 USAID reviews and comments on draft Final Report 28 USAID reviews and comments on draft Final Report 29 USAID reviews and comments on draft Final Report 30 USAID reviews and comments on draft Final Report 109 July 2017 Sun Mon Tue Wed Thu Fri Sat 1 2 3 ET revises Final Report based on USAID comments 4 ET revised Final Report based on USAID comments 5 ET revised Final Report based on USAID comments 6 ET revised Final Report based on USAID comments 7 ET revised Final Report based on USAID comments 8 9 10 ET revised Final Report based on USAID comments 11 ET revised Final Report based on USAID comments 12 ET revised Final Report based on USAID comments 13 ET revised Final Report based on USAID comments 14 ET submits revised Final Report and One-Page Summary to USAID 15 16 17 18 19 20 21 ET submits Raw Data to USAID 22 23 24 25 26 27 28 ET makes Final Presentation to USAID 29 30 31 DRAFT 110 ANNEX 17: DISCLOSURE OF ANY CONFLICTS OF INTEREST 111 112 113 114