November 29, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by DevTech Systems, Inc. under Contract Number AID-520-C-13- 00001. Learning Project Midterm Performance Evaluation Report i Monitoring & Evaluation Program (MEP) Lifelong Learning Project Midterm Performance Evaluation Report Submitted to: Ángel López, Contracting Officer’s Representative USAID/Guatemala Submitted by: DevTech Systems, Inc. Contract AID-520-C-13-00001 Acknowledgment: The coordination of this document was under Kristin Rosekrans as a Team Leader, with the consultants, Samuel Fadul, Armando Secaira, Maryam Ahranjani and Mario Moreno (MEP). Photo Credit: Armando Secaira DISCLAIMER: This report is made possible by the support of the American People through the United State Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech and do not necessarily reflect the views of USAID or the United States Government ii TABLE OF CONTENTS Executive Summary ..........................................................................................................................1 Resumen Ejecutivo............................................................................................................................5 I. Introduction and Background...................................................................................................9 I. Evaluation Design and Methodology ...................................................................................11 II. Problem Analysis......................................................................................................................14 III. Description of Lifelong Learning.........................................................................................18 IV. EXPECTED RESULTS ........................................................................................................19 V. Findings and Conclusions ...................................................................................................23 a. Learning to Read ...........................................................................................................................................23 1. Implementation of improved approaches to early grade reading instruction and materials.........23 2. Human resource development in the education system.................................................................................35 3. Parental and community participation in education leading to improved learning and reading achievement.............................................................................................................................................................39 B. Education for Employment and Lifelong Learning.................................................................................42 1. Equitable access to alternative basic education and workforce training for out-of-school youth (OYS).........................................................................................................................................................................42 2. Improve the quality of alternative basic education and workforce training .................................................44 3. Youth Engagement........................................................................................................................................45 VI. Adjustments made by Implementers and Recommendations.....................................47 VII. Appendices .............................................................................................................................60 1 Lifelong Learning Project Midterm Evaluation EXECUTIVE SUMMARY In March 2014, USAID/Guatemala awarded the Lifelong Learning Project (LLP) to Juarez and Associates, which has a performance period from 2014 to 2017 for the base period and from 2017 to 2019 for the option periods. Lifelong Learning’s objective is to improve education quality and increase access for underserved populations (i.e., indigenous children and out-of￾school youth). There are two components to the project: A) Learning to Read which provides technical assistance to the MOE to increase teacher effectiveness, improve classroom learning environments, and foster effective first and second language (Mayan languages and Spanish) literacy acquisition (amongst other complementary strategies) and B) extending alternative basic education and workforce training opportunities to out-of-school youth and increasing their civic engagement. USAID required a mid-term performance evaluation of LLP is to inform the next phase of implementation. The mid-term evaluation of LLP employed mixed-methods that included interviews and focus groups with implementing partner organizations, government and non-government stakeholders, and project participants (beneficiaries). It also employed documental analysis and quantitative analysis of data from the implementing partner’s longitudinal study of student reading test. Additionally, it included several visits to schools in the municipalities of LLP implementation. Overall, the interventions carried under both components are promising and beginning to show positive changes; some changes are more evident that others and several adjustments are suggested for strengthening and adjusting aspects of the project to ensure maximum benefit. The actions carried out for improving early grade literacy in bilingual contexts is helping to strengthen a specific area of the national model for Bilingual and Intercultural Education (EBI) at a central MOE level and in five departments in intervention. The model supported by LLP complements the national EBI model in that it is not restrictive; it aims to improve biliteracy development through strengthening the capacity of teachers and MOE staff to implement bilingual instruction but does not exclude other approaches. The LLP model is congruent with the MOE Strategic Plan and various interventions are beginning to be appropriated by the MOE at the departmental level. However, few stakeholders in the MOE departmental offices, teachers and parents have a comprehensive view of the model. There are fewer guidelines mandated by the central MOE and more departmental MOE autonomy, which is positive yet also implies that continued effort must be placed on strengthening departmental human capacity. For the remainder of the project, a strong emphasis should be placed on assuring that MOE department staff and teachers take ownership over the interventions and approaches supported by LLP and that they share a common vision of how to provide instructional support systematically. Strong communication between the implementing partner and MOE central and departmental levels is important, as is documenting best practices that result from the interventions. Furthermore, while attitudes about children learning in the Mayan language seem to be changing, continued effort at a national and local level is necessary, as is a strategy for increasing the number of teachers who can teach in Mayan languages. 2 In 2014, Juarez implemented a mechanism to track students’ progress in reading and writing in Spanish and Mam/K’iche’. This included the elaboration and validation of standardized tests with the plan to apply them to students in grades one to three in target schools in 2015, 2016, and 2017 (including 4th graders for 2015). In 2015 students were evaluated at the beginning and end of the school year, though it is too early to expect substantial improvements due to the short time of implementation as well as contextual factors those beyond the scope of the project. However, the analysis of the results does point to some positive changes. It is recommended that certain aspects of the LLP data bases be improved, that additional analysis techniques (like propensity score matching) be employed, and that there be further examination of the quantitative data to verify/identify variables that have a positive or negative influence on reading abilities. The effort to strengthen capacity through human resource development is demonstrating positive effects with teachers and MOE personnel at the departmental levels. There is clarity and appreciation of newly acquired knowledge, which is providing both theoretical and practical foundations of bilingual literacy development. Incipient changes regarding attitudes about “coaching” are evident, yet it is important to consolidate a common vision (between implementing partners, CTAs, teachers, and MOE central and departmental personnel) for how to systematically capture changes in teaching practices as well as having the right support instruments for coaching. For the preparation of teacher educators, the consortium of universities has achieved good articulation and cooperative work until now. In the future, their role could be to document and consolidate the experience of the teacher educator professional development program as well as to serve as an advisory body for universities and the MOE in areas such as pre-service teacher education, coaching, intercultural and bilingual education, education leadership, educational measurement and evaluation, and development of support materials. There is a clear need for materials to support literacy improvement in Spanish and the Mayan languages; this includes teacher guides, story books, big books, textbooks/readers for students, amongst other support materials. Some advances have been made in the departments to produce materials (such as the anthologies) and some departmental personnel would like to continue this effort; this will require more support from LLP both in assuring technical capacity and that the resources are provided by the central MOE. Additionally, while the MOE has provided materials for schools supported by the prior government, it is unclear the degree to which these are present in the schools that LLP is targeting. It is important to have an inventory of what materials exist in the schools and classrooms to assure that teachers have the necessary resources for effectively helping their students develop their language abilities. Additionally, teachers need guides with didactic strategies for L1/L2 literacy instruction and to develop their own Mayan language abilities. LLP’s implementing partner has recognized this need and is compiling materials for a teacher’s kit to be given to them at the end of their professional development process. Providing them with concrete didactic approaches and strategies and example lessons in K’iche’, Mam, and Spanish (depending on the particular need) should be a priority. Also, involving teachers in this process is recommendable such as having teachers develop, pilot and share lessons/strategies as part of their professional development. Furthermore, providing them with tools/materials to develop the Mayan language they must know for bilingual instruction is key; this could be done in coordination with the ALMG and/or other language specialists. 3 Finally, the sub-component to involve parents in their children’s education is complementary for improving student learning and it is demonstrating positive effects. However, to assure that the desired effects are achieved in all target areas, links must be further strengthened between the quality and community education units (central and departmental levels) as well as assessing the degree to which this intervention is functioning in the schools as intended. The LLP component to extend opportunities to out-of-school youth for education, workforce training, and civic engagement is generating positive effects as well as demonstrating lessons learned that can help inform future strategies for this new terrain of support. Overall, findings reveal that there are now more educational alternatives and workforce development training opportunities available to out-of-school youth than prior to LLPs implementation. As of September 2016, LLP had registered 5,259 out-of-school youth. However, a relatively small proportion of the eligible population is actually enrolled in formal educational alternatives or workforce development thus far. The first year of implementation was a learning curve for the implementing partners due to unforeseen realities unknown prior to intervention such as the difficulty of finding youth, the lack of data on this population, parental resistance to their children’s involvement, and high costs of transportation. Much of these obstacles are being resolved, however several substantial challenges remain. Positive changes noted thus far include: 1) young people expressing very positive changes in their lives since participating in the education and training (such as self-esteem, self-confidence, overcoming their fears, and the capacity to deal with adults) as well as advancing their education and in some cases finding work opportunities, 2) the organization of three groups to engage youth to develop a life plan, 3) youth organizations strengthened that are raising the awareness of authorities at local, community, municipal, and central government levels about OSY and the issues they face, 4) a network established of four well-respected organizations which are working together to provide alternative basic education and workforce training to OSY, 5) advances in strengthening the MOE division responsible for attending to OYS (DIGEEX), including the development of SIREEX to identify, assist, and track youth, 6) the development of quality standards for alternative basic education and workforce development, and 7) more youth engagement with government, non-governmental organizations, and community organizations regarding their needs and how to get them met. Challenges for LLP’s implementing partner that remain include: 1) the need to establish a cohort-segmented report to track educational and occupational processes of OSY including an ID or bar code for each youth and the necessary equipment to do so, 2) expanding workforce training opportunities to include additional options to meet the needs of the demands of the youth in regards to their interests and existing local work opportunities, 3) prioritizing the engagement of parents (especially for youth who are 15-17 years old) by providing information about the benefits of participation, how they can support them, and measures taken to protect their youth when they participate, 4) establish codes of conduct and security protocols for youth and staff working with youth to strengthen positive behavior as well as trust with parents, 5) assure that the indigenous values, traditions, language and other elements that reflect their culture are present in materials and activities with youth and when engaging parents, 6) create links with a number of new or existing governmental programs within the Ministries of Economy, Labor and Education to exchange experiences focusing on OSY, instead of only schooled youth, 7) further strengthen the operation of SIREEX with personnel hired by the LLP project, considering the 4 possibility of working inside DIGEEX and 8) continue to clarify and refer to OSY in all communications with municipal, departmental, and central authorities of the Guatemalan government as to their explicit responsibility of OYS. 5 Proyecto de Aprendizaje de toda la Vida Evaluación Intermedia RESUMEN EJECUTIVO En marzo de 2014, USAID/Guatemala adjudicó el Proyecto de Aprendizaje de Toda la Vida (LLP) a Juarez and Associates, el cual tiene un período de ejecución de 2014 a 2017, para el período base, y del 2017 a 2019 como períodos de opción. El aprendizaje permanente tiene como objetivo mejorar la calidad de la educación y aumentar el acceso de las poblaciones sub￾atendidas (es decir, los niños indígenas y jóvenes que no asisten a la escuela). El proyecto consta de dos componentes: a) El Aprendizaje de la Lectura que proporciona asistencia técnica al Ministerio de Educación (MdeE) para aumentar la eficacia de los maestros, mejorar los ambientes de aprendizaje en el aula, y fomentar la adquisición de alfabetización (entre otras estrategias complementarias) en lengua materna y en segunda lengua (lenguas mayas y español) y b) Extender la educación básica alternativa y oportunidades de capacitación laboral para jóvenes que no asisten a la escuela y aumentar su compromiso cívico. USAID requiere una evaluación de rendimiento a mediano plazo de LLP como informe hacia la próxima fase de aplicación. La evaluación intermedia de LLP empleó métodos mixtos que incluyeron entrevistas y grupos focales con las organizaciones asociadas en la ejecución, con interesados gubernamentales y no gubernamentales, y con los participantes en el proyecto (beneficiarios). También empleó análisis documental y análisis cuantitativo de los datos del estudio longitudinal de las pruebas de lectura de los estudiantes. Además, se incluyeron varias visitas a las escuelas en los municipios de aplicación del LLP. En general, las intervenciones de ambos componentes son prometedoras y empieza a mostrar cambios positivos. Algunos cambios son más evidentes que otros y se sugieren varios ajustes para fortalecer y refinar aspectos del proyecto para asegurar el máximo beneficio. Las acciones llevadas a cabo para mejorar el grado de alfabetización temprana en contextos bilingües están ayudando a fortalecer un área específica del modelo nacional para la Educación Bilingüe e Intercultural (EBI) en un nivel central del MdeE y en cinco departamentos de intervención. El modelo apoyado por LLP complementa el modelo EBI nacional en el sentido de que no es restrictivo; apunta a mejorar el desarrollo del bialfabetismo mediante el fortalecimiento de la capacidad de los docentes y el personal del MdeE para implementar la instrucción bilingüe pero no se excluyen otros enfoques. El modelo LLP es congruente con el Plan Estratégico del MdeE y el MdeE está empezando a adoptar diversas intervenciones a nivel departamental. Sin embargo, pocos de los involucrados de las oficinas departamentales del MdeE, maestros y padres tienen una visión global del modelo. Hay menos directrices establecidas por el MdeE y más autonomía departamental del MdeE, lo cual es positivo, pero también implica que el esfuerzo continuado debe guiarse hacia el fortalecimiento de la capacidad humana de los departamentos. Para lo que resta del proyecto, se debe hacer hincapié en garantizar que el personal de los departamentos del MdeE, y los profesores, asuman la responsabilidad sobre las intervenciones y enfoques apoyados por LLP y que compartan una visión común de cómo proporcionar apoyo instructivo sistemáticamente. Es importante la comunicación muy cercana entre el ejecutor que implementa y el MdeE central y departamental, así como la documentación de las mejores 6 prácticas que son el resultado de las intervenciones. Además, aunque las actitudes acerca del aprendizaje de los niños en los idiomas mayas parecen estar cambiando, son necesarios los esfuerzos continuos a nivel nacional y local, ya que es una estrategia el aumento del número de maestros que pueden enseñar en idiomas mayas. En 2014, Juarez implementó un mecanismo para rastrear el progreso de los alumnos en lectura y escritura en español y Mam/K'iche'. Esto incluyó la elaboración y validación de pruebas estandarizadas con el plan para aplicarlas a los alumnos en los grados primero a tercero en escuelas de destino en 2015, 2016 y 2017 (incluyendo estudiantes de 4º. grado para el año 2015). En 2015 los estudiantes fueron evaluados al inicio y al final del año escolar, aunque es demasiado pronto para esperar mejoras sustanciales debido al corto tiempo de ejecución, así como factores contextuales que van más allá del alcance del proyecto. Sin embargo, el análisis de los resultados apunta hacia algunos cambios positivos. Es recomendable que mejoren ciertos aspectos de las bases de datos LLP, que se empleen técnicas de análisis adicionales (como emparejamiento de puntuación de propensión), y que se lleve a cabo análisis adicional a los datos cuantitativos para verificar/identificar las variables que tienen una influencia positiva o negativa sobre destrezas de lectura. El esfuerzo para fortalecer la capacidad mediante el desarrollo de los recursos humanos está demostrando efectos positivos con los maestros y personal del MdeE a nivel departamental. Hay claridad y apreciación de los conocimientos recientemente adquiridos, que está proporcionando las bases teóricas y prácticas para el desarrollo de la alfabetización bilingüe. Son evidentes los cambios incipientes con relación a las actitudes acerca de "coaching" (entrenamiento), sin embargo, es importante consolidar una visión común (entre los asociados encargados de la ejecución, CTA, maestros y personal de oficina central y departamental del MdeE) para saber cómo capturar sistemáticamente los cambios en las prácticas de enseñanza, así como tener los instrumentos correctos de apoyo para el entrenamiento. Para la preparación de los educadores de docentes, el consorcio de universidades ha logrado hasta ahora una buena articulación y trabajo cooperativo. En el futuro, su papel podría ser el de documentar y consolidar la experiencia del maestro educador del programa de desarrollo profesional, así como para servir como órgano consultivo para las universidades y el MdeE en áreas tales como la educación previa al servicio de los docentes, el entrenamiento, la educación intercultural y bilingüe, liderazgo educativo, medición y evaluación educativa y desarrollo de material de apoyo. Existe una clara necesidad de obtener materiales para apoyar la mejora de la alfabetización en español e idiomas mayas; esto incluye guías para los maestros, libros de cuentos, libros grandes, libros de texto o lectores para los estudiantes, entre otros materiales de apoyo. Se han hecho algunos avances en los departamentos para producir materiales (como las antologías) y algún personal departamental quiere continuar este esfuerzo; esto requerirá más apoyo de LLP tanto en aseguramiento de capacidad técnica y que los recursos sean proporcionados por la oficina central del MdeE. Además, mientras que el MdeE ha proporcionado material para las escuelas apoyadas por el gobierno anterior, no está claro el grado en el que están presentes en las escuelas que LLP está apoyando. Es importante contar con un inventario de los materiales que existen en las escuelas y aulas para garantizar que los profesores dispongan de los recursos necesarios para 7 ayudar en forma efectiva a sus estudiantes a desarrollar sus destrezas de lenguaje. Además, los profesores necesitan guías con estrategias didácticas para alfabetización de L1/L2 y para desarrollar sus propias habilidades del idioma maya. El colaborador encargado de implementación de LLP ha reconocido esta necesidad y está recopilando material para un kit del profesor que se entregaría al finalizar su proceso de desarrollo profesional. Adicionalmente, debe ser una prioridad, proporcionarles con los enfoques didácticos y estrategias concretas y lecciones en K’iche’, Mam y español (dependiendo de la necesidad particular). También, es recomendable la participación de los maestros en este proceso en aspectos como hacer que los profesores desarrollen, dirijan y compartan lecciones/estrategias como parte de su desarrollo profesional. Además, es clave proporcionarles herramientas y materiales para desarrollar el idioma maya que deben conocer para la instrucción bilingüe; esto podría efectuarse en coordinación con la ALMG y/u otros especialistas del idioma. Por último, el sub-componente para involucrar a los padres en la educación de sus hijos es complementario para mejorar el aprendizaje de los estudiantes y ha demostrado efectos positivos. Sin embargo, para asegurar que se logren los efectos deseados en todas las áreas objetivo, debe fortalecerse los vínculos entre la calidad y las unidades de educación comunitaria (central y departamental), así como evaluar el grado en que esta intervención está funcionando en las escuelas como se pretendía. El componente LLP para extender las oportunidades para jóvenes que no asisten a la escuela para su educación, capacitación en el trabajo, o para su compromiso cívico está generando efectos positivos y ha demostrado que las lecciones aprendidas pueden ayudar a desarrollar estrategias futuras para este nuevo terreno de apoyo. En general, los resultados revelan que ahora hay más alternativas educativas y de desarrollo laboral disponibles para jóvenes desescolarizados antes de la implementación del LLP. A partir de septiembre de 2016, el LLP ha registrado 5,259 jóvenes desescolarizados. Sin embargo, una proporción relativamente pequeña de la población elegible está realmente matriculada en alternativas educativas formales o en el desarrollo de la fuerza laboral hasta el momento. El primer año de ejecución fue una curva de aprendizaje para los asociados en la implementación debido a realidades imprevistas desconocida antes de la intervención, tales como la dificultad de encontrar a los jóvenes, la falta de datos sobre esta población, la resistencia de los padres a la participación de sus hijos, y los elevados costos de transporte. Se están resolviendo muchos de estos obstáculos, sin embargo, aún quedan varios retos importantes. A continuación los cambios positivos vistos hasta la fecha: 1) Jóvenes expresando cambios muy positivos en sus vidas a partir de su participación en la educación y la formación (como la autoestima, la autoconfianza, la superación de sus miedos, y la capacidad de tratar con adultos) así como avanzar en su educación y en algunos casos encontrar oportunidades de trabajo; 2) La organización de tres grupos para involucrar a la juventud para desarrollar un plan de vida; 3) Fortalecimiento de organizaciones de jóvenes que están incrementando la conciencia de las autoridades locales, comunitarias, municipales y del gobierno central sobre OSY y los problemas que enfrentan; 4) Se estableció una red de cuatro organizaciones muy respetadas que trabajan juntas para proporcionar alternativas de educación básica y capacitación laboral a OSY; 5) Avances en el fortalecimiento de la división del MdeE responsable para atender OYS 8 (DIGEEX), incluyendo el desarrollo de SIREEX para identificar, apoyar y dar seguimiento a los jóvenes; 6) La elaboración de normas de calidad para la educación básica alternativa y desarrollo laboral; y 7) Más compromiso de la juventud con el gobierno, organizaciones no gubernamentales y organizaciones de la comunidad relacionadas a sus necesidades y cómo hacer que se cumplan. Los desafíos que quedan para el colaborador implementador de LLP son: 1) La necesidad de establecer un informe segmentado-cohorte para efectuar el seguimiento de los procesos educacionales y ocupacionales de OSY incluyendo un código de identificación o de barras para cada uno de los jóvenes y el equipo necesario para hacerlo; 2) Ampliar las oportunidades de capacitación laboral para incluir opciones adicionales para satisfacer las necesidades de las demandas de la juventud en cuanto a sus intereses y oportunidades existentes de trabajo locales; 3) Priorizar la participación de los padres (especialmente para los jóvenes que tienen entre 15 y 17 años) mediante el suministro de información sobre los beneficios de la participación, cómo se puede apoyar, y las medidas adoptadas para proteger a sus jóvenes cuando participen; 4) Establecer códigos de conducta y protocolos de seguridad para los jóvenes y el personal que trabaja con los jóvenes para reforzar el comportamiento positivo así como de confianza con los padres; 5) Asegurar que los valores, tradiciones, idiomas y otros elementos de las comunidades indígenas reflejen su cultura y estén presentes en materiales y actividades con la juventud y cuando se involucre a los padres; 6) Crear enlaces con una serie de programas gubernamentales nuevos o existentes dentro de los Ministerios de Economía, Trabajo y Educación, para el intercambio de experiencias centradas en OSY, en lugar de sólo jóvenes escolarizados; 7) Fortalecer aún más el funcionamiento de SIREEX con personal contratado por el proyecto LLP, considerando la posibilidad de trabajar dentro de DIGEEX y 8) Seguir aclarando y consultar a OSY en todas las comunicaciones con autoridades municipales, departamentales y autoridades centrales del gobierno de Guatemala en cuanto a la responsabilidad explícita de OSY. 9 Lifelong Learning Project Midterm Performance Evaluation Report DRAFT I. INTRODUCTION AND BACKGROUND In March 2014, USAID/Guatemala awarded the Lifelong Learning Project (LLP) to Juárez and Associates. The period of performance is from 2014 to 2017 for the base period and from 2017 to 2019 for the option periods. Lifelong Learning’s objective is to improve education quality and increase access for underserved populations (i.e., indigenous children and out-of-school youth). The implementing partner is required to provide technical assistance needed to improve the quality and equity of students’ reading outcomes. Technical assistance is focused on increasing teacher effectiveness; improving classroom learning environments; fostering effective first and second language (Mayan languages and Spanish) acquisition and reading; extending learning opportunities to underserved populations, especially girls and indigenous groups; and expanding parents’, communities’ and stakeholders’ participation in student learning. Reading outcomes are being improved by reaching as many schools as possible through the Ministry of Education (MOE) structure. On the other hand, the project seeks to enable youth in the Western Highlands (WH) region, (specifically in Quetzaltenango, Huehuetenango, San Marcos, Totonicapán, and Quiché) to achieve greater economic self-reliance and higher levels of civic engagement and educational achievement. LLP is working to provide Guatemalan youth with increased education, economic and civic participation opportunities with the aim of increasing stability and economic growth in the WH. Access will be increased for as many out-of-school youth (OSY) as possible. Lifelong Learning project directly supports USAID Education Strategy’s Goals: Goal 1: Improved reading skills for children in primary grades. Goal 2: Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals. Goal 3: Increased equitable access to education in crisis and conflict environments. The project also supports USAID/Guatemala’s Country Development Cooperation Strategy 2012-2017 (CDCS), Development Objective 2: “Improved Levels of Economic Growth and Social Development in the Western Highlands”, Intermediate Result (IR) 2.3: “Education quality and access improved.” Target beneficiaries are the historically underserved and excluded populations in the rural areas of Guatemala’s WH. 10 Lifelong Learning has two components: Component A, Learning to Read and Component B, Education for Employment and Lifelong Learning. Each component has its own set of results and sub-results. (The project is described in more depth in the following section). USAID required a mid-term performance evaluation of the Lifelong Learning Project to adjust as needed the project’s scope in order to meet the intended goals. The mid-term evaluation presented in this report has been carried out through the Monitoring and Evaluation Program contract that was established between USAID and DevTech Systems, Inc. (DevTech) in 2013, which aims to support monitoring and evaluation activities of its Economic Growth Office (EGO). Specifically, the evaluation is part of Result No. 5: “Monitoring and Evaluation Services for other USAID/Guatemala Offices, Conducted Upon Request.” Since the results presented in this mid-term evaluation of the LLP are intended to inform adjustments to the project’s scope in order to meet the intended goals, it is a formative evaluation. Findings regarding the effects generated by LLP in relation to the expected results are discussed, as well as key factors that may be contributing to or interfering with the project’s goals being met. Specifically, the findings identify changes that have thus far resulted from the project’s interventions, how these interventions are working together to accomplish the results, and what effects are a direct result of the project, as opposed to actions carried out by local governmental and non-governmental actors. However, because the context within which the project is implemented has a direct influence on its effects, the evaluation also seeks to gain a deeper understanding of the ways in which other interventions (including policies, national and local initiatives, etc.) are interacting with those carried out by the project. This will help to determine the most effective ways for partner organizations (governmental, non-governmental, and donor organizations) working towards a common goal when possible and ways to adjust in the case of conflicting or interfering elements. This midterm evaluation of the project aims to answer: 1. What are the results of Lifelong Learning’s systemic approach to promote reading skills’ improvements in early grade students? 2. What are the results of Lifelong Learning’s strategy to provide access to alternative basic education opportunities for out-of-school youth? 3. How has the implementing partner adjusted its strategies and approaches according to the evidence and analysis produced by the project? As stated in the third evaluation question, it is essential to analyze how the implementing partner has been adjusting its strategies and approaches in accordance with evidence and analysis resulting from the project. Therefore, the perspective of various participating stakeholders was considered, as well as document and data analysis (as described in the following section). The section of findings and conclusions of this report is organized in accordance to the expected results for each component and subcomponent as well as the evaluation questions. Evaluation question one corresponds directly to LLP component A and the second question directly to LLP 11 component B. The results for evaluation question three are presented by component in section VII (“Adjustments made by implementers and recommendations”), followed by recommendations to generate reflection and appropriate modifications where necessary towards program improvement. I. EVALUATION DESIGN AND METHODOLOGY The mid-term evaluation of LLP employed mixed-methods that included interviews and focus groups with implementing partner organizations, government and non-government stakeholders, and project participants (beneficiaries). It also employed document analysis and quantitative data analysis. This section describes each aspect of the methodology. Data Collection The selection of project stakeholders and locations for data collection was intentional. Key LPP project stakeholders from partner implementing organizations, including governmental and non￾governmental actors, were selected based on their positions of decision-making and/or role in the interventions of LLP’s components A and/or B. Also included in the sample were stakeholders who are directly or indirectly involved with the project’s implementation and/or were able to offer information and views that contribute to answering the evaluation questions through interviews and focus groups (as well as other means). Interview and focus group protocols were elaborated with questions aimed to capture information specific to the interviewees’ role in the project. Selected stakeholders were based in the capital city as well as each of the five departments where LLP operates. These include members of Juárez and Associates, Child Fund, Plan International, Grupo CEIBA, Tierra Nueva, AJKEMAB, ACUDE, FUNDAP, Universities, MOE central officials and staff, MOE Departmental Directors and staff, teachers, parents, members of youth and community groups. Several initial informational meetings with USAID and Juárez and Associates were carried out prior to data collection as well. In total, over 275 informants were included in the sample. (See Appendix 1 for consolidation of data collected). Several other means of collecting data were employed, such as questionnaires with youth participating in Component B interventions. Additionally, three schools were selected to visit based on the criteria that they were are relatively convenient to access and reflect a variation of progress in terms of the project’s expected results and conditions (linguistic make up, presence of other donors, etc.). Implementing organizations and Ministry of Education (MOE) staff recommended the schools. In one case, the visit was unannounced. During each visit several classrooms were observed for roughly 20 minutes, followed by a conversation with the teacher and then the school principal. The visits Juárez were to capture the following information: 1) what type of materials are being used by the teacher and students and displayed on the walls (including in what language), 2) the type of classroom activities taking place and the level of engagement of the kids (including L1 and/or L2 language use), and 3) the content of the students’ notebooks is (e.g. ability and type of writing in Spanish and/or Mayan languages). Talks with the teachers and principals included questions such as support received for implementing the bilingual education model, participation in the professional development courses, perceived changes, and challenges and facilitating factors for implementing the new strategies/approach. Because it was not possible to do systematic school visits, only anecdotal information was collected from the three visited. However, these visits proved valuable by 12 shedding light on issues to further understand (notably, how changes are being tracked and ways this can improve). Data Analysis The evaluation team leader and local team members worked closely throughout the evaluation process. Local team members shared information they collected throughout the period of evaluation; the team leader collected data with local team members while in-country (in the capital city and three departments) and analysis was carried out by the team both in-person and virtually. This ongoing collaborative analysis helped to assure accuracy and comprehensiveness as well as to triangulate data sources to strengthen validity. Findings were organized by evaluation question and consolidated in analysis matrices with the corresponding LLP expected results and sources of evidence from which conclusions were drawn. Documental and data analysis were also central throughout the evaluation. This consisted of reading and synthesizing key information to: 1) identify the specific intervention strategies and their underlying and/or explicit theories of change, 2) compare baseline data with most recent data on learning outcomes (literacy in both languages) and youth education and employment, 3) determine key aspects in which evidence produced by/about project LLP has revealed a need for adjustment, 4) adjustments that have been made and not, and 5) identify information that responds to answering the three evaluation questions. Areas that require more in-depth analysis through empirical data collection were also identified through documental and data analysis. Documentation included all relevant implementer reports such as the partner organization’s contract and implementation plan, work plans, monitoring and evaluation reports, quarterly performance reports, studies relevant to LLP, and documents on the Guatemalan context. Some specific examples of documents include the Community Libraries Assessment Report, National Training, Human Resource Reform Status Report, Student Linguistic Profile Report, and the Participatory Youth Assessment and Electoral Strategy. Data relevant to component A was analyzed to conduct a secondary analysis in order to detect changes in several outputs produced by LLP. Though there was some quantitative data available for component B, the data is not enough to detect changes (in outputs) produced by LLP. (See appendix 2 for more detail). LLP is conducting a longitudinal study of student test scores to measure changes in literacy (L1 and L2) for the duration of the project. However, LLP did not measure the students’ abilities at the end of 2013 nor the beginning of 2014, so no baseline data is available. LLP did assess students’ language (Spanish, K’iche’ and Mam) at the beginning of 2015 (pre-test) and the end of the same year (post-test) for second, third, and fourth grade. In first grade, students decided the language they wanted to be evaluated in the pre-tests. However, in the post-test students were evaluated in Spanish and K’iche’ (or Mam) languages, making it hard to analyze first grade with the same of analysis used for the remaining grades. Teachers surveys for first, second, third and fourth grade (pre and post 2015) were used for analyses as well as a data set identifying teachers that are in the intervened municipalities. 13 To conduct the longitudinal study with student test scores, LLP selected several municipalities with characteristics similar to the intervened municipalities in order to have a control group. The control municipalities were selected using the following criteria: (a) municipalities must be in the same department, (b) the population within the municipality should belong to the same linguistic community, and (c) municipalities should not be faraway, geographically speaking. However, no Propensity Score Matching (PSM) technique was used, which is the method employed by the evaluation team. Thus, the analyses conducted for this evaluation using the longitudinal study databases are not comparable with the analysis conducted by LLP. The PSM analysis employed by the evaluation team was with students (treatment and control group) and with teachers (treatment and control groups) to generate comparable groups (instead of matching municipalities like LLP). Additionally, several standard regression models were fit (with students) and Hierarchical Linear Models (HLM) (with teachers and students), whereas LLP used student growth percentile regression (Castellano & Ho, 2013) to generate students’ growth score percentile. Finally, several latent variables with teachers’ surveys were created by using the Rasch model. Then, standard regression models were used to determine differences between teachers in the control and intervened municipalities. These analyses led to several results, including estimating the percentage of students who improved their abilities for the treatment and control groups. With the PSM technique to match teachers (with treatment and control groups), a set of latent variables was constructed by using the Rasch model. The latent variables were: a) activities at the library, b) reading comprehension activities, c) participation in learning communities, d) teachers’ background, e) teachers’ abilities, f) evaluation methods, g) evaluation tools, h) abilities in Spanish language, i) abilities in Mayan language, j) use of materials, k) parents’ involvement, and l) vocabulary activities. (See appendix 3 for detailed explanation of analyses). The team considers these analyses to robust because: 1) PSM forms equivalent groups (treatment and control) with similar characteristics, 2) the standard regression model used is the Difference and Differences model (Diff&Diff) (Cerulli, 2015; Gertler, Martinez, Premand, Rawlings, & Vermeersch, 2011; Percoco, 2014), which controls for fixed factors and variable factors across time, 3) the analyses was weighted by using the propensity score to control the sample size, and 4) several control variables were included. Additionally, the analyses conducted on teachers controlled for a nested data structure by using hierarchical linear modeling (HLM). One limitation in the analysis is that, due to LLP intervention design, the likelihood of contamination is high.1 This may be the reason that, in many cases, we could not find differences between control and treatment groups. 1 This is due to other donor interventions, amongst other reasons. However, in terms of social outcomes, this contamination is beneficial for the students of the highlands in Guatemala. 14 II. PROBLEM ANALYSIS Since the 1980s, with the support of international cooperation and local efforts, Guatemala has been trying to address the need for intercultural bilingual education (EBI) for a context where about half of the population speaks a language other than Spanish. Many of these efforts have been led by programs implemented with funds from USAID as well as other donor agencies (European and other international agencies/donors). The context is complicated in a country that has remained highly polarized (pre-conflict and post-conflict) due to the significant and persistent differences (cultural, economic, and political). One of these issues is education. Most interventions have faced opposing viewpoints such as those with the interest to preserve and strengthen the native language and others who believe Spanish should be the national language of instruction (including some who aim to include English or another foreign language). Over the past four decades, there have been important institutional advances, both in legislation on the subject, to help ensure access to bilingual education for indigenous children and strengthen bilingual education in general. Such advances include the creation of institutions such as the Directorate of Bilingual and Intercultural Education (DIGEBI), the Vice Ministry of EBI, the Academia de Lenguas Mayas de Guatemala (ALMG) and others arising from the Peace Accords or as a result of pressures from domestic and international social movements. Examples of these include the Agreement on the Identity and Rights of Indigenous Peoples signed in 19952 and Guatemala’s ratification of the International Labor Organization’s Convention 169 concerning Indigenous and Tribal Peoples in 1996. At present, after almost 40 years of incipient progress, there is some recognition and discussion by various sectors and institutions on the issue as a human right and educational priority for the country. However, according to the Guatemala Human Rights Commission (Fact Sheet 2010), in 2005, of the 7,832 schools in places with significant indigenous populations, only 1,869 provided bilingual education (23.9%). Another source reports that (in 2005) only 19% of primary school students had access to bilingual schools and roughly 66% of indigenous primary students are in classrooms with teachers who only speak Spanish.3 A more recent source (2016) reports that indigenous children between the ages of 7-12, 4.8% receive instruction in a Mayan language, 29.4% receive instruction with a bilingual model, and 65.8% receive instruction only in Spanish (Mayan language and Spanish).4 Regardless of the percentages/numbers reported by different sources, it is evident that continued efforts are needed 2 Agreement on Identity and Rights of Indigenous Peoples, Mar. 31, 1995, Guat.-URNG, UN Doc. A/49/882--S/1995/256, 36 ILM 285 (1997), available at http://www.usip.org/library/pa.html (last visited Jan. 24, 2007). 3 Barr, M., Barth, E., Becker, H., Falk, B., Geisen, E., Giese, M., ... & Lennon, M. Members of the 2011 AGBL Guatemala team, citing Increasing Education Access, Quality, and Equality in Guatemala. Rep. United States Agency for International Development. Web. 18 May 2011. . 4 Educación Bilingüe e Intercultural en Guatemala: Avances, tendencias, brechas, y retos en terminus de inversion. Asociación Grupo de Análisis Estratégico para el Desarollo (GAED) para Save the Children. Abril de 2016, Guatemala. 15 to create access to bilingual education (especially for monolingual and bilingual indigenous children). The educational reform under the Peace Accords, specifically with regard to the National Curriculum Base (CNB), stipulates the need to contextualize its content and approach so that its relevance is ensured with the linguistic and cultural characteristics of the communities and villages that make up the country. However, at present it is evident that there is still no consensus about the EBI model as to what approach/methodology and specific interventions are responsive, effective, and feasible. At least three ways to raise the approach strategy are identified: 1) those who insist on an approach that strengthens the Spanish language or other foreign language (e.g. English) with the danger of a gradual extinction of the languages of the original towns, 2) those who insist on the EBI approach that starts with the L1 (mother tongue) in the first three grades then transferring to the Spanish language from fourth grade on (assuming the native language was Mayan), and 3) those who advocate for a fully bilingual model or approach that incorporates all the cultural elements of the mother tongue as the expression of life, knowledge and political and social organization of indigenous peoples (a model that may involve, among other things, political autonomy of indigenous peoples as well as the organization of educational and technical institutions for an indigenous education system). The model proposed and implemented by USAID responds to the second approach, which seeks to build on the native language to achieve bilingualism and biliteracy within bilingual and intercultural contexts. Thus, emphasis is placed on key interventions that can ensure quality service such as strengthening the capacity of teachers, supervisors and specialists from the Ministry of Education and the provision of materials to support literacy in both languages. Additionally, importance is placed on monitoring and performance evaluation of methodological approaches, using action research to improve teaching and learning, and effective participation of parents and mothers. While these strategies are technically valid, the intervention strategies are within a context that presents various challenges such as the recurrent institutional weaknesses of the State (lack of financial, human and material resources) and circumstances of the students and parents, such as poverty and extreme poverty, lack of employment opportunities, discrimination, and exclusion. A very recent example is evidence data Report on Human Development by UNDP Guatemala (Sept. 2016), in which as a result of a recent survey, education does not appear as part of the first four priorities of Guatemalans, rather citizens are concerned about this issue above others such as health, work, income, housing and insecurity. Additionally, the educational reality shows that despite the efforts and marked progress, there are still weaknesses in the implementation of CNB, institutionalization and implementation of an adequate system of training of educational resources and the system of educational support needed (despite having its definition, strategy and operational plan developed with technical support from USAID programs). Other obstacles are the lack of a formal position of School Director, weaknesses in initial teacher training (FID), and logistical deficiencies of support programs (such as school snack, textbooks, supplies, scholarships and educational bag for teachers). Furthermore, there is low coverage in pre-primary and secondary as well as ongoing poor performance indicators in standardized tests of language and mathematics. These factors 16 become more complex when addressing EBI in certain regions of the country with a majority indigenous population due to the multiple languages (24 of which are officially recognized) and schools that are comprised of children who are either monolingual in their Mayan language or Spanish or have different degrees of language development in each or both languages (all in one classroom). Additionally, the percentage of teachers who can teach in Mayan languages has been extremely limited as has the capacity of teachers to address the linguistic complexity of attending to a group of students who often have different primary languages. Against this background, certain experts agree that some international cooperation programs, due to the constant changes of authorities, end up adjusting their strategies to the context, suddenly abandoning systems and processes that required more time for consolidation and institutionalization. For example, in recent years there has been an increase in the number of over-age young people in the system or young people of school age who are outside the education system, which is estimated to be roughly 850,000.5 This partly due to the socioeconomic context of their families, but in other cases, due to the low quality of educational offerings. These phenomena in recent years have increased with the number of children who began to migrate to the U.S. The aforementioned situation of out-of-school youth (OSY), limited educational and work opportunities for youth, and increased migration have heightened donor attention to the need to address the situation. Thus, LLP has included a component with this aim, while the component to strengthen EBI seeks to give continuity to previous interventions. This has resulted in the coexistence of "two projects" in one, though the components are denominated A and B. However, there is no evidence of a clear definition of how each component’s strategies intertwine to address the issues the intend to: a school system that virtually expels minors without offering opportunities, and a socio-political system that fails to offer primary education alternatives to families so that their children access to quality education that is linguistically and culturally relevant. Component B of LLP seeks to prevent rural to urban migration and decrease the likelihood of youth engaging in crime or other illicit activities in a targeted area of five departments in the Western Highlands (specifically, in Quetzaltenango, Huehuetenango, San Marcos, Totonicapán, and El Quiché). Lifelong Learning generally seeks to address these problems by improving education quality and access, as well as workforce training and access, for underserved youth. 5 While there is no one source that estimates the number of OSY in the targeted age range of 15-24, a review of three sources leads to this estimate. See Boletín No. 5 del Instituto Centroamericano de Estudios Fiscales ICEFI. ¡CONTAMOS! ¿En dónde están los ausentes? Un análisis sobre los mecanismos, costos y formas de financiamiento para eliminar la exclusión educativa en Guatemala. 2012 - 2021. P. 33 available at http://icefi.org/sites/default/files/contamos-5.pdf; Encuesta Nacional de Condiciones de Vida y Población Joven. ENCOVI 2011. Editado por el Instituto Nacional de Estadística INE y el Programa Juventud de la Unión Europea. Noviembre, 2011 at p. 10; Informe Anual de la Política de Desarrollo Social y Población 2015. Secretaría de Planificación y Programación de la Presidencia SEGEPLAN, Diciembre 2015 pp. 140-141. 17 Out-of-school youth are an invisible majority in rural Guatemala. The high dropout rates in this region have not been adequately addressed in the past by the Guatemalan government. In particular, the Guatemalan Ministry of Education has not been able to wrap its arms around the problem of out-of-school youth, and LLP seeks to facilitate the GOG’s ability to track and assist this population. The target beneficiaries identified in the Scope of Work are all youth ages 15-24 in the historically underserved and excluded populations in the rural areas of the Western Highlands. When possible, special attention is supposed to be given to girls living in the region and targeted by Feed the Future and the Global Health Initiative. The targeted communities, which are inhabited predominantly by indigenous Mayan people, tend to have high rates of poverty, extreme poverty and low rates of educational attainment. Many youth in the region speak both Spanish and their mother language(s), and others speak only their mother language(s). Low coverage rates in secondary education are a product of severe quality challenges in primary education. Very limited paid work opportunities exist in the targeted communities because there are no enterprises and formal economy in the region. The theory of change of LLP is that if Guatemalan youth receive relevant education, economic and civic participation opportunities (that is, jobs, participation in the community, and enrollment in further education), they will contribute positively and productively to their local communities and their country. Thus, component B of LPP aims to increase access to education and workforce training for out-of-school youth through several interventions including extending flexible learning opportunities to underserved populations (especially girls and indigenous groups), helping to establish a network of NGOs as well as public and private youth-serving providers and strengthen their technical capacity to serve youth. The project seeks to help youth achieve greater economic self-reliance and higher levels of civic engagement. Civic engagement is a growth area because there is a relatively low rate of civic participation in some areas, which is thought to contribute to the problems of out-migration and crime. Additionally, the project seeks to place the issue of OSY on the public agenda as a policy priority, which is assumed to fundamental for addressing the afore-mentioned issues. The intervention logic asserted by the theory of change for Component B assumed there would be strong demand for alternate basic education and workforce development, but those assumptions have not matched reality. There are multiple obstacles faced by youth such as the opportunity cost of not working (most often in the informal sector), the cost of transportation, and the lack of willingness and support of parents for their children to take advantage of the opportunities (alternative basic education, workforce training, civic engagement activities, etc.). Furthermore, attending to OSY is a new area for USAID and most organizations in Guatemala and there are few international experiences that have demonstrated results of doing this successfully. Additionally, there is a lack of accurate information on this population. Thus, the project interventions for this component, unlike component A, did not have a strong basis for a sound theory of change and instead must adjust and learn throughout implementation. 18 III. DESCRIPTION OF LIFELONG LEARNING Objectives of this program are to foster pertinent learning environments by providing relevant Opportunities to Learn (OTL) for children and out-of-school youth (OSY). As reading is the foundation of lifelong learning, investments in early grade reading have a high return rate. More of a third of Guatemalan children are failing first grade, so interventions in early grade reading in bilingual contexts (Mayan languages and Spanish) is a pressing need for the development of the country. An intervention in early grades will reduce dropout out and repetition, and completion rates in primary and enrollment rates in secondary education will increase. Under this intervention, the Contractor proposed and is implementing activities that build upon successful education interventions to date to improve reading skills in the early grades, with special focus on five WH departments. This intervention complements and advances USAID/Guatemala’s implementation of the Education Reform in the Classroom project (REAULA 2009 - 2014 as well as the Education and Standards Research Program (ESRP 2005-2009) and previous programs of the USAID/Guatemala Better Educated Rural Society Strategy (1999-2004) to enhance equitable access to quality intercultural bilingual education. It also builds upon gains made in the last decade in the areas of curriculum policy, teacher training, applied research, monitoring and evaluation, information technology, improved classroom management as well as systems to promote transparency and accountability through the public education apparatus. This program also targets youth education and employment, a new area of intervention for the education sector of USAID/Guatemala. This intervention maintains emphasis on the critical need to develop basic literacy and numeracy as foundational skills, but targets a different group, i.e. OSY in selected rural areas, and adds a focus on the need to provide them with workforce skills. GOG is not responding to an important bulge of young population with increasing opportunity costs and without education opportunities. Education options will be tailored to different pathways, local social capital, and labor market assessments. The provision of education and workforce opportunities for youth is a conflict- and violence-prevention measure. This intervention will also provide valuable feedback and a learning agenda to GOG plans and interventions targeting OSY. A key strategic document underpinning this intervention is the USAID “Youth in Development Policy” (October 2012). USAID will provide opportunities to OSY in selected rural areas as one way of preventing rural to urban migration and the likelihood of youth engaging in crime or other illicit activities, building workforce skills and increasing access to sustainable livelihood pathways. The theory of change is that if Guatemalan youth receive relevant education, economic and civic participation opportunities (defined by jobs, participation in the community and enrollment in further education), this will enable them to contribute positively and productively to society. Target beneficiaries are female and male youth, ages 15-24, in the historically underserved and excluded populations in the rural areas of Guatemala’s WH. When feasible, special attention will be given to girls living in this region and targeted by Feed the Future (FTF) and the Global Health Initiative (GHI). Interventions will strengthen existing options for out-of-school populations, scale up innovative and best practices, and create new education, work-readiness, and livelihood alternatives for youth. Developing effective strategies geared towards functional literacy and numeracy, life-skills, workforce readiness, reproductive health, effective parenting, 19 and job skills training will increase opportunities for positive participation in society for rural youth. Ensuring that youth are engaged in positive learning activities has been promoted as the key to achieving positive youth outcomes in challenging environments. Youth who are connected to school are half more likely to engage in violent activity than those who are not connected to school. The program explores current youth education and employment activities in the target zones in the design of the project, as well as considers activities in other sectors that may provide opportunities for youth education and employment such as health, environment, agriculture, and economic development. The Contractor is researching these efforts and exploring ways to carry on the dialogue with the MOE about youth education beyond primary school for sustainability. The Contractor expects the following results in pursuit of the above objectives: IV. EXPECTED RESULTS Component A: Learning to Read Sub-Component A.1: Implementation of improved approaches to early grade reading instruction and development, distribution and use of supportive reading materials. Sub-Component A.1 seeks to achieve the following results: Result A.1.1: The national standards-based model for intercultural and bilingual education (EBI Model) focusing on bi-literacy for Pre-Primary to Grade 3 is being successfully implemented in target schools. Requirement A.1.1: The Contractor shall present for USAID approval an EBI Model Implementation Strategy at Classroom Level showing how it will implement a standards-based intercultural and bilingual education model at the classroom level, focusing on language and reading development and including monitoring and assessment of gender-disaggregated student learning outcomes Sub-result A.1.1.2: The MOE has developed implementation guidance for adoption of at least one set of Intercultural and Bilingual Education best practices in bilingual areas. Requirement A.1.1.2.: Support the MOE in the development of EBI Operational Guidance and Standards. Result A.1.2: Student reading progress in target zones is being tracked using mechanism(s) to measure reading fluency and comprehension in early grades. Requirement A.1.2: The Contractor shall apply validated performance testing instruments (e.g., affordable and reliable testing instruments) in randomly selected testing zones (including program and non-program schools) with a random sample of students to assess student reading outcomes and the impact of implementation of the EBI model under this contract. Progress on 20 this requirement shall be included in the Monitoring and Evaluation Plan (M&E Plan) and through the reporting on Secondary Analysis and Results Tracking (SART). Result A.1.3: Literacy materials for L1 and L2 for children and teachers (pre-primary through third grade), including teacher guides, text books, and supplementary materials, analyzed, revised, developed, validated and in use in classrooms. Requirement A.1.3.1: The Contractor shall submit a Materials Development and Validation Plan. Requirement A.1.3.2: The Contractor shall assess the USAID’s support of community libraries. Result A.1.4: Implementation of the National Textbook and Educational Materials Policy. Requirement A.1.4: The Contractor shall submit to USAID for approval a National Textbook and Educational Materials Policy Status Report and a Technical Assistance Plan that describes the current status of the adoption of this policy and lays out a plan for implementation. Sub-Component A.2: Human resource development in the education system leading to improved learning and reading achievement Sub-Component A.2 seeks to achieve the following results: Result A.2.1: Higher education degree programs to prepare a cadre of Teacher Professional Development experts developed in cooperation with local universities and the MOE are being implemented and subsequently evaluated for impact. Requirement A.2.1: The Contractor shall submit for USAID approval an Evaluation Plan for USAID-supported Training that describes an approach, timeline, and costs for conducting a study to determine whether a representative sample of graduates of the various programs described above are putting the knowledge they have learned into practice in their professional activities and if so, how. Result A.2.2: A cadre of Education Development Experts is trained and functioning in the education system to support classroom-level improvements. Requirement A.2.2: The Contractor shall select competitively no less than three cohorts of eight members each to be trained through the Frank Fairchild Scholarship Program and no less than 150 MOE local staff for Educational Leadership Programs in departments where the Contractor shall work. If feasible, at least 50% of the candidates will be Mayan speakers and at least 50% will be women. Result A.2.3: Education Human Resources Professional Development System being implemented and continually updated. 21 Requirement A.2.3: The Contractor shall submit for USAID approval a copy of the most current SINAFORHE Strategy and Implementation Plan being used in the MOE that identifies areas for Component A collaboration and technical support. Result A.2.4: Groups leading efforts to reform human resource policies receive technical support so that these reforms can be adopted and put into practice in Guatemala. Requirement A.2.4: The Contractor shall design an approach for a Human Resource Reform Status Report and Technical Assistance Plan that describes the current status of the adoption of the mentioned human resource reforms that received USAID support. Sub-Component A.3: Parental and community participation in education leading to improved learning and reading achievement. Result A.3.1: Increased parental and community involvement in supporting the education of primary school children, with a focus on reading improvement. Result A.3.2: Public awareness of and support for critical education themes increased. Requirement 3.2: The Contractor shall present for USAID approval an Early Reading Socialization Approach. Result A.3.3: Policy and programmatic decisions made based on results from quality monitoring system and applied research studies. Component B: Education for Employment and Lifelong Learning The goal of Component B is to enable youth in the WH to achieve greater economic self-reliance and higher levels of civic engagement and educational achievement. Sub-Component B.1: Equitable access to alternative basic education and workforce training for OSY Result B.1.1: Network of NGO, private and public youth-serving providers and stakeholders established and active in collaboration and coordination of services. Result B.1.2: An outreach, tracking, and referral/transition system established to identify and meet youth needs for targeted services and supports. Result B.1.3: Barriers to equitable access to alternative basic education (education relevance, gender discrimination, transportation, clothing, materials, and food security) of targeted students are reduced or eliminated. Requirement B.1.1: The Contractor shall conduct a Participatory Youth Assessment (PYA) that both engages key stakeholder groups and provides critical information to inform USAID project design and implementation in concert with local actors. At least, the following broad lines of inquiry will be covered: 22 a) Building on USAID’s Education Assessment findings, the Contractor shall enquire on current conditions of the OSY cohort segmented by sex, age, socio-economic status, and ethnicity in the WH, with respect to employment, education, and civic engagement. Where possible, these data should be benchmarked to other regions of Guatemala and the Central American region. b) Labor market/demand side c) Basic education provision d) Workforce skills provision e) Sexual and Reproductive health f) Youth civic engagement g) OSY strengths and buffering factors h) Description of youth’s involvement in illicit activities within specific municipalities i) Gender-based and domestic violence issues affecting youth access to education, training and employment j) Parental and community involvement in youth’s decisions k) Policy and institutional landscape Requirement B.1.2: The Contractor shall develop a Partner and Stakeholder Action Plan Sub-Component B.2: Improved quality of alternative basic education and workforce training Result B.2.1: Shared quality standards developed for alternative basic education and workforce training and adopted by service providers Result B.2.2: Improved organizational and institutional capacity of youth serving organizations. Result B.2.3: Improved technical capacity of youth service providers Result B.2.4: System of monitoring and learning and program improvement established and used Requirement B.2.1: The Contractor shall create Alternative Basic Education and Workforce Training Program Quality Requirements Requirement B.2.2: The Contractor shall include as an annex of the first annual report a Plan describing how systems are developed to train, support, and monitor youth service providers. Requirement B.2.3: The Contractor shall submit a Monitoring and Evaluation Plan for supported youth- serving organizations as an annex of the first annual report. Sub-Component B.3: Youth engagement The USAID Youth in Development Policy 2012 aims to mainstream meaningful youth participation throughout USAID programming, especially youth programming 23 Result B.3.1: Youth-serving organizations mainstream youth participation Result B.3.2: Youth engaged in mobilization and policy dialogue for equitable access Result B.3.3: Increased opportunities for meaningful youth civic engagement and service in their communities Requirement B.3.1: The Contractor shall provide a description of Youth Civic Participation Implementation as an annex of the first annual report Requirement B.3.2: The Contractor shall provide a description on Youth Progress on Equity and Inclusion as an annex of the second annual report Requirement B.3.3: The Contractor shall provide a description of Youth Engagement Implementation as an annex of the second annual report V. FINDINGS AND CONCLUSIONS This section presents the findings of the mid-term evaluation in regards to each evaluation question. The main reference points for the analysis are the expected results for the project, as well as the proposed strategies and interventions to achieve these results. Because this is a formative evaluation, the analysis and conclusions are centered on the effectiveness of the project’s implementation since it initiated (in March 2014) how the various approaches and strategies have contributed to or deterred from achieving the results expected up to this time period. The analysis is based on evidence derived from information and data in project-related documents, artifacts, databases, and information collected from implementers, beneficiaries, and other project stakeholders. Conclusions drawn from this analysis provide a basis from which to recommend ways of adjusting interventions for the remaining period of implementation in order to achieve the maximum benefit possible within the project’s capacity. a. Learning to Read This section aims to answer the first evaluation question: What are the results of Lifelong Learning’s systemic approach to promote reading skills’ improvements in early grade students? 1. Implementation of improved approaches to early grade reading instruction and materials As described in the previous section, the project seeks to improve reading skills of children in kindergarten through third grade through helping teachers develop better approaches to early grade instruction and ensuring that teachers have and are using reading materials to support this. It seeks to do this through the helping to ensure that the national standards-based model for intercultural and bilingual education focusing on bi-literacy for Pre-Primary to grade 3 is being successfully implemented in target schools. It is expected that the implementer will provide 24 guidance to the MOE on how to adopt at least one set of Intercultural and Bilingual Education best practices in bilingual areas. Successful implementation of national standards-based EBI model focusing on biliteracy for pre-primary to grade 3 in target schools and development of implementation guidelines for EBI best practices It was clarified by the implementing partner (Juárez and Associates) that the concept and approach of the model to strengthen literacy development in pre-primary through third grade in bilingual and intercultural contexts supported by LLP does not strictly correspond to the Bilingual and Intercultrural Education (EBI) model of the Ministry of Education. Rather, it seeks to strenthen specific aspects of the MOE model (e.g. bilingual literacy development in the early grades, participation of parents, assessment, etc.). The EBI model of the MOE, which various organizations (local and international) have sought to strengthen over the years, is a broader model which encompasses more languages and is not restricted solely to literacy (i.e. may include other disciplines such as math and science). These are not conflicting models; the model supported by LLP is not restrictive in that it aims to improve biliteracy development through strengthening the capacity of teachers and MOE staff to implement bilingual instruction but does not exclude other approaches (e.g. bilingual instruction also in math, use of additional literacy development strategies, etc.). The EBI model supported by LLP includes various interventions including 1) more time dedicated to teaching reading, 2) better techniques for teaching reading, 3) more texts in the hands of students, 4) teaching in mother-tongue (respond to linguistic reality), 5) testing learners reading progress. Thus it is expected that, after two years of project implementation, each of these interventions should be observable to some degree in the five departments and municipalities targeted by the program, as well as at least incipient changes in the target schools. One key strategy employed by the evaluation team was to examine the degree to which the overall model was being perceived and implemented in regards to its congruency with the stakeholders’ understanding of the model, their vision and goals, and the degree to which the capacity and conditions exist for achieving the desired outcomes. Overall, the model that the project aims to support through its various interventions is congruent with the MOE Strategic Plan.6 The great majority of stakeholders/informants for this mid-term evaluation perceive that the overall approach of strengthening the ability of teachers to teach children reading and writing in their primary language and second language, adapting the strategies to each linguistic reality, is the right way to help improve their learning and academic outcomes. There is also agreement that the linguistic assessment is a helpful tool for choosing the appropriate model for each classroom, and that increased reading time and reading materials are important. It is recognized that the project provides opportunities to strengthen the weaknesses in the education system and that it is practical in that it can respond to each local context, aims to involve family and community members in children’s educational process, and provides 6 Plan Estratégico de Educación 2016-2020, Ministerio de Educación, 2016. 25 methodologies and tools for supporting teachers in language instruction. The Deputy Technical Director of one of the departmental directorates (DIDEDUC) summarized his project experience and contributions in the following terms: "It is an excellent project because it takes into consideration all aspects conducive towards students´ learning, since our context is mostly bilingual". (“Es un proyecto excelente porque contempla todos los aspectos que propician mejores aprendizajes de los alumnos, dado que nuestro contexto es mayoritariamente bilingüe”). Yet, even though stakeholders perceive each of these interventions as important, it was also apparent that very few stakeholders in the departmental offices as well as teachers and parents had a comprehensive view of the overall model. In other words, when discussing the model, each actor in the process addressed actions taken but without linking them to other subcomponents of the model. The tendency is to describe what is done, but the strategies to achieve it are not highlighted much. One of the challenges is the coordination of the various subcomponents and interventions of LLP so that it reflects having a strategic direction, since some actors feel they are nothing else than mere executors of partial elements of the model. While it wasn’t evident that there was a clear understanding of the overall model, MOE departmental staff (including CTAs) did express clarity and positive views regarding their newly acquired knowledge about elements such as their role, approaches to coaching, and bilingual literacy development. Also mentioned was that there seems to be a change in attitude both by the teacher and CTA regarding coaching and instructional support (i.e. a shift from supervisor visits to control or merely check off items on a list to being an authentic support via coaching). Though this points to nascent likely occurring as a result of the project, it is not possible to generalize (i.e. this is occurring with the majority of CTAs and teachers) and also premature to conclude that such changes in practices and attitudes will be sustainable beyond the duration of the project. For the remainder of the project, there must be a strong emphasis in assuring that MOE department staff and teachers take ownership over these new approaches and share a common vision of how to provide instructional support systematically. Furthermore, adequate resources and conditions must be in place for this to happen. There was overall acknowledgement that to be able to effectively provide this type of support to all of the schools was not possible with the many responsibilities assigned to CTAs combined with their large number of schools to cover. This will be further discussed in the section on human resource development. Some of the elements mentioned the most frequently as both perceived positive changes as well as continuing obstacles were: a) Attitudes about children learning in the Mayan language, b) teachers who are bilingual and biliterate in the Mayan language, c) teachers who effectively implement the new strategies, d) capacity of department MOE staff to provide adequate support to teachers, and e) local language materials in the classroom. Attitudes about children learning in the Mayan language In terms of attitudes towards children learning in their primary language, many parents commented on how they are glad that their children are learning in Mam/K’iche’ because they can understand and learn more at school that way (if first taught in a language they understand). Some parents also talked about the importance of preserving their language. 26 MOE Dept. staff as well as implementing personnel pointed to resistance on the part of certain groups as well as parents to children learning in their Mayan language (even when it is their mother-tongue language. This is more pronounced in departments that are on the border with Mexico (San Marcos, Huehuetenango, and El Quiché) due to the patterns of migration and process of acculturation (such as adopting English idioms and not seeing the Mayan language as relevant). As expressed by one informant, "The program is dreamlike, it is acceptable to seek to raise the quality of education in a bilingual and intercultural context, but to talk about EBI is a also a very broad and complicated issue, since there has been a lengthy acculturation process and that has confused those of us that teach." (“Es un proyecto excelente porque contempla todos los aspectos que propician mejores aprendizajes de los alumnos, dado que nuestro contexto es mayoritariamente bilingüe”). Sensitizing parents and society in general regarding the importance of children learning in their primary language and the benefits of bilingualism is something that must be continually worked on for several reasons: 1) according to the law as well as the MOE official policy, children have the right to learn in their mother-tongue language, 2) research and evidence from practice demonstrates that this improves L2 acquisition and overall improved academic outcomes when done effectively.7 Additionally, there are emerging practices such as translanguaging that recognize and embrace language practices of bilingual/multilingual children who access different linguistic features or languages to maximize communicative potential.8 While children learn each autonomous language, their language practices may include a mixture of languages and idioms that reflect their sense-making process as they draw upon their rich linguistic repertoires. This is also referred to as dynamic bilingualism and now promoted by leading experts in the field (such as Cummins’ newer work).9 There are concrete pedagogical approaches and strategies that could be considered for encouraging and supporting the use of translanguaging practices when appropriate. This is especially promising in an environment with such a high number of bilingual children, such as in various target municipalities.10 It is important to note that there appears to be different visions regarding how the intercultural and bilingual model should be implemented in regards to its approach to language and culture. The majority of respondents made little mention of bilingual education as a cultural expression but rather the focus was on (bi)literacy development. However, some personnel in the MOE Department offices and teachers expressed the importance of culture as central for language development. This opinion was most explicitly stated by the Vice Minister of Bilingual Education who summed it up as “la cultura debe ser la fuente del aprendizaje” (“culture should be the source of learning”). While he acknowledges that the project is strengthening the 7 Thomas & Collier, 1997; Cummins, 1976, 1991; August, Shanahan, & Shanahan, 2006 8 Garcia, O., Johnson, S.I., & Seltzer, K. (2017). The Translanguaging Classroom: Leveraging Student Bilingualism for Learning. Carlson, Inc. 9 Garcia, O., & Sylvan, C. (2011). Pedagogies and Practices in Multilingual Classrooms: Singularities in Pluralities. The Modern Language Journal, 95(3): pp. 385-400. See also Cummins, J. (2008). Teaching for transfer: Challenging the two solitudes assumption in bilingual education (pp. 1528-1538). Springer US. 10 Perfil Lingüístico de Estudiantes 2015. USAID Leer y Aprender. 27 technical ability to provide education, he believes that language is being treated as something separate from culture rather than culture being transmitted through language. In other words, his view is that the cultural part is missing from the bilingual education model being supported by the project. He believes that bilingual education is a social and political process and would like the project to help strengthen that. For him it is important to document bilingual experiences that have cultural as central to language learning, or stem from a sociocultural vision. The project is not restrictive in terms of this vision being strengthened; the emphasis on local capacity building for understanding teaching approaches with pedagogical principals (instead of focusing only on teaching strategies) and developing local and contextualized materials may provide the opportunity to adapt the model according to the desired vision. Bilingual and biliterate teachers Though no recent study or documentation on the national status of teachers’ linguistic capacities seem to exist, there is a wide recognition of the lack of teachers equipped to teach in the Mayan languages (i.e. a lack of bilingual and biliterate teachers needed for the proposed model, even in the five targeted departments). Strategies have emerged in response to this. For example, some teachers enrolled in courses with the Academia de Lenguas Mayas de Guatemala (ALMG) to develop their Mam/K’iche’ language capacity and some learned through the professional development they received as well as seeking out resources on their own. A strategy used by CTAs and school principals has been to assign to the lowest grades the teachers who are able to teach in Mam/K’iche’. While there is no immediate solution to this limitation, efforts such as the aforementioned can be supported and expanded as well as working closely with the ALMG to devise structural approaches towards improving this situation. Teachers who effectively implement the new strategies and capacity of MOE to provide them adequate support Implementers and stakeholders (central and departmental level as well as teachers and parents) expressed that the CTAs are providing technical support (coaching) in the target schools, which has been helpful to teachers. The COC (conversation-observation-conversation) approach has been well received by DIDEDUC directors, CTAs, and teachers. In some cases, innovative practices have been noted at the classroom level and in the educational support strategies as a result the action-research requirement of the CTAs in their professional development courses. Children attending school more and deserting less was also reported by stakeholders, and this was attributed to the new methodological approaches that are more engaging. This was evidenced in one school visited; first-grade children were highly engaged and participative in both languages (which were taught in two separate moments) and writing in both languages was in small paragraphs in their notebooks and decorated the walls. Another positive element mentioned by stakeholders are parents learning to read and write from their children, some of who are motivated to continue learning (informally or through an adult education program). While this anecdotal information is promising, it is not generalizable to all target school areas or systematically documented. Capturing these positive changes, whether initial project goals or unintended positive effects, is important. LLP, through the professional development provided by the universities, CTAs and other MOE staff received training in how to use instruments for coaching. These instruments guide the 28 coaching (COC) process and can capture changes in teacher practice (as well as CTA practices. One example is an instrument for pedagogical accompaniment in the classroom developed by in by the Departmental MOE in El Quiché (which may also be). While the CTAs and other MOE staff in all five departments received training in the use of this instrument, evidence of its use (including informants mentioning it) was only evident in El Quiché. In this department, it was observed that CTAs who participated in the professional development are beginning to apply this instrument when they visit schools/teachers. They also expressed enthusiasm about it, as did other departmental MOE staff in El Quiché. Though still in a pilot stage and in need of validation and improvement, it is a promising tool for supporting the coaching process and potentially to keep track of changes in teacher and CTA practices. However, there is a lack of a shared vision and clarity regarding the purpose of the instrument. According to respondents such as MOE sub-directors and directors, it is a tool to make sure the CTAs are carrying out the required visits and also a means of having information on the status of school/teaching quality. However, the implementing organization visualizes this as a way to reinforce a change in paradigm and practice of CTAs and teachers from one of control and supervision to one of coaching, support, and learning for teachers. It is unclear who it will be used by and how it will serve as a practical tool for decision-making by CTAs and others. It is also unclear if all CTAs that participated in the professional development (in all departments) are using it and the degree to which it is standardized (in terms of how the CTA and teacher use it as well as the MOE departmental directors and other staff). If such an instrument is going to be adopted and institutionalized by departmental MOEs and become sustainable, it is important that there is a common vision of its purpose and use, especially as SINAE begins to be implemented. Additionally, the linguistic profile applied by the teachers to identify students’ literacy level at the beginning of the school year (in pre-primary and first grade) has proved valuable for determining the right model to use in the classroom, know each child’s individual linguistic abilities, and as a professional development tool for the teacher. While there are standards that indicate the language abilities that students should develop during the school year (including for the early grades), there is not a similar instrument to for the teachers to assess the progress of the students over the year for grades two and three nor guidelines for this to be done in pre-primary and first grade at the end of the school year (and potentially mid-year as well). This is important in order to capture individual child progress and adapt instruction accordingly. It is also a valuable formative evaluation tool and feeds into the professional development teachers receive(d). Finally, it was expressed at the departmental and central MOE levels that the departmental MOE offices are less dependent on receiving guidelines from the central MOE to carry out the Bilingual Education model in a specific way. This presents both opportunities and challenges. The central DIGEBI office expressed a concern that too much decision-making may have been delegated to the departments and that there is a lack of communication with the central office. It is unclear which specific aspects this refers to. In other words, if the aim is to develop a national intercultural and bilingual education model there may be a need to have certain standardized guidelines and practices (e.g. to capture data and best practices, strengthen pedagogical approaches, etc.) while other elements of the model can be left to the discretion of each department. The Vice Minister of Bilingual Education expressed that he has been reaching out to 29 departmental MOE directors and regional groups in order to strengthen communication, which is a positive move in this direction. An existing challenging is the requirement of defining strategies for the expansion of socio-linguistic models for other regions of the country along with other institutional entities that have such a legal mandate. Juárez expressed the importance of continuing to work towards improving literacy (L2 and/or L1) through the implementation of the intercultural and bilingual education model supported by LLP in the areas of intervention as well as continuing to work at a central and national level to strengthen the MOE’s capacity for implementing EBI at a national level and increasing public awareness regarding its importance. Tracking student reading progress In 2014, Juárez implemented a mechanism to track students’ progress in reading and writing in Spanish and Mam/K’iche’. This included the elaboration and validation of standardized tests with the plan to apply them to students in grades 1-3 in target schools in 2015, 2016, and 2017 (including 4th graders for 2015). In 2015 students were evaluated at the beginning and end of the school year. It is too early to expect substantial improvements in literacy due to the short time of implementation as well as contextual factors that are beyond the scope or have a direct effect on the project, including that much is the sole responsibility of the MOE. However, the analysis of the results from the 2015 longitudinal study does point to some changes. For first grade, a statistically significant difference in reading abilities in Spanish (β3 = .154, p￾value = 0.185) was not found. However, in K’iche’ language (β3 = .595, p-value = < .001) and Mam (β3 = 1.421, p-value < .001) students in the intervened municipalities show higher abilities than students in the control municipalities. This suggests that students in the control and intervened municipalities have the same average abilities in Spanish. However, students in the intervened municipalities acquired higher abilities in reading K’iche’ and Mam compared with students in the control municipalities. For students in second grade, a statistically significant difference was not found in reading abilities in Spanish (β3 = .13, p-value = 0.31) and Mam (β3 = -.12, p-value = 0.71). However, in K’iche’ language (β3 = .64, p-value = < .001) students improved their average abilities. This suggests that students in the control and intervened municipalities have the same average abilities in Spanish and Mam languages. However, students in the intervened municipalities acquired higher abilities in reading K’iche’ compared with students in the control municipalities. In terms of the percentage of second grade students who showed growth, results were similar for the Spanish language: in the intervened municipalities 51.70% showed growth while in the controlled ones growth was 50.10%. However, in K’iche’ language, the percentage of students in the intervened municipalities who showed growth was higher than the percentage in the control municipalities, with 58.30% and 43.30% respectively. Finally, in Mam language, the percentage of students in intervened municipalities was lower (52.70%) than in the control municipalities (58.60%). For third grade, the analyses showed no statistically significance difference in the any of the three languages: Spanish (β3 = 1.74, p-value = 0.46), K’iche’ (β3 = -0.32, p-value = 0.15) and Mam (β3 = -0.35, p-value = 0.35). This suggests that students in the control municipalities perform at the same level as students in the intervened municipalities. 30 Regarding the percentage of third-grade students who showed a growth, in Spanish, the percentage in control municipalities (32.60%) was lower than the percentage in intervened municipalities (38.30%). In K’iche’ language, the percentage of students who showed growth was lower in the intervened municipalities (40.60%) compared with control municipalities (51.20%). This was similar in Mam language, in which the percentage was lower in the intervened municipalities (52.70%) compared with control municipalities (58.60%). As in third grade, in fourth grade the analyses showed no statistically significant differences in the any of the three languages: Spanish (β3 = 0.04, p-value = 0.67), K’iche’ (β3 = -0.07, p-value = 0.71) and Mam (β3 = 0.28, p-value = 0.28). This suggests that students in the control municipalities perform at the same level as students in the intervened municipalities. In fourth grade, we also examined the percentage students, between control and intervened municipalities, regarding growth in reading abilities. In Spanish, the percentage was higher in the intervened municipalities (66.50%) compared with the control municipalities (63.60%). Similar situation happened in K’iche’ language, the percentage was higher in the intervened municipalities (65.20%) compared with the control municipalities (58.0%). Finally, the intervened and control municipalities had similar growth, 50.0% and 49.30%, respectively, in Mam language. 31 Table 1. Summary of Results of Language Abilities by Grade 32 There were several variables that showed a significant difference in terms of both positive and negative effects. In terms of positive effects, the groups of students who were enrolled in Pre￾school show higher Spanish abilities than students who were not enrolled in Pre-school for second grade (β14 = .14, p-value = .001), third grade (β11 = 0.91, p-value < 0.001), and fourth grade (β14 = 0.37, p-value = 0.02). This is also true for Mam language (β14 = .72, p-value < 0.001). Having books at home showed a positive effect in second grade in Spanish (β17 = .56, p-value < .001) and in Mam (β17 = 1.03, p-value < 0.02). Similarly, second grade students who like to read achieve higher Spanish abilities than students who do not like to read (β16 = .45, p-value = .06). Likewise, second grade students whose teachers give them homework perform higher in Spanish than students who said that their teacher does not give them homework (β15 = .63, p-value = .01). Additionally, second grade students who receive help at home with homework achieve higher Mam language abilities than those who do not (β11 = 1.10, p-value < 0.001). Regarding variables that showed a significant negative difference, students who have repeated a grade achieved lower Spanish abilities in first, second, and fourth grade. This was also the case for first-graders and Mam language abilities (β12 = -.390, p-value = < .001), though it was the opposite in K’iche’ language where students who had repeated achieved higher abilities than students who have not repeated a grade (β12 = 0.400, p-value = 0.02). However, fourth grade students who have repeated a grade had lower reading abilities in K’iche’ than students who have not repeated (β10 = -0.37, p-value < 0.001). In Mam language, second grade students that help in home with tasks have lower performance than students that do not help (β5 = -0.61, p-value = 0.05). Likewise, second grade students who work achieve a lower Mam language ability than students who do not (β6 = -0.45, p-value = 0.03). This is similar for third-graders in K’iche’ language: students who help at home with tasks achieve lower abilities than students who do not help (β5 = -0.71, p-value = 0.02). Surprisingly, third grade students who work perform higher than students who do not (β11 = 0.40, p-value = 0.01). Having siblings at the school showed a negative difference in K’iche’ language for second graders (β12 = -.24, p-value = .03) and Spanish language for fourth graders. The HLM analysis confirmed the aforementioned results. However, in most of the latent variables we did not find a difference between teachers from control municipalities and teachers from intervened municipalities. Latent variables include: a) activities at the library, b) reading comprehension activities, c) participation in learning communities, d) teacher’s background, e) teacher’s abilities, f) evaluation methods, g) evaluation tools, h) abilities in Spanish language, i) abilities in Mayan language, j) used of materials, k) parents’ involvement, and l) vocabulary activities, This applies for first, second, third and fourth grades. One variable that revealed a statistically significant positive difference is Spanish language ability in first grade: teachers from intervened municipalities showed higher scores than teachers from control municipalities (β15 = 0.663, p-value= 0.032). However, the variable “teacher background” in third grade showed a statistically significant difference in favor of teachers from control municipalities (β15 = -1.003, p-value= 0.040). See table below for a summary of the results. 33 Developing literacy materials for L1 and L2 for children and teachers As mentioned previously, one of the key elements mentioned in regard to being able to successfully implement the intercultural and bilingual literacy development in the early grades model is having local language materials in the classroom. Important materials to support effective teaching and learning consist of: 1) texts/readers and storybooks for students, 2) other supplemental reading materials such as big books, and 3) teacher guides with example lessons and pedagogical strategies for literacy instruction in L1 and L2. All of these should be adaptable to the model chosen for the various schools/classrooms, such as bilingual K’iche’/Mam and Spanish or monolingual Spanish. The MOE distributed materials including flipcharts and big books for literacy instruction in both languages in the schools prioritized by the former government, though none of these were seen or mentioned in the departments, municipalities and schools included in the data collection for this midterm evaluation. In the visits to the schools, the only materials in the classroom that were seen were textbooks/workbooks for children developed by CRS (Momostenango) that had both Spanish and Mam, including vocabulary, lessons, and short stories. In El Quiché, a teacher had bought a book to teach K’iche’ vocabulary to her students, which did not contain teaching strategies/lessons for bilingual literacy development (rather she had bought it to help her with her K’iche’ language knowledge). There were some textbooks/workbooks that were put aside and appeared to have little use. The lack of more school visits made it impossible to determine the degree to which bilingual literacy materials for teachers and students exist in schools. It is unclear how many schools in LLPs target municipalities have such materials or if there is a reliable inventory of which schools and grades have such materials. Finding such an inventory, or encouraging the departmental directorates to create one if it doesn’t exist, is an important way of assuring that teachers have the necessary resources for effectively helping their students develop their language abilities (Mayan and/or Spanish). In the MOE departmental offices it is clear that there is a need for materials (especially for K’iche’ and Mam literacy development), and some have begun to take ownership over the responsibility to develop educational materials. This varies between departments in terms how and the degree to which they are doing this. For example, the San Marcos departmental MOE has taken the lead in the training processes and in the development of educational materials, especially in Mam as a second language. In El Quiché there are efforts to collect locally developed materials (which exist in electronic form) and have them printed. Teachers love to make materials; encouraging these efforts and working material elaboration into the professional development and action research could be additional ways to get materials into the classroom to support literacy development in Mayan languages and Spanish. With the support of the Material Management Group created at the departmental and interdepartmental levels, some teachers and parents learned to manage the materials development process, and this resulted in a product known as a reading lecture anthology in Mam, K’iche’ and Spanish. One aim is to articulate processes and search for material development options, though it is unclear if similar material development activities or sustainable strategies are planned. 34 There is a clear need for teacher guides, which was expressed by informants at a central and departmental level as well as directly by the teachers. The need is both pedagogical and language-related (i.e. for teachers to have guidance in didactic strategies for L1/L2 literacy instruction and to develop their own Mayan language abilities). However, at both the central and departmental MOE levels, there did not appear to be a plan to develop such guides. LLP is compiling materials for a teacher’s kit to be given to them at the end of their professional development process, which will help to meet this need. Providing them with concrete didactic approaches and strategies and example lessons in K’iche’, Mam, and Spanish (depending on the particular need) should be a priority. Also, involving teachers in this process is recommendable in terms of ownership and professional development, such as having teachers develop and pilot and share lessons/strategies as part of their professional development. Furthermore, providing them with tools/materials to develop the Mayan language they must know for bilingual instruction is key; this could be done in coordination with the ALMG and/or other language specialists. Implementation of the National Textbook and Education Materials Policy The project initially sought to support the definition of the textbook and instructional material policy, but the National Education Council (CNE) ruled that the policy was not necessary due to the existing policy on Quality Education (No.2), which is believed to encompass a policy on textbooks and materials. Instead, it was concluded that what was required was to define the strategy and the Implementation Plan regarding texts and materials. LPP has accompanied the MOE in defining the strategy and the implementation plan for this. At present, the MOE needs to review the documents and be willing to implement the strategy under the expected objectives of this process. Furthermore, the strategy of LLP is not to finance the printing of the materials necessary for bilingual literacy development, rather to provide technical assistance for material development and strengthen the MOE ability to anticipate procurement and distribution of materials on a multi-year plan. Printing and distributing materials is the responsibility of the central MOE (DIGEBI) and must be taken into account as they make their Annual Operating Plan (POA). It was noted that for next year 2017 they already budgeted a considerable amount of resources for materials. It is presumed that the MOE will print them and will provide them for use, but it is unclear how their distribution and use will be verified. The Ministry of Education formed a commission, which received technical guidance from Juárez and Associates. LLP contributed to establishing the selection criteria for the acquisition of books (stories, etc.) in Spanish and Mayan languages. This technical assistance was provided under the National Reading Program, as well as the procedures for the selection and use of materials. It is unclear the degree to which this will be appropriated and followed by the MOE. Encouraging coordination between the central and departmental MOE offices is important to assure that the anticipated procurement and distribution is based on actual needs (demographically and linguistically) and that the materials reach the schools at the beginning of the school year. Furthermore, assistance should be provided to ensure that the quality of the materials is such that they will be durable for several years (i.e. not just physically durable but also textbooks/readers for students and all other materials to be re-used and not disposable yearly). Additionally, 35 strengthening the capacity for material development at the departmental levels (such as training material development specialists and involving teachers) can help ensure quality and appropriate materials exist. 2. Human resource development in the education system LLP is using various strategies to help strengthen the capacity of the central and departmental MOE offices and personnel to implement the EBI model and improve early literacy development. One overall finding is that the DIDEDUC, schools and the central management of the Ministry of Education are not prepared at this moment with sufficient human and technical resources to give continuity to the strategies, processes and mechanisms developed by the learning model driven by the project. While positive changes and process are emerging, including ownership over some of these, much emphasis must be placed on assuring structural and financial sustainability of the various interventions, knowledge, and tools acquired to strengthen literacy development and EBI. Higher education degree programs to prepare a cadre of teacher professional development experts; Education development experts trained and functioning in the education system to support classroom-level improvements Continuous professional development is one of the most fundamental elements for improving teaching practices and learning outcomes. There is widespread awareness amongst LLP stakeholders that the university programs carried out through the project (for teachers, CTAs, and other MOE staff/specialists) that consist of a sequence of courses is likely to have better success than teacher training workshops, as were previously carried out by the MOE. This was expressed by informants and also observed in focus groups in which those participating in professional development discussed having learned foundations of bilingual literacy development, neuroscience, etc. instead of referring solely to “técnicas” (practices). Informants also expressed that the LLP approach to professional development is further strengthened by the fact that the courses are carried out after working hours, have college credit, and must comply with requirements. Other positive aspects mentioned were that it is not only on-site and at-a-distance, but there are other key elements such as reading circles, learning communities and educational support. All of these make for a more comprehensive program to support and accompany teachers. Regarding the teacher accompaniment in the classroom provided by the UVG (in charge of the teacher professional development), teachers expressed that this was extremely helpful though they are also aware that this is not indefinite. Assuring that the CTAs assume this role (with the same quality as the UVG) is very important for effective implementation and sustainability. CTAs and other MOE departmental personnel participate in professional development that includes teacher accompaniment in the classroom (or coaching), which is the Leadership Specialization Program for Educational Accompaniment with emphasis on literacy and intercultural bilingual education. One challenge, however, is maintaining (and/or increasing) the level of involvement of the tutors that provide this professional development in order to assure that the capacity for coaching is developed to the best degree possible. While some tutors are full-time professors, in other universities they are not, and this can affect the quality of support 36 that they provide. Because the hourly contracting of tutors/professors is standard procedure of the university, it may be outside of the control of LLP. However, it is important to note this challenge and encourage the LLP implementers to seek a way to assure that the MOE departmental personnel (especially the CTAs) fully develop their knowledge and practice of coaching and truly internalize the new model promoted by LLP. The learning communities at a departmental level are considered to be a great value added; an important change is the inter-departmental work, which is seen as a result of the benefits derived from experiences amongst the different departments. The participation in learning communities and reading circles in some cases has allowed teachers to share experiences and be able to participate in the development of strategies for classroom implementation and innovation. However, some CTAs and teachers expressed that there continues to be resistance to change, teamwork and commitment to carrying out individual study plans/tasks in learning communities. To make these learning communities sustainable at the departmental and community levels beyond the time of the project will be a challenge. Additionally, informants mentioned that reading circles are carried out only in Spanish, creating difficulties in the use of the Mayan language despite having some tools and technical resources. The use of technology and e-learning are innovations that are promising in terms of forming a long-term strategy for teacher professional development both in the five departments and perhaps as part of a national model. Capacity for e-learning is being installed as participants learn how to use technology and implementers (such as tutors) develop this expertise. Additionally, program designs have had the support of universities that previously did not know how to follow-up these programs. Universities have been motivated to approach teachers. However, the use of technology presented difficulties in many cases as well. In addition to many teachers being very unfamiliar with such technology use, access to internet was a limitation throughout. The use of tablets, having all of the modules installed prior to disbursing them to teachers, would likely help resolve this situation. It has been noted that the change in the literacy teaching approach has been positively received overall and incipient changes can be observed (at least at the level of self-reported learning and applications of what was learned through the courses). There is more discussion of the alphabetic principle and the importance of reading fluency and reading for comprehension. Teacher professional development has also taken a more central position, including by the departmental MOE personnel. Some of the participants in the Specialization in Literacy in Bilingual and Intercultural Environments offered by the Pan American University consider that this system and approach to professional development should serve as a model for the Initial Teacher Training that began in 2015 at the university level. In some cases, a change has been seen in the attitude of teachers in their responsibility to meet a full-day of attending courses as well as their weekly course hours. Some of the teachers expressed that the knowledge acquired in training programs they are applying in the classroom and feel proud of the results accomplished. However, there are teachers who, because they have another job, do not have time to participate as fully. Informants in several departmental MOE offices expressed that resistance to change persists with some teachers; though some are 37 prepared and have experience, they do not exercise leadership and do not commit to the teaching objectives. Furthermore, it is unclear what the role of school principals in the LLP interventions is; it was mentioned by informants from central, departmental, and municipal levels that their participation in the program is important. It is important to note that the professional development programs carried out by the universities are currently being evaluated to inform the next phase of implementation, which will likely provide valuable information. A large number of education supervisors, CTAs, and technical staff of the MOE in the five departments of LLP intervention have participated in the Leadership Specialization Program for Educational Accompaniment with emphasis on literacy and intercultural bilingual education, which is implemented by the School for the Training of Teachers of Higher Education (EFPEM￾USAC). As a result of this, informants mentioned positive changes in educational support such as the existing supervision and monitoring proposal being substituted by one centered on providing accompaniment and support for teachers. Though still incipient, it was evident in focus groups and interviews in the departments that there is change in the way supervision is seen; it is shifting from being perceived as an audit oversight to being understood as friendly and helpful educational support for teachers. However, there are not enough human resources in the departmental MOE offices to accompany each school and classroom since CTAs have other administrative responsibilities that must be met. Apparently, the Ministry of Education has taken up the need to implement the National System of School Coaching (SINAE), for which, reportedly, it is providing about 90 million quetzales in the 2017 draft budget. For the start-up of the implementation of this system, the Ministry is planning to begin with 8 departments (San Marcos, Quetzaltenango, Huehuetenango, Totonicapán, El Quiché, Sololá, Alta Verapaz and Petén), but its implementation hinges on the decision of the National Congress. Education Human Resources Professional Development System implemented and continually updated and technical support to groups leading efforts to reform human resource policies so reforms can be adopted and put into practice in Guatemala The teacher educator professional development program carried out through a consortium of universities was one of the final elements incorporated into LLP. In 2012, with the support of USAID/REAULA and other actors involved with pre-service teacher education, the training programs for primary school teachers began to be transferred from the “Escuelas Normales” to the universities; the first cohort was from the University of San Carlos (Humanities Department). Foreseeing the challenge that this presented to the universities who would now assume responsibility of preparing future primary school teachers, LLP created the teacher educator professional development program to prepare professors that would be in charge of this. According to what members of the Consortium said, until now the work between the universities has achieved articulation and cooperative work. The universities are careful to note that the Consortium is for the universities and they are mindful of what they do; each one recognizes the independence of each institution, including the autonomy of USAC. Also, rather than competing, 38 all share their material resources and other elements relevant for the program. One of the representatives expressed it as follows: "It is a great success to see different universities working towards the same objective and purpose, and that is reflected in the responsibility they adopt with a curricular module. Not only are they responsible for creating it together with specialists paid for by USAID, they are committed according to the modules content; the same university that generates it is the one that uses it along with the teachers and these teachers are exposed to different ways of seeing the world". The Consortium members meet weekly and each university decides who their representative is. They provide mutual support to carry out work with an interdisciplinary team towards the same goal. It is also notable that they do not receive payment for this. It is important to note that, despite the aforementioned strengths and achievements related to the work of the Consortium, there appears to be some tension when it comes to certain decision-making. Although its members are representatives of universities, some of them do not have the status of dean; thus they do not have the authority to make descisions about programs, professors, and other aspects. The forum of deans expressed mistrust regarding this situation. For example, the Consortium is beginning to be seen as the link or representing entity to the MOE for teacher education/training; this diminishes importance of the deans who assert their authority over these matters within the universities. In this sense, some deans have expressed the need to promote better communication and clarity regarding the role of the Consortium and its future. In the case of Guatemala, the MOE does not legally have the institutional responsibility of higher education. Therefore, private universities and the state university (Univerity of San Carlos) have the autonomy to develop educational programs, courses, orientations, etc. as they choose. In the future, the role of the Consortium could be to document and consolidate the experience of the teacher educator professional development program as well as to serve as an advisory body for universities and the MOE in areas such as pre-service teacher education, coaching, intercultural and bilingual education, education leadership, educational measurement and evaluation, and development of support materials. Although previous USAID projects supported the definition and realization of SINAFORHE and SINAE, according to information provided, SINAFORHE was unable to start-up during the last administration. Although this administration sees the project positively, there has not been a policy decision leading towards starting this process, despite being included in the 2016 to 2020 Strategic Plan of the Ministry of Education as one of the activities planned to be carried out. Continuing to works towards its implementation, providing the required technical support will be important, as encouraging the departmental MOE offices to prioritize placing those who have or are receiving professional development supported by LLP to assume positions to help further the goals of improving (bi)literacy development. 39 3. Parental and community participation in education leading to improved learning and reading achievement This component aims to: 1) increase parental and community involvement to support children’s education with a focus on reading, increase public awareness of and support for critical education themes, and support policy and programmatic decisions based on results from applied research studies and a quality monitoring system. Contributions have been made towards increasing parent involvement. Parents have begun to understand that they have a role in being involved their children's learning. The fact that information is being presented using their own language and local culture, and seeing their Mayan language and culture being acknowledged and respected in meetings and other established processes appears to be a positive element in that encourages participation. Some parents feel that they too are learning new things, and in some reported cases there have been interactions with children while they watch their children´s performance and the quality of teacher instruction. Some begin to verify the delivery of their child's homework, to participate in reading events, and to have closer ties with teachers. They value very highly the fact that in meetings they are able to exchange experiences with other parents. As stated by a mother: “I like to go because children are happy to see us. If you do not go and it is something important, sometimes they say they will loose and feel sad if you do not encourage them but then they feel happy when they are supported.” Another mother expressed: "I feel happy when my children express their skills on stage, I feel proud and they start to participate more. When children go out to play to another village I like going to see how they play.” Reasons for involvement and ways that parents can get involved vary. For example, one mother expressed: "My child is in Primary and, since my educational level is also of Primary school, I cannot help him." Some parents are seeing the need to prepare themselves in order to help their children with homework, and therefore seek CONALFA or any other adult education system. Other parents may see their role as one of checking to see that what is occurring in the school, as expressed by a father: "We have to see how teachers work, sometimes there are children hitting other children, or stealing, as a parent I like to go to monitor and support them in these situations." Another father emphasized the importance of involvement through the following statement: "We have to be with the children, asking them how they are doing, if the program is good, check their schedule and the time allotted for recess. We must review their homework, start reading a book so they learn, set an example to the children, guide them. Previously, girls were not allowed to study. " Based on what the interviewed people have stated, father participation has increased (though not above 10 percent). Traditionally, mothers are the participating parent, and it has been reported that young children have been accompanying their mothers to the parents’ sessions in which there have been playful and didactic materials in Mayan languages. The use of the traveling bag and the classroom for parents in some cases serves as an instrument for increased parental involvement in order to solidify their children´s reading and writing knowledge. At the same time, school principals have begun to agree lending books from the 40 school library for the children and parents’ use. In addition, the development of materials by parents is beginning to take place, as is the case of the community newspaper, etc. As part of the educational community, some local leaders such as members of the Community Development Councils (COCODEs) and certain mayors support and endorse these processes. At times they participate and start to show that they are responsible and can influence municipal education plans. Additionally, it was noted that at the municipal level there is father and mother involvement and enthusiasm to participate. In some cases, it is stated that at the municipal level there is a greater recognition of the importance of the critical education issues. The aforementioned is anecdotal and it is too early to talk about widespread and system changes in the areas of intervention, yet it does give an indication of several emerging changes. Additionally, Plan International experts who expressed their opinion regarding the major advances they perceive in the field have highlighted changes. They concur that father participation may be increasing but is still around 8%, the rest being mothers. Their perception also concurs with what parents stated regarding parents having a better ability to talk with their children about educational issues and improved communication between parents and children, more accompaniment by parents with their children to school, parents requiring their children to complete homework and at times decreasing the burden of other household chores for this. Other changes that were reported by informants include parents asking their children to read recipes or product instructions, improved parenting skills, better relationships with teachers. In addition, some parents are creating spaces to read with their children, and it has been noted that parents have enriched their mother tongue vocabulary. Several strategies have instigated such changes such as the school for parents, the traveling reading bag, reading fairs, promotion of oral activities, and activities in which parents have been involved in reading games (a strategy that was not foreseen originally). The sub-contractor in charge of the component of parent involvement (Plan International) has used a strategy that does not work directly with parent committees; rather the school principal convokes the parents with children in pre-primary to third grade to attend meetings. The principal provides orientation to the parents, based on a plan that they are provided with (by the sub-contractor) which is a working document (work-in-progress). Plan International also provides support to finish it and put it into operation. Additionally, a literacy protocol for each grade will be introduced. Some challenges remain. For example, LLP has worked with the General Directorate for Strengthening Community Education (la Dirección General de Fortalecimiento de la Comunidad Educativa/DIGEFOCE) as well as at a departmental level with DEFOCE (Departmental unit in charge of strengthening community education) so that there is a better link with the education quality improvement unit (i.e. to improve quality through parental involvement). However, more work must be done to strengthen this link so that the proposed strategy of involving parents in improving education quality can generate the desired effects. Furthermore, the degree to which this component of parental involvement is functioning and generating positive effects in the schools is unclear. It is reportedly functioning in the 36 schools of intense intervention (“escuelas intensivas”) that have school for parents and in all of the “escuelas intensivas” en general; reportedly all 216 schools already have the intervention of school for parents. However, it was detected in discussions with informants in the five departments that the degree to which this is 41 functioning varies within (school level) and between departments as does the link between quality education and DEFOCE. Some commented that there are principals and teachers who are not committed to assuming their responsibility with the educational community. In one region, El Quiché, departmental personnel expressed little or no knowledge of this component and in the school visit the first grade teacher also had little knowledge of it (though the principal mentioned that it was functioning for third grade). Some municipalities are assigning more resources towards local educational needs such as physical space, electricity in youth centers and teacher salaries. Additionally, some are strengthening their municipal education plans and including issues that are critical for improving educational quality. It has also been noted that some parents are opposed to EBI and demand a better education, which can occur when they see that teachers do not meet the schedule and do not know the methodology to use in both languages; they perceive that their children progress slowly and with shortcomings. There is a need to increase in public awareness at a local level regarding the quality of education, bilingual education, reading, and issues regarding conditions for learning opportunities. According to personnel of the sub-contractor in charge of this sub-component, there are plans to analyze and define strategies to address such challenges that include taking into account the results of the parent assessment, which contain relevant data for actions to be taken. Overall, the intervention of working with parents is promising based on the aforementioned information. However, as mentioned, this component is not functioning as planned in all of the intervention schools and in some cases there was a lack of knowledge at the departmental level (MOE) about this component of the project and its interventions; this reflects the need for more direct communication and coordination between the implementing organization and the MOE departmental offices of the target municipalities. It is also unclear the degree to which school principals in general are directly involved in each target school, which appears to be a key factor for this component’s success. Having a way to register and keep track of each school’s level and type of advancement with this component would be a helpful way of assuring its success during the remainder of the project. In sum, regarding component A, each of the interventions of the EBI model supported by LLP is important and they are complementary in terms of the overall goal of improving literacy development and bilingual education. There are clearly incipient changes, as evidenced by the information collected and analyzed for this mid-term evaluation. Some best practices to be mentioned are: 1) the professional development programs that are providing both foundations for understanding and putting into practice ways to improve literacy development and bilingual education, 2) the strategy of universities providing professional development in the local context, 3) the learning communities at the departmental and community level, 4) the valuable experience of producing anthologies and other materials to support literacy development in the local language, 5) the commitment and change of attitude and practices of some CTAs and teachers, 6) the organization of time in classroom teaching including a fixed amount of time for teaching the L1 and L2 as well as reading, 7) teacher and parents both understanding the importance of joint coordination in student activities and increased participation of parents in general. These are all practices that can be documented and shared both in the areas LLP intervention and on a national scale. 42 B. Education for Employment and Lifelong Learning 1. Equitable access to alternative basic education and workforce training for out-of-school youth (OYS) Overall, findings reveal that there are educational alternatives and workforce development training available to out-of-school youth. Also, a relatively large number of out-of-school youth are registered (as of September 2016, LEA had registered 5,259 out-of-school youth). However, a relatively small proportion of the eligible population is actually enrolled in formal educational alternatives or workforce development. According to the LLP project progress report (10/5/16), 466 youth (8.86%) are studying alternative basic education in elementary school (primaria acelerada PEAC), 398 (7.57%) are studying 7th to 9th grades (básicos flexibles,) and 369 (7.01%) are in workforce training with FUNDAP. There are no data from INTECAP. In total, 23.44% of registered out-of-school youth are receiving education and training. Therefore, the majority (76.56%) still is not receiving alternative education and/or training, although some of them may be participating in forums, workshops, committees and networks related to the project. Also notable is that 51.8% of the registered out-of-school youth are minors (15-17 years old); it was not possible to obtain aggregated data regarding the number of these minors who are receiving alternative education or workforce training. However, it is important to capture both this data and information on the positive changes occurring as a result of the project, such as those discussed in the following paragraph. (The need for a monitoring system is discussed in a subsequent section). Several young people expressed very positive changes in their lives since participating in the education and training. These include self-esteem, self-confidence, overcome their fears, and the capacity to deal with adults. A19-year-old girl in Momostenango, expressed: “The program has greatly helped young men and women to study. I thank you because I am a single mother and I have a 6-month old daughter; I want to be an excellent nurse and a have future for me and my family. Keep going forward USAID personnel in helping many young people”. (“El programa ha ayudado mucho a jóvenes y señoritas a que estudien. Les agradezco porque soy madre soltera y tengo una hija de 6 meses; quiero ser una excelente enfermera y tener un futuro para mí y mi familia. Que el personal de USAID siga adelante ayudando a muchos jóvenes.”). As part of the plan for organizational capacity building, LLP has organized three groups to collaborate in order to reach OSY goals and engage youth to develop a life plan. These groups are: OSY Consultant Committee, Stakeholders Regional Network and an Advisor Committee for barrier reduction for youth education and employment. These groups have been organized during 2016 and are beginning to accomplish the LLP objectives and goals. Because there are various members who were recently invited by LLP to join the groups or committees, they need more time to engage with the process. The work they will be able to do will likely be useful and will allow OSY to incorporate into the labor market. However, time is still required for these groups to be able to provide feedback and advice for the organizations, OSY and LLP. 43 LLP has organized youth workshops called Youth Raise your Voice (Joven Alza tu Voz) to strengthen youth organizations and strengthen the commitment to work together and bring the issue of OSY to national public awareness. Those workshops have given voice and audience in different public and private spaces where youth rights have not been recognized before. This action has been shown to be very useful to open the minds and raise the awareness of authorities at local, community, municipal, and central government levels. One strong result was that in August 2016, a group of OSY organized an event at the central MOE which resulted in recognition by the MOE authorities the need to give attention to the issue of OSY and include this in their future annual plans and budgets. Network of collaborating NGOs, public, and private youth-serving providers/stakeholders There are four organizations offering formal education and workforce development training: Asociación para la Cultura y el Desarrollo (ACUDE), Asociación Grupo Ceiba, Fundación para el Desarrollo de Programas Socioeconómicos en el Altiplano (FUNDAP) and Instituto Técnico de Capacitación y Productividad (INTECAP). These are well-established and respected organizations which are working together to provide alternative basic education and workforce training to OSY. These services are partially meeting the needs of the demands of the youth in regards to their interests and existing local opportunities. However, there is a diversity of interests, some of which may be very appropriate for the reality of the informal local market and linked to local production (e.g. like musical instruments, sustainable energy production, etc.) A system for outreach, tracking, and referral/transition to identify and meet youth needs There are numerous success stories recounted by the enrolled youth, many of whom report life￾changing experiences in the program. However, there is no evidence of a formal tracking system and the sub-contractors currently do not have the tools to create an effective one (e.g. bar codes, optical readers, and scanners). The registrant list of 5,259 out-of-school youth could serve as a basis for a tracking system, although the current registration form is missing some key elements (e.g., numbered questions). Furthermore, sub-contractors Child Fund y Plan International currently enter data manually, which raises potential for human error. Any tracking system should also include a mechanism for qualitative, as well as quantitative, data. LLP has invested significant time and expertise in the development of the SIREEX system (Sistema de Registros Educativos de Educación Extraescolar/ Education Records System of Non-formal Education) within the Ministry of Education. SIREEX is expected to significantly expand services to out-of-school youth. It will also allow DIGEEX within the MOE to identify, assist, and track youth. The development process has been shared and co-financed by Swiss Contac. In 2017, the system will be piloted. The development of SIREEX has been an enormous project requiring in-depth understanding of the Ministry’s current system (SIRE/Sistema de Registros Educativos). SIREEX will use the unique identification number (CUI) each Guatemalan citizen currently has and that the national registry (RENAP) creates according to each person’s date of birth and DPI number. 44 Elimination/reduction of barriers to access equitable alternative basic education LLP is reducing barriers to access, particularly transportation, with direct subsidies. However, some barriers remain. Some youth (females especially) mentioned that lack of parental permission and support is a significant barrier, and LLP does not currently address parental engagement. One reason cited for this lack of parental permission was that parents tend to have a perception of existing dangers that their children (especially females) may encounter. There are ways to overcome these barriers, which are in the recommendation section. 2. Improve the quality of alternative basic education and workforce training The project offers concrete and significant improvements to the alternative basic education and workforce training opportunities for out-of-school youth. LPP’s sub-contractors are experienced and well-known throughout the country. To expand upon current successes, the sub-contractors may undertake a number of important efforts now that they know the standards and have completed self-evaluations. There are no national benchmarks, but there are international benchmarks for this type of work. Two important lessons from international benchmarks are the importance of creating models of entrepreneurship and access to credit. These two components facilitate self-employment when sufficient traditional (government and private sector employment) opportunities do not exist to meet the supply of eligible youth. Adopt shared quality standards for alternative basic education and workforce training, improve capacity of youth-serving organizations, and establish and use a monitoring system for program improvement Service providers defined standards in thematic and institutional areas, and they work with the organizations to ensure compliance. The Office of Out-of-School Youth (Dirección de Educación Extra Escolar/DIGEEX) of the Ministry of Education also will be able to use these standards and the technical team of this office is already familiar with the standards, so now it is just a question of their appropriation and application. Compliance with the standards is voluntary and the consultant who was hired to assist with compliance is helping the organizations. The organizations did complete a self-evaluation, reviewed the findings, and are implementing improvements. However, the governing boards of the organizations have not been involved in the standards-development and self-evaluation processes, but it seems like they would be willing to assume and comply with the standards. The organizations indicate that to improve capacity, additional investments in workshops and labs are necessary. Ideally, a virtual platform would be created to improve their ability to do simulations and long-distance education. LLP established a close collaboration with DIGEEX and hired a consultant to develop a strategic plan for the next four years. This plan is a Road Map to better incorporate DIGEEX into the MOE and obtain more resources to accomplish its mission. There is a weakness in the government’s non-formal education subsystem because central and governmental authorities (in addition to the MOE) do not dedicate attention to OYS. This strategic plan (Road Map) can contribute making visible the high numbers of OYS and identify ways that they can be 45 recognized and attended to. The Guatemalan Congress is the top entity that authorizes annual education budgets, and the MOE must inform and be persistent regarding the importance of addressing the needs of OYS. With the strategic plan, DIGEEX has a useful tool to establish the urgency of attending thousands of adolescents and youth who are out of the education and training system. The level of impact of this plan’s design and implementation will be seen in the next years. Another way to collaborate with DIGEEX LLP has been the development of the SIREEX system (Sistema de Registros Educativos de Educación Extraescolar) within the MOE. SIREEX is expected to significantly contribute to identifying and fine-tuning the location and needs of OSY to be able to expand services to them. This is explained in more detail below. Regarding a monitoring system of students in alternative basic education and workforce training, (a requirement for the LLP), evidence was not found that this exists. There are independent reports of students’ advancement that are prepared by each education and workforce training organization, but these are not yet organized in a tracking system. Those organizations, which receive LLP funding through Juárez, have been recently incorporated and are still attending cohorts of students. Juárez receives monthly reports of youth attendance. However, the actual enrollment forms, attendance lists and other current ways of trying to manage information is not effective and organized enough to meet what is ultimately required by LLP for a comprehensive OSY database. Basic information would include their job placement or self-employment initiatives; anecdotal information and some success stories are captured but not comprehensive and complete information that includes elements recommended by USAID: gender￾disaggregated information about students achievement to improve basic literacy, numeracy, and life skills as well as information about successes, challenges, and drop-outs. Such information is important in order to inform program improvement and establish successful tools and interventions for OSY, thus establishing this monitoring system should be prioritized. To develop a monitoring system, workforce development providers need tools (such as hardware and software) that are not currently part of the budget. They have no budget for this purpose and it is not defined in their project contracts. 3. Youth Engagement In general in Guatemala, the government, non-governmental organizations, and community organizations have not involved youth voices in the design and implementation process. This has begun to change with the actions carried out through support provided by LLP. Without a change in vision to meaningfully incorporate young people’s input, real and sustainable engagement on the part of the youth seems unlikely. In this sense, LLP has begun to set the foundations for this type of change. Youth-serving organizations mainstream youth participation The providers of educational services, the municipal authorities (Consejos Municipales de Desarrollo or Comudes), and community organizations (Cocodes and others), as well as government operators (the Ministries of Education, Health, Development, Labor, and others), are not consulting with young people in their design and implementation processes. Opening spaces 46 for youth participation is an arduous task that first requires a change in adults’ point of view. LLP participating groups and organizations have already made requests to the Ministry of Education (in August, 2016), and the requests have already sparked a change in the Ministry’s view toward out-of-school youths’ voices. For example, as recounted by LPP implementers, after the Minister was made aware of the work being done with OSY, he has included this issue in his speeches (which had not occurred previously). Additional unifying efforts are needed to foster more changes in the beliefs and practices of central authorities, local government, and community groups’ in terms of youth participation. Engage youth in mobilization and policy dialogue for equitable access There are already a large number of community youth committees (211), as well as municipal youth committees (11). One hundred and twenty-eight community plans of action exist in the region after performing one hundred and fifty-one (151) Participatory Youth Assessments (PYA). Recently there is a movement to permanently include youth in the municipal development councils (COMUDES), which is a positive step since COMUDES may assign funds designated for youth development. Advancements in that process of inclusion are being made but there should be continued coordination with other organizations to ensure continuity and to ensure the funds and budgets for youth are in fact allocated. If permanent youth involvement in the COMUDES fails to actually occur, momentum will be lost and the committees that already include youth may lose credibility. Increase opportunities for meaningful youth civic engagement and service in their communities Thanks to participation in LLP, out-of-school youth are becoming conscious of their rights and responsibilities and organizing themselves. The project has promoted awareness among youth of their value and potential. A number of the OSY expressed that they have lost their fear of self￾expression and that they are becoming aware of and requesting respect for their rights, at the same time as they reinforce their sense of responsibilities to themselves, their families, and their communities.11 These trends are novel in the rural indigenous communities of the targeted municipalities. Because the out-of-school youth earn the trust of adults and elders by performing community service, promotion and expansion of existing youth involvement in community service will further enrich adults’ participation and acceptance of LPP. There are traditions of voluntary community service, including serving as caretakers in the community forests in Totonicapán, school guards (who care for schools on the weekends and during vacations), and water collectors; these are not part of content in the civic education or political participation guides. The above examples could easily be shared and scaled up to other communities by LPP through the materials on Life Plans, Gender Equality, Sexual and Reproductive Health, and Entrepreneurship. Rates of youth participation, parental engagement, and likelihood of sustainability would increase by explicitly incorporating and expanding upon concrete examples of youth engaged in voluntary community service. 11 These OSY were in Quetzaltenango, Totonicapán and El Quiché and enrolled in educational programs. 47 LLP has achieved many successes and surely will increase participation and chances for sustainability if it takes into account and reflects more of the culture, language, and values of indigenous communities; this would also allow LPP to promote prevention of crime, violence, and illegal activity in a culturally relevant way. VI. ADJUSTMENTS MADE BY IMPLEMENTERS AND RECOMMENDATIONS This section discusses strategies and approaches undertaken by the implementers to make adjustments to interventions based on evidence (evaluation question 3) as well as the evaluation team’s recommendations for the next phase of implementation. Component A: Adjustments to LLP by Implementers LLP has given continuity to many of the interventions supported by the previous USAID￾supported project REAULA in the department of Totonicapán and El Quiché, "Development model based on national standards for intercultural bilingual education, focusing on literacy for early grades”. Many recommendations from the implementation of this project were incorporated into the LLP project. One of these was to continue to support the Ministry of Education in implementing the EBI model with a focus on literacy through departmental offices where it is more likely to be consolidated, yet maintain close links with the central level. In this case, it continued support for DIDEDUC, but as discussed in the findings, there are indications that the linkage between the departmental and central levels has not been sufficiently strong; this is important to assure effective implementation locally to share best practices and have a common vision of guidelines and plans with the central level from which the major decisions and guidelines of educational policy emerge. It was also recommended that, using the experience of bilingual human resource training at postgraduate level, there be a process of systematic professional development of bilingual teachers in service that would allow them to gain mastery of language teaching in local languages. The implementation of the teacher education programs to work in bilingual and intercultural contexts implemented by LLP with three universities has allowed for preparing more human resources to implement the model. However, in the opinion of most respondents, this does not guarantee that the process is maintained systematically and sustainable. In regards to participation of teachers in the professional development courses, the enrollment was not what it was expected to be for the first cohort. Various reasons were mentioned to explain this including teachers not receiving notification early enough (or at all) or already being involved in a different professional development opportunity. The implementer was aware of this problem and did assure a higher number of enrollments for the second cohort. The implementer has continued to monitor processes that relate to resources (such as materials) to support bilingual literacy development and help ensure this is included in the annual plans of DIGEBI and/or Departmental Directorates of Education for printing, distribution and support of its use in schools. Though this strategy has been further strengthened, the Ministry of Education 48 has not had the financial ability to provide textbooks and support materials to schools, especially in Mayan languages of the region. A first attempt has been made by longitudinal study (2015 and 2016) to measure progress in literacy in the early grades in Mam and K’iche’, but what is still pending is the analysis the findings and taking measures for making the necessary adjustments to the EBI model and LLP interventions in particular. There is no doubt that LLP is strengthening the capacity of a number of teachers, CTAs and technical staff of the Ministry of Education to better implement quality reform in the classroom, especially as regards appropriate strategies, methodologies, and effective systems for teaching reading and writing in bilingual and intercultural contexts. What is not clear is the strategy to ensure the systems, standards and resources for the institutionalization and sustainability of progress. Additionally, it is unclear how the experience of both the Consortium and preparation of trainers and training programs for teachers in service may be adopted to enrich initial training. The structure of LLP as being implemented in conjunction with two sub-contractors that have no previous experience with USAID contracts in Guatemala has led to some difficulties. For example, certain interventions have been taken longer to implement (especially in regions where the sub-contractor had no or little previous presence) such as the production of materials, realization of events, and hiring of personnel. It is clear that this situation has improved, apparently due to more communication between the main implementer and sub-contractor as well as the later learning how to respond better to the contract mechanism requirements. Component A: Recommendations 1. Improve communication at a central level regarding the EBI model for literacy development in bilingual contexts: Strengthen communication between the implementing partner and the central MOE personnel (technical Directorates such as DIGEBI, DIGECADE, DIGEFOCE, DIGECUR) in regards to innovations that emerge from the LLP interventions to help generate guidelines and decisions to improve effective strategies for improving bilingual literacy development; this will likely help to ensure that the project’s interventions that are evidencing positive changes are sustained and scaled up. 2. Increase communication between the implementers and Vice Minister of Bilingual Education and his staff. This should include regional departmental MOE directors and staff. Taking into account other approaches embraced by DIGEBI and local initiatives may facilitate developing a common vision of an intercultural and bilingual education model that would likely lead to more sustainability and strengthening of an intercultural and bilingual education approach that leads to improving learning outcomes nationally. 3. Strengthen communication regarding the national intercultural and bilingual education model. If the aim is to develop a national intercultural and bilingual education model, there may be a need to have certain standardized guidelines and practices (e.g. to capture data and best practices, strengthen pedagogical approaches, etc.) while other 49 elements of the model can be left to the discretion of each department. Encouraging the strengthening of communication and coordination between the central and departmental MOE offices about this is recommended in tandem with providing support to the MOE in strengthening the national bilingual education model. 4. Emphasize ownership by government (MOE) counterparts to ensure sustainability. While positive changes and process are emerging with the EBI model through the support of LLP and by the MOE on its own accord, some of which are being appropriated, much emphasis must be placed on assuring structural and financial sustainability of the various interventions, knowledge, and tools acquired to strengthen literacy development and EBI. For the remainder of the project, a strong emphasis should be placed on assuring that MOE department staff and teachers take ownership over the interventions and approaches supported by LLP and that they share a common vision of how to provide instructional support systematically. Adequate resources and conditions must be in place for this to happen as well, for which technical assistance in planning and budgeting may be important. 5. As part of the LLPs support for the development and implementation of a policy and implementation plan for textbooks and educational materials, assure that there is a reliable inventory of existing materials in the departments, schools, and classrooms. It is unclear how many schools in LLPs target municipalities have such materials or if there is a reliable inventory of which schools and grades have such materials. Finding such an inventory, or encouraging the departmental directorates to create one if it doesn’t exist, is an important way of assuring that teachers have the necessary resources for effectively helping their students develop their language abilities (Mayan and/or Spanish). 6. Provide technical support for assuring quality materials for students (texts and stories in L1/L2) that are durable and respond to linguistic and demographic needs. Encouraging coordination between the central and departmental MOE offices is important to assure that the anticipated procurement and distribution is based on actual needs (demographically and linguistically) and that the materials reach the schools at the beginning of the school year. Furthermore, assistance should be provided to ensure that the quality of the materials is such that they will be durable for several years (i.e. not just physically durable but also textbooks/readers for students and all other materials to be re￾used and not disposable yearly). Additionally, strengthening the capacity for material development at the departmental levels (such as training material development specialists and involving teachers) can help ensure appropriate and high quality materials exist. 7. Provide strong oversight and support for materials to reach departments and schools. There should be strong communication between the central office and departmental office, as well as attention on the part of the implementers, to make sure that the materials are printed and reach the departments. Additionally, there must be solid and feasible departmental MOE plans to get them to the schools and include orientation on their use. Strengthening technical capacity at various levels is required such as 50 DIDEDUC, as is the validation of classroom materials with the support of linguist specialists. The validation concept should be used in teaching and in the training process. 8. Provide teachers with concrete didactic approaches and strategies and example lessons in K’iche’, Mam, and Spanish (depending on the particular need). The project is planning on providing teaching resources to teachers when they complete their professional development. It is important to assure that these resources include didactic strategies and approaches adaptable for the chosen model (L1/L2). Involving teachers in this process is recommendable in terms of ownership and can enhance their professional development. For example, as part of the coursework, teachers can develop, pilot and share lessons/strategies with each other and receive feedback from peers and the professors/tutors. Furthermore, it is recommended that teachers be provided with tools/materials to develop the Mayan language they must know for bilingual instruction; this could be done in coordination with the ALMG and/or other language specialists. 9. Implementation of the SINAE is fundamental for the EBI model to be effective. One fundamental reason for this is to create MOE departmental positions to assume the managerial and administrative duties that CTAs are now in charge of so that they can effectively accompany teachers in the classroom. Furthermore, when SINAE is implemented, it is important to assure that departmental MOE staff who have received training are put in positions aligned with their newly acquired skills. 10. Help ensure that there is a shared vision of the purpose and use of instruments for teacher accompaniment. This is essential for such an instrument to be adopted and institutionalized by departmental MOEs and become sustainable, especially as SINAE begins to be implemented. This common vision about how to systematically capture changes in teaching practices as well as having the right support instruments for coaching should include the implementing organizations (including the three universities), CTAs, teachers, as well as central and departmental directors. 11. Work with the MOE and other entities to increase the number of teachers who are bilingual and biliterate with Mayan languages. The shortage of teachers who are able to teach in Mam and K’iche’ (and other Mayan languages) is a widely recognized limitation. Efforts to support local strategies to address this should be supported and promoted, such working with and encouraging school principals to assign Mam/K’iche’ speaking teachers to the lowest grades when necessary and encouraging (and perhaps supporting) teachers develop/strengthen their Mam/K’iche’ language capacity. Working closely with the ALMG and universities may be a way to devise structural approaches towards improving this situation. 12. Capture the positive changes taking place from the different interventions whether initial project goals or unintended positive effects. Examples of practices could include changes in CTA and teacher attitudes and practices, action research carried out, types of involvement of parents in the project and changes in their attitudes and practices, learning communities, and more. Anecdotal information and patterns/trends should be documented, both of which are valuable for expanding and sharing best and effective 51 practices to inform the project, other MOE interventions, and scaling up. Documenting these could be worked into the professional development courses and/or done through a systematic plan with Departmental MOE personnel. The instrument being used in El Quiché by the CTAs could be helpful for this. 13. Support partner organizations and implementers, especially MOE and universities, to work dynamic bilingualism/translanguaging into classroom practices. There are concrete pedagogical approaches and strategies that could be considered for encouraging and supporting the use of translanguaging practices (or dynamic bilingualism) when appropriate, which is especially promising in an environment with such a high number of bilingual children such as in various target municipalities. This could be worked into the professional development provided by LLP as well as other in-service and pre-service teacher education programs. 14. Sensitize parents and society in general regarding the importance of children learning in their primary language and the benefits of bilingualism. This is something that must be continually and systematically worked on to develop understanding and support for EBI. More awareness can be developed at a national level through actions in coordination with the MOE central and departmental personnel, community-based actions (including through the work with parents in sub-component A.3) as well as through other actors and mediums. 15. Define strategies for the expansion of socio-linguistic models for other regions of the country along with other institutional entities that have such a legal mandate. Working with Mayan organizations (K’iche’ and Mam) that can contribute to this from their own culture, mechanisms, strategies and systems may be helpful in this regard (such as some practices emerging in Component B interventions). 16. Develop instruments (similar to the linguist profile) for teachers to assess the development of children’s written and oral linguistic abilities (in L1 and L2, depending on the class’ model) for the early grades (pre-primary through third). While there are standards that indicate the language abilities that students should develop during the school year (including for the early grades), there is not a similar instrument to for the teachers to assess the progress of the students over the school year for grades two and three nor guidelines for this to be done in pre-primary and first grade at the end of the school year (and potentially mid-year as well). This is important in order to capture individual child progress and adapt instruction accordingly. It is also a valuable formative evaluation tool and feeds into the professional development teachers receive(d). 17. Define the role of school principals in LLP and include them in professional development and other interventions. It is highly recommended that school principals participate more directly in the project interventions, especially through participating in professional development. If their role is to provide oversight, decision-making, and support to teachers and the EBI literacy improvement interventions it is important that a strategy for their involvement be included. 52 18. Assure high enrollment in teacher professional development courses for EBI. Getting the out announcement for the teacher professional development (implemented by the UVG) with adequate time and assuring that teachers are both aware of deadlines and the process is important to assure that enrollment level is high in the future cohorts. 19. Develop strategies to improve the e-learning process for teachers and other MOE technical staff. The use of technology and e-learning are innovations are promising in terms of forming a long-term strategy for teacher professional development both in the five departments and perhaps as part of a national model. One potential way to overcome some technology difficulties faced in the program (such as lack of access to internet) is the use of tablets with all of the modules installed prior to disbursing them to teachers. 20. Analyze recommendations from the evaluations of the professional development programs implemented by the universities as part of LLP. These evaluations are almost finalized. It is highly recommended that the findings and conclusions from theses evaluations be analyzed and discussed with and between the implementing organizations as necessary to make decisions (i.e. Juárez, MOE, university representatives, etc.). 21. Help to redefine/broaden the role of the consortium of universities as that of an advisory board for various aspects of teacher education. In the future, the role of the Consortium could be to document and consolidate the experience of the teacher educator professional development program as well as to serve as an advisory body for universities and the MOE in areas such as pre-service teacher education, coaching, intercultural and bilingual education, education leadership, educational measurement and evaluation, and development of support materials. 22. Assure that the subcomponent of strengthening parental involvement in education achieves it desired effects. To do this, links must be further strengthened between the quality and community education units (central and departmental levels) as well as assessing the degree to which this intervention is functioning in the schools as it is intended to be functioning. 23. Improve aspects of the LLP databases. Though it was possible to work with the databases provided by the implementer, there are some elements of them that could be improved. First, there is not a directory or catalog of the database; it would be helpful, for future users, to create a catalog. This catalog should include specifications for each database (datasheets). In addition, the databases did not include codebooks. Codebooks are extremely important because they help to understand the structure of the database as well as allowing to put the databases in any format other than SPSS. Moreover, within the databases there were variables, values and missing values that were not properly labeled; this reduced the team’s ability to understand the variable and it values. A couple of databases (e.g. teachers pre-surveys and students pre surveys) were in a raw format that required data cleaning in order to do some analysis. By correcting these issues, analysts (other than LLP researchers and analysts) will have a clear understanding for future uses of the databases. 53 24. Examine changes in reading outcomes we use the databases of LLP Longitudinal Study. For this midterm evaluation we use a quasi-experimental design with the Propensity Score Matching (PSM) technique. This technique permitted us to conform groups (treatment and control) with similar characteristics, thus establishing differences in the outcome variables (reading abilities), presumably, due to the LLP intervention. However, we think the effects we found are not strictly attributable to LLP (rather partial attribution). This is because the longitudinal database for LLP does not have a design that allows for identifying a representative treatment group of teachers who have all received the same interventions. For example, some teachers have received only the professional development, others have received only CTA visits, some have received both, some have had more parental engagement as a result of LLP than others, and some have not received any interventions even though they form part of the treatment group. In addition, the control municipalities seem to be (positively) contaminated because part of the intervention is taking place at the department level. More specifically, technical personnel from the departmental MOE directorates (supervisors, CTAs, and others) have received certain benefits from LLP. Thus there is a high likelihood that these beneficiaries produce a spillover effect on the control municipalities. PSM is a useful way of controlling for these intervening variables and obtaining more reliable causality outcome data. 25. Further examine quantitative data to verify/identify variables that have a positive or negative influence on reading abilities. Many of these variables are likely within the scope and capacity of the project to influence, such as promoting pre-school attendance, having books at home, encouraging homework assigned by teachers, students getting help at home with homework, and encouraging reading. Most of the aforementioned are already part of LLP and can be further emphasized and built upon. Some of these variables may be outside the scope and ability of the project (like children helping with tasks at home) but can be included in themes discussed with the parent learning groups. Additionally, these variables can be further explored as well as additional relevant variables that could shed light on complementary interventions for LLP and the MOE in general. Component B: Adjustments to LLP by Implementers The most significant change made by the implementer for out-of-school youth has been to decrease the target number of youth participating. Initially, the goal was to reach 11,000 youth. Given the challenge of reaching such a large number, the goal has been reduced to 7,578. Juárez and Associates has explained that this reduction was adjusted because of the difficulty to find youth available to participate with LLP. The implementers and partners adjusted to this by going door-to-door to detect the need, which helped find and involve more OSY. It is important to note that many young people have to migrate looking for income to other areas inside Guatemala, working in crop fields, trades, or urban commerce. Some youth showed a lack of interest in participating because they don´t see alternatives to change their actual situation. (e.g. as an informant expressed: “We are poor, and we will remain always poor”.). Also, the 54 actual cost of safe transportation to the training and education sites was greater than the anticipated budget, which was adjusted for. There are more than 5,000 youth enrolled in LLP. To recruit those and follow up with each one has been a difficult process. However, there are indicators that will permit following the individual student progress reports, with the performance measurement of reading, numeracy, and life skills, and workforce development. Nevertheless, according to anecdotal information from technical staff, a number of youth may be identified who have dropped out, some temporarily and others seemingly permanently for various reasons. Some youth seem to have pursued temporary, seasonal work opportunities in agriculture and others have migrated to the Mexican border to take advantage of the informal economy around the decreased value of the peso (buying and selling products). Still currently there is no formal tracking system in place for each participant. This project requirement needs to be reinforced to organize a system to assist in the success of all youth enrolled. FUNDAP and Grupo Ceiba offer trainings through the project that also have been improved by adapting better to the life circumstances of out-of-school youth. In particular, to fully take advantage of their trainings, the youth need to incur significant costs for transportation, costs that they simply cannot absorb on their own, and LLP approved a transport subsidy. Program participants and facilitators note that one of the reasons for low consistent student participation may be that adequate systems are not in place to offer enough resources for transportation, particularly for young women who may have to travel alone or in small groups. Parents sometimes are not convinced that it is safe for their daughters and sons to commute and therefore do not give permission for their children to participate. Even though LLP has given transport subsidy, implementers ask for more actions to resolve these issues, like involving parents with some training. Educational and functional standards have been developed by the providers of basic alternative education and workforce development. The organizations are conscious (as demonstrated by their self-evaluations) that they need to improve their performance. LLP hired a consultant to visit each entity and has implemented a process of follow-up. This process of improving standards is moving forward and is in its initial stage of implementation. These changes were voluntarily adopted, and a critical next step is for the Boards of Directors also to apply the standards, which may be a challenge. The project has realized significant accomplishments in a short time, but more time is necessary for the organizations to fully implement the successes into regular operations. Another requirement of the Scope of Work is to identify the entities providing basic education and workforce training for OSY and what the quality and scope of those services are. One finding of LLP were weakness of entities offering those services, and the need to improve the instructors’ ability to teach and to guide the OSY. To address these needs, the LLP organized a Diploma and a Master’s degree with the Professional School of Teaching Middle School (Escuela de Profesorado de Educación Media) EFPEM, and began a first cohort. After a hard-won beginning, this began on July 2016 with 105 students on Diploma Program and 25 on Master Program. This program creates a new and significant opportunity for EFPEM and for technical instructors from all the country because there are not any universities who offer this field of study. 55 This first cohort has only two semesters under its belt and still needs more time to complete the two-year curriculum, so it is necessary for LLP to finance the next one and a half years to finalize their studies and measure the results and impact on the teaching and learning process. This is a new career design with a quite innovative method, and should be part of EFPEM future educational careers offered and financed with their own resources and students fees if its demonstrated effective for students and technical education entities. LLP is building this new educational space, but currently it does not have enough time to develop and demonstrate effectiveness. Since LLP is working with minors outside of school (between 15 and 17 years old) and in consideration of the potential risks of their involvement due to external factors, it appeared necessary to establish protocols for avoiding gender-based violence and for travel, lodging, physical safety, and risk management. These protocols were not part of the original design of the LLP and considering some of the evaluations of the risks for minors and adults participating in LLP; such protocols were developed and are in effect. Now it is necessary to ensure knowledge and compliance with the protocols. LLP promotes the participation of youth in public spaces in the municipalities and communities. To achieve success in this civic participation space, adults have to be convinced that youth have a right to have opinions and to express themselves openly. LLP has encountered positive responses from some municipal authorities but not from others, depending on the mayor and the municipal council. The relationship with each municipal authority has to be analyzed on a case-by-case basis, and each one may require a specific approach to youths’ civic participation. USAID has invested time and funds to enhance knowledge and skills of municipal human resources, but with each new government team, those efforts have to be repeated. Thus, LLP has had to apply different strategies to aboard each municipality authorities. One of LLP’s successes thus far is that it has defined ways to inform and communicate with all mayors within the covered geographical area; this implied LLP has collaborating with other USAID-sponsored programs such as Health and Education Policy Plus to achieve positive results with regard to communication and municipal commitment to OSY. Through LLP, the youth have been organized in their municipalities, although with the passage of time, their continued involvement will become more apparent. LLP receives support from the advisory council, which seeks to support organizations working with youth and to involve local authorities so that the issues of local youth will be addressed. LLP has collaborated with these other organizations to work toward common goals. A successfully reached goal, at present date, has been to incorporate the OSY, because before only school-enrolled youth had been involved in these efforts. The adjustments applied with municipal authorities have demonstrable results. LLP is contributing to the active civic engagement of young women, giving them tools and reasons to delay marriage and motherhood. This is a substantial positive change since most young women in the project area traditionally beginning to form their own families at 14 or 15 years of age. This modification (also based on LLP’s theory of change) relies upon changes in perception (including self-perception) of women not just as wives and mothers. To produce sustainable long-term changes with regard to marriage and motherhood, the role of and examples set by the community personnel working with LLP is crucial. Based on informants’ statements, it 56 appears that there are different levels of recognition and acceptance of members of the technical team. Youth and parents explained in interviews that they preferred persons from their own municipal communities, instead of others, even when they speak their own language. From example, persons from Momostenango and Santa Maria Chiquimula prefer home visits from people that are from the same municipalities over people from Totonicapán or Santa Cruz del Quiché. It is important to pay attention to the potential mistrust of participants and address it accordingly. To incorporate youth participation and service with their own communities has been difficult. LLP has made strides to achieve community acceptance. Several informants suggest including forms of organizing and participation that build on the vision and existing modes in indigenous communities to obtain more buy-in and sustainability. A number of existing civic-minded practices may be expanded and shared with other indigenous communities. Component B: Recommendations In accordance with the broad lines of inquiry established in the scope of work document for out of school youth, these are the suggested recommendations: 1. Establish a cohort-segmented report to track youth educational and occupational processes. To fulfill these requirements it is necessary to establish an ID or bar code for each youth, and track everyone through the different steps they take. Adjustments to the enrollment forms and process should be made, as well as support with technical equipment to follow up on their participation. If USAID requires the specific tracking system to show results with LLP about OSY, it will be necessary to invest funds in implementing a strong system. The implementation of SIREEX with MOE will not give enough information in the short time (one or two years), and to have quantitative data for new proposals this system has to be fully implemented. There are successful young participants who will demonstrate a positive effect with basic education, workforce training, leadership and community participation programs; these elements can be captured by such a system. 2. Labor market/demand side. Include the USAID/Guatemala´s Feed the Future Strategy, where it establishes: “Guatemala has tremendous potential for expanding its agricultural production.” Incorporate into the workforce technical courses such as agriculture, horticulture, animal production, and even arts and music (as expressed by youth in the PYA) using existing programs, expertise, and curricula. LLP can join efforts with the Feed the Future Strategy Value Chain program who can instruct on innovative ways of transferring technologies (e.g. drip irrigation and use of low tunnels) for production of high value horticulture crops for use within home-based gardens, while also encouraging nutrition counseling for producers and workers. Additionally, LLP can work together with other U.S. Government supported projects in the Agricultural sector (such as Infant & Young Child Nutrition Project and Nutrition and Food Security Impacts of Agriculture Projects) for improving child rearing and feeding patterns, improving access to health services, reducing morbidity 57 (through hygiene and sanitation) and promoting home and school garden activities (including poultry and dairy activities). Peace Corps volunteers could be employed to organize those training´s. Additionally, with the Ministry of Culture and Sports and the Department of Culture, LLP can promote musical activities in the indigenous languages and stimulate new artists and talents. LLP has close relations with the municipal governments and can organize theaters, puppets, concerts, musical and songs contests, and mural paintings with OSY trained in those fields. 3. Basic Education provision. Continue supporting the entities actually providing basic education to OSY, with the standards, consultants, and other resources. Include the Ministry of Education Program for Over Age Population or Programa Educativo para Población en Sobre Edad (PEPS), which already operates under its Department of Bilingual Education’s (DIGEBI) Office of Attention to Vulnerable Populations. This program consists of intensive, more relevant elementary education, with educational content in the K'iché and Mam languages. To include PEPS in the educational offer, in other municipalities, will allow more youth who have not completed primary school and are over age of attend formal school. Given the strong communication LLP maintains with the local educational offices, PEPS could be implemented in the municipalities out of Totonicapán that are covered by the project. The DIGEBI´s official in charge of PEPS believes that LLP could join DIGEBI to apply the program in the project municipalities. 4. Workforce skills provision. Recognizing how difficult subcontracting mechanisms are, it is suggested to seek (a non-financial) partnership with others organizations for coordination and mutual learning. These entities include organizations such as the Barbara Ford Center for Peace (in Santa Cruz Quiché), the K'iché Technological Institute (in Chichicastenango), the Center of Rural Formation UTZ SAMAJ in Tecpán, Chimaltenango, and others. More could be included, establishing a memorandum of understanding with each one, with USAID acceptance. 5. Incorporate Peace Corps volunteers in institutions/organizations working with LLP interventions: This could contribute to strengthening elements of Guatemala´s rural development. Examples include integrating Guatemalan financial literacy and microcredit organizations into workforce development and numeracy education. Credit and saving cooperatives have educational programs and will be interested in supporting and educational materials and workshops. 6. Ensure that indigenous cultural values are sufficiently included in activities, materials, etc. supported by LLP: Integrating existing community values and practices and using them as a platform for growth may allow LPP to promote prevention of crime, violence, and illegal activity in a culturally relevant way. There are organizations that have already developed applicable methodologies and materials that can be learned from (such as CARE, SHARE, Catholic Relief Services/CRS, Save the Children, and Junior Achievement (Empresarios Juveniles Guatemala). 58 7. Sexual and reproductive health. To increase involvement and ensure future sustainability, the vision, culture and perceptions of indigenous people should be further integrated into all curriculum, including sexual and reproductive health curriculum. LLP should continue to exchange experiences regularly with organizations that can share accumulated knowledge about how to establish positive behavioral changes (such as Population Council with their program “Opening Opportunities” / “Abriendo Oportunidades”). 8. Youth civic engagement. LLP has opened wide spaces for OSY participation in the targeted municipalities and has given voice to those youth who before rarely participated. To strengthen the OSYs community involvement, LLP should exchange experiences through workshops with municipality and departmental networks including with local government and NGOs who had been working with municipal authorities (such as Save the Children International/Guatemala, UNICEF, CARE, SHARE, CRS and others). 9. OSY strengths and buffering factors. Continue to strengthen OSYs’ networks at the municipal and departmental level, including with local government and NGOs that work with the local authorities. Youth strengths include their eagerness, courage, and desire to change their environments. Those strengths should be harnessed to collaborate with local institutions or volunteer committees to apply their development plans. LLP could establish a contest to encourage innovative activities and winning entries would receive funds to implement the best proposal(s) presented by local OSY committees. Within additional activities of the scope of work (perhaps with the availability of funds) LLP could organize a youth engagement contest, where the winning entry of each municipality would be financed and implemented (subject to USAID approval). Entries may include a song in a Mayan language, a play for youth and adults, a puppet shows for preschool children, ideas for events for youth with handicaps, mural paintings on school walls, and others. 10. Gender-based and domestic violence. In Youth Participation Assessments, the domestic and local violence issues have not appeared, but these issues were raised in conversations and interviews with teachers, instructors and tutors, as some of the barriers to youth development. It could be useful to apply the guide “For the End of Physical and Humiliating Punishment” (that has been prepared by other organizations) and arrange workshops with adults, families and local community authorities. An additional preventative action would be to implement the safety protocols, including signed commitments of ethical behavior by personnel of all the sub-contractor organizations. 11. The engagement of parents of 15-17 year old out-of-school youth should be prioritized, This includes information about the benefits of participation, how they can support the youth, and measures taken to protect their youth when they participate. Parental engagement efforts should include technical staffs’ knowledge and application of security protocols, and youth and staff codes of conduct should be developed and implemented to strengthen positive behavior and trust with parents. 59 Medical and life insurance should be provided to participating youth who travel outside their municipalities for training and education. Parents should be informed of security protocols, codes of conduct, and insurance. Properly communicated, these protections should increase parental participation in their children’s learning, support for their children’s involvement, and related safety processes. 12. Parental and community involvement in youth´s decisions. LPP should continue to incorporate spaces for parents and family members of the OSY into the activities of LLP through activities such as hosting family events. Even though they have been included in events like rural Meeting Points, inter-generational talks, and community events, some parents expressed a desire for stronger communication and relationships with tutors and mentors. Events like workshops in native languages incorporating indigenous cultures, traditions and stories will surely be viewed positively by parents, which in turn will promote recognition for youth and women decisions. Full acknowledgement, embracing, and incorporation of the vision, culture, and civic traditions of indigenous peoples may contribute to greater acceptance, ownership, sustainability and relevance of parental and community. To that end, it will be useful to establish links with indigenous institutions and organizations that have experience in adult involvement and change of behavior and attitudes (such as the National Council of Mayan Education, Rigoberta Menchu Foundation, and others). 13. Strengthen general external affairs operations. LLP should link with a number of new or existing GOG programs within the Ministries of Economy, Labor and Education, to exchange experiences focusing on OSY, instead of only schooled youth. Clarify and refer to OSY in all communications with municipal, departmental, and central authorities of the Guatemalan government as the explicit responsibility of the Out-of-School-Youth. Develop a more robust social behavior change communication strategy, with MOE collaboration. DIGEEX has functioning community radios and FEGER has a radio network with youth rights programs. 14.Strengthen the operation of SIREEX with personnel hired by the LLP project. Preferibley they would be working inside DIGEEX with the staff to track students registered in the non-formal education system where hard indicators should be defined to measure, record, and identify success or failure of students. This system can be tested and monitored with OSY registered in LLP. This system can track all students registered in NUFED, CEMUCAF, PEAC, ICT Learning Centers, Academies of Free Courses and other organizations operating within non-formal education system; this can help to track the route and progression of enrolled students as well as institutions that achieve the best results in improving the livelihoods of students. It would useful to share experiences with other educational projects that have such systems, some of which include monitoring graduated students, hard indicators, and formats for monitoring. 60 VII. APPENDICES 61 Appendix 1: Sample of Data Collected for Midterm Evaluation of Lifelong Learning Project Method Sample Artifacts/ Documents Components A and B Interviews (n= 4) Secondary analysis of LPP databases  Juárez & Associates, Chief of Party LEA  USAID Guatemala, COR of Project LEA (# I =2)  Plan International, Director  Teacher and students from the intervened and control municipalities of LEA longitudinal study. Support documents (Plans, Reports, etc.) Component A Total # of informants =154 ---- Implementing partners=28 Direct beneficiaries =123 Others=3 Interviews (n= 11)  Juárez & Associates, Chief of Party  Department Directors of each of the 5 departments  Univ. Panamericana, Director de Especialización Lic. and Pos-grado en Lecto-Escritura en Ambientes Bilingües e Interculturales (Beca Frank Fairchild)  Vice Minister of Bilingual Education  Member of team carrying out mid-term evaluation of Del Valle LEA teacher training  Vice Dean of Education of Universidad Rafael Landívar (URL), member of the Consortium and Tecnical Secretary of GCNPE  Education Dean of UVG and person responsable for the Diplomado en Lectoescritura en ambientes Bilingües e interculturales Focus groups (n=20)  Juárez & Associates, Chief of Party LEA and component A team members (# P=3; # FG=3)  Plan International, LEA coordinator and team members (#P=3)  University Consortium (#P =3)  Juárez & Associates Regional Director and Comp. A team members (# P=3; #FG =2)  Technical sub-director; Teacher Ed. Coordinator and Primary School Coordinator with scholarship from Frank Fairchild (# FG =1)  CTAs y Teachers of Elementary School (# FG =1)  Technical pedagogic personal staff (# FG =5, LEA depts.)  Parents of children in intervention schools (# FG =3: San Marcos, Totonicapan, El Quiche)  Advisor to the Minister of Education; Directors of DIGEBI, DIGEFOCE, DIGECURE, DIGECADE (# P=5)  Universidad de San Carlos, Team responsible for the Especialidad en Liderazgo y Lecto￾escritura (P=2) Coaching instrument Spanish and Mayan language (Quiche, Mam) literacy tests 1-4 List of schools where other donors are also working School visits (n= 3) School Visits (# SV = 3); Totonicpan: Momostenango (# V=2) and El Quiche: San Pedro Jocopilas (# SV = 1) Photos of materials, class, student work. 62 Method Sample Artifacts/ Documents Secondary Analysis12 Data bases: First grade initial sample, n = 855; final sample n = 298 Second grade initial sample, n = 1391; final sample n = 754 Third grade initial sample, n = 1460; final sample n = 931 Fourth grade initial sample, n = 1424; final sample n = 890 Method Sample Artifacts/ Documents Component B Total # of informants =126 -- Implementers =27 Direct beneficiaries = 81 Parents = 18 Interviews (n= 11)  Vice Minister of Bilingual Education  Juárez & Asociados technical staff at Quetzaltenango Office  Technical staff of Ajkemab organization at Santa Cruz del Quiché and Quetzaltenango (#P =5)  Technical staff of Tierra Nueva organization, from Huehuetenango meeting at Quetzaltenango (#P=2)  Technical staff of Grupo CEIBA, in Momostenango (#P =2)  Technical staff of FUNDAP at El Refugio Educational Center, in Quetzaltenango (#P =2)  Director of DIGEEX  Departmental DIGEEX Coordinators from Quetzaltenango, Huehuetenango and San Marcos (#P=3)  Child Fund, Director and technical team (#P=5)  ACUDE, staff or MEDUCO, flexible education project in Totonicapán (#P=2)  EFPEM-USAC Staff of Master and Diploma for Teachers in Innovative Education (#P=2)  Opening Opportunities Director of Population Council Quality standards for alternative basic education and work training for youth Form for registering out-of￾school youth Focus Groups (n=13)  Youth participant in Foro´s in Santa María Chiquimula, Totonicapán and Joyabaj, Quiché (#P = 30)  Youth participant in Meeting Point at Paraje Chicaxtun, Totonicapán (#P=21)  Youth participant in Human and Technical Development Centers in Momostenango, Joyabaj and San Pedro Jocopilas (#P=7)  Youth participant in technical education in FUNDAP (#P=10) Photos and videos of the youth in workforce training (e.g. Automotive mechanics, 12 Initial sample refers to the total number of students in the original databases provided by LEA whereas final sample is the total number of students after using Propensity Score Matching technique. 63 Method Sample Artifacts/ Documents  Youth with their parents participant in parents meetings in San Juan Ostuncalco and San Pedro Jocopilas (#P=13)  Juárez & Asociados staff youth responsible at Guatemala City (#P=6)  Technical staff of Child Fund (#P=4)  Parents in meetings in San Juan Ostuncalco and San Pedro Jocopilas (#P=18)  FUNDAP (#P = 2)  Juárez & Associates, Chief of Party LEA, Coordinator of comp.B team, and M&E specialist (#P=3) cosmetology, clothing design/sewing, cooking, etc.) Descriptive Analysis Databases and list of information  Mathematics, reading and financial test  Youth out of School program records  Trainers-of-Trainers records of participation Notes: 1) # P= number of participants; # I = number of interviews; # FG = number of focus groups; # SV = number of school visits. 64 Appendix 2: Quantitative Data on Out of School Youth (OSY) The evaluators received three files: 1) the registration list with 5,259 OSY names, ages, approved studies, and family information, 2) the trainers-of-trainers list, with 392 names and the workshops conducted, and 3) test results of 589 students in functional mathematics, functional reading and functional financial skills workshops. The first file simply provides names, some with ID and others without them, the second and third file do not include IDs. Thus it is difficult to trace each young within the others files. The second file includes 392 youth who are trainers-of-trainers. They participated in four workshops designed to teach them how to teach their peers’ life skills, citizen participation, public policy, and sexual and reproductive health. According to the attendance records 118 (or 30.10% of the total 392 trainers) participated in all four workshops, 18 (4.59%) participated in three workshops, 197 (50.26%) participated in two workshops, and 59 (15.05%) received only one training. There is no data in this file to indicate whether or how many youth subsequently received training from those 392 trainers. The third file includes information with the test results of 589 students enrolled in flexible education programs and workforce trainings. The results reflect performance in functional mathematics, functional reading and functional financial skills. The difficulty here is that the data gathered in this file does not facilitate establishing attainment of goals or bench marking. If another test is administered in the future, then there should be data to compare and establish earning skills. Another data source provided is a table of students registered with two organizations providing flexible education programs. First, there is a list of 249 youth receiving technology training and 164 receiving flexible education programs in the 7th, 8th, and 9th grades through Asociación Grupo Ceiba. Second, there is a table of 496 students registered with (ACUDE) who are receiving flexible education programs in the 2nd, 3rd, 4th, 5th and 6th grades over two years. But these students seem to be disconnected from other LLP activities and organizations, because their schedules do not allow time for working with tutors and also participating in workshops, forums, and other meetings.” According to the available quantitative data, there is no way to assess whether the students are making the progress required by the donor and there appears to be no tracking system in place to follow the youth, their educational and workforce training, and their employment progress. At present, there is still out of time to have employability data because work force training programs have recently (September 2016) finished. Each training institutions have their own registration and progress record, but there will necessary to standardize the process to have the same information from each one. More resources (human, technical, and financial) are necessary to establish and track over time the relationship between interventions and outcomes for participating OSY. Monitoring and evaluation teams, including with contract and grantee organizations and sub-organizations, need additional to develop tools and may need technical assistance and training to do so. 65 ACUDE Interview and Main Findings ACUDE provides flexible education programs for youth who have not finished primary school. These youth are generally over the age to be able to enroll in formal schooling, and typically they do not desire to go back to regular schools. The program ACUDE applies is the same used by COMITÉ NACIONAL DE ALFABETIZACIÓN (CONALFA) and consists of two years of studies with a tutor and with materials in Spanish. There were 61 groups in four municipalities in Totonicapán. These groups have a total of 496 students who are very motivated to finish their studies and receive their diplomas. In fact, they want to continue their studies, and they are proud to be part of LLP. The tutors estimate the majority will pass this year, but actual passage rates will not be available until final exams are completed in November. ACUDE tutors and technical staff recognize the need of OSY to receive workforce training and establish by their own initiative a training course to become Tourism Guides with INSTITUTO GUATEMALTECO DE TURISMO (INGUAT). Actually there are 17 youth the course, 15 girls and 2 boys in Momostenango. ACUDE is not working with students in others municipalities where they have no formal presence. So it is likely that youth in other areas covered by the project are not receiving this intervention. With LLP subvention CONALFA has received a strong support, helping to revise and develop a new and updated version of their educational materials, according quality standards prepared by LLP. This action will benefit all students covered by CONALFA for 2017 and beyond. ACUDE has been the bridge between LLP and CONALFA, and will contribute with other educational institutions like Fundación Ramiro Castillo Love FRCL. LLP is not financing all of the costs associated with ACUDEs work with the OSY. Fundación Ramiro Castillo Love (FDCL) is paying part of those costs. It will be necessary to reorganize the budget for the next years, to agree with FRCL if LLP wants ACUDE to continue working with its OSY. There is a close and friendly relation to presume there will be interest and commitment to continue working together, through ACUDE and LLP agreement. 66 Appendix 3: Explanation of Quantitative Data Analysis 1) A PSM technique to conform control and treatment students’ groups. This was done for each grade, except for first grade. 2) We fit the Diff&Diff model to the form ௜ܺ௜ߚ෍ + ௜݋݌ܶ݅݁݉ × ݐܽݎܶଷߚ + ௝݋݌ܶ݅݁݉ଶߚ + ௜ݐܽݎܶଵߚ + ଴ߚ = ௝௜Y ௣ ௜ ௜ߝ + where Yij is the reading score for each time j Trati is the treatment condition of the students ( 0 = students from control municipalities, and 1 = students from intervened municipalities), Timej is the variable time (0 = beginning of 2015 and 1= end of 2015). Trat x Tiempo is the interaction of the treatment condition and time. Xi are all the control variables included in the model. β1, β2, and β3 capture the effect of the variables Trat, Tiempo, and Trat x Tiempo, and the coefficients βi capture the effect of each control variable Xi. β3 is the coefficient of interest because it capture the differences between the groups of students. Finally, εi is the unique random error for each student. 3) We estimated the percentage of students who improve their abilities. This was done for the treatment and control groups. 4) We again used PSM to match teachers. Thus we get a treatment and control groups of teachers. 5) We fit several HLM models (with fixed effects) to determine differences in between control and treatment teachers. In addition, a set control variables were use in the models to get robust results. The model use is of the following form Level-1. ܻ௜௝ = β଴୨ + ∑௤ β୯୨ܺ௤௜௝ + ݎ ௝௜ and level-2 ߚ = ௝଴γ଴଴ + γ଴ଵܶ௝ + ∑୮ γ୮୨ܹ௣௝ + ݑ௝௣. Where Xqij represent the set of control variables at level 1 and βqj captures the effect of each control variable. Tj represents the treatment condition (0 = teachers in the control municipalities and 1 = teachers in the intervened municipalities). Wpj are the control variables at the teacher level. β0 captures the mean of each teacher. γ00 represents students’ gran mean scores. γ01 captures the treatment effect, γpj captures the effect of each Wpj. Finally, rij ane uij are de random errors of level 1 and 2, respectively. 6) With the matched teachers, we constructed a set of latent variables by using the Rasch model. The latent variables were: a) activities at the library, b) reading comprehension activities, c) participation in learning communities, d) teachers background, e) teachers abilities, f) evaluation methods, g) evaluation tools, h) abilities in Spanish language, i) abilities in Mayan language, j) used of materials, k) parents’ involvement, l) vocabulary activities. 67 7) Once again, a Diff & Diff models was fit (the models have the same form as before), but this time the latent variables were the outcome variable. In addition, the propensity scores were used for weighting the regression outputs, and a set of control variables were included. 8) All models were fit for second, third, and fourth grade, except for the models with the latent variables. In those models first grade was included. 68 This page is intentionally left blak 69 Para mayor información contactar a: DevTech Systems, Inc. 1700 North Moore St. Suite 1720 Arlington, Virginia 22209 703-312-6038 www.devtechsys.com