1 Final Performance Evaluation of the USAID/Guatemala Lifelong Learning Project September 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by SSG Advisors, d/b/a Resonance. It was authored by Virginia Lambert. The research team consisted of Virginia Lambert (Team Leader), Armando Secaira (Education Specialist), Luis Crisostomo (Education Specialist), Jenniffer Lopez (Translator/Coordinator), and Daniela Cardozo (Analyst/Research Assistant). FINAL PERFORMANCE EVALUATION OF THE USAID/GUATEMALA LIFELONG LEARNING PROJECT This publication was produced for review by the United States Agency for International Development by SSG Advisors, d/b/a Resonance, through USAID Contract # AID-OAA-M￾14-00024. This report was prepared by: Resonance 1 Mill Street, Suite 201 Burlington, VT 05401 Resonance Contact: Carrie Conway Director Tel: (802) 578 4780 Email: cconway@resonanceglobal.com The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Resonance (formerly SSG Advisors) is a woman-owned SBA-Certified 8(a) small disadvantaged business based in Burlington, Vermont with a global presence. By harnessing the power of collaboration among companies, communities, and government, Resonance is improving lives and achieving transformative impacts around the world. CONTENTS Acronyms........................................................................................................................................................... 1 Executive Summary .......................................................................................................................................... 3 Introduction ....................................................................................................................................................... 6 Evaluation Purpose and Evaluation Questions........................................................................................... 6 Project Background.......................................................................................................................................... 8 Evaluation Methods and Limitations............................................................................................................. 9 Evaluation Questions: Findings and Conclusions.................................................................................... 12 Conclusions ..................................................................................................................................................... 32 Annexes ........................................................................................................................................................... 33 - Annex I: Evaluation Statement of Work................................................................................................ 33 - Annex II: Data Collection Instruments................................................................................................ 40 - Annex III: Sources of Information ........................................................................................................... 57 o List of Persons Interviewed o Bibliography of Documents Reviewed - Annex IV: Disclosure of Any Conflicts of Interest.............................................................................. 69 1 ACRONYMS ANAM Asociación Nacional de Municipalidades ACUDE Asociación para la Cultura y el Desarrollo CDCS Country Development Cooperation Strategy CEIBA Asociación para la Promoción y Desarrollo de la Comunidad COCODE Consejo Comunitario de Desarrollo CODEDE Consejo Departamental de Desarrollo COMUDE Consejo Municipal de Desarrollo CONJUVE Consejo Nacional de la Juventud CTA Coordinador Técnico Administrativo DIDEDUC Dirección Departamental de Educación DIGEEX Dirección General Extra-Escolar EBI Educación Bilingüe Intercultural EQ Evaluation Question FID Formación Inicial Docente INAP Instituto Nacional de Administración Pública J&A Juarez and Associates KII Key Informant Interview LAC Latin America and the Caribbean LLP Lifelong Learning Project MOE Ministry of Education MOU Memorandum of Understanding NGO Non-governmental Organization OMJ Oficina Municipal de Jovenes OSY Out-of-school Youth PADEP Programa Académico de Profesionalización Docente POA Plan Anual de Operación PRODESSA Proyecto de Desarrollo Santiago SEGEPLAN Secretaría de Planificación y Programación de la Presidencia SIAF Sistema Integrado de Administracion Financiera SICOIN Sistema Integrado de Contabilidad SINAE Sistema Nacional de Acompañamiento Escolar SOW Statement of Work TEC Total Estimated Cost 2 UPANA Universidad Panamericana UNICEF United Nations Children’s Fund USAID United States Agency for International Development USAC Universidad de San Carlos UVG Universidad del Valle de Guatemala WH Western Highlands (Altiplano Occidental) 3 EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Guatemala Lifelong Learning Project (LLP), which has been implemented by Juarez and Associates (J&A) since March 2014, has two broad components designed to address two disparate sets of education challenges in Guatemala. Component A, Learning to Read, builds on a history of USAID collaboration with the Ministry of Education (MOE) to address the requirements for early grade reading and writing in the context of bilingual intercultural education (EBI, Educación Bilingüe Intercultural) in rural schools characterized historically by low rates of achievement and high rates of desertion. Component B, Education for Employment and Lifelong Learning, was developed as a new innovative activity, which focused attention on a previously invisible segment of the rural population, youth ages 15 to 24 years, who had dropped out of school, many at an early age, and now have few opportunities for education or employment. The geographic focus for both components is the five departments of the Western Highlands (WH), namely, Quetzaltenango, San Marcos, Huehuetenango, Totonicapan, and El Quiche, and 12 target municipalities within these departments1 . Otherwise, the two components have been implemented separately. USAID/Guatemala designed the final performance evaluation of the LLP to assess specific aspects of the two components rather than the overall project performance, by asking the team to investigate three discrete evaluation questions for each component. The evaluation is explicitly forward-looking: “The purpose of the evaluation is to provide answers to key questions that will facilitate a technically strong program closeout and inform future USAID/Guatemala education programming.” (See the Statement of Work.) The Mission completed an extensive mid-term performance evaluation in 2016, including both quantitative and qualitative measures of key objectives, which served as the basis for a Total Estimated Cost (TEC) increase for the project in 2017. The final performance evaluation is directed particularly to the activities since the mid-term performance evaluation and the TEC, drawing on the findings and conclusions of the mid-term. The questions are primarily qualitative, seeking information on the experience of the beneficiaries and partners as input for next steps in programming. The three evaluation questions for Component A are: 1. At what level (national, departmental, municipal) do key stakeholders think USAID should focus its reading interventions? What are the opportunities and challenges of working at each level? 2. What is the perception of the teachers who received materials from LLP of the usefulness and relevance of the materials kit, and the tools to assess the linguistic profile of the students? 1 San Pedro Jocopilas and Joyabaj in El Quiche, Concepcion Tutuapa in San Marcos, Totonicapan, San Bartolo Aguas Calientes, Momostenango, Sta Lucia La Reforma, and Sta Maria Chiquimula in Totonicapan, Conepcion Chiquirichapa and San Juan Ostuncalco in Quetzaltenango, and Chiantla and Sta Barbara in Huehuetenango. 4 3. How likely is it that the participating universities will continue to implement in-service teacher training programs without USAID support? (considering resource needs, capacity, and willingness) The three evaluation questions for Component B are: 4. In what ways have the youth engagement activities been beneficial to youth participants and their communities? 5. How can USAID work most effectively with the Municipal Youth Offices (OMJ, Oficina Municipal de Jovenes)? (considering that several USAID interventions work with them) 6. What lessons can be learned from the development, validation, and implementation of the tests for functional skills in reading, writing and mathematics for out-of-school-youth (OSY)? The evaluation was carried out from July-September 2018, by a team of five evaluators contracted by SSG Advisors d/b/a Resonance. In addition to the expatriate team leader, the team included a specialist in bilingual education for Component A and a youth education specialist for Component B, as well as a translator/coordinator and an analyst/research assistant. The data sources for the evaluation were background documents on the project and on the approach to bilingual intercultural early-grade reading from USAID and J&A, and primary qualitative data collected in Guatemala City and in the five departments of the WH. A sample of seven of the LLP target municipalities was selected for interviews with participants in the two components of the program. Data collection in Guatemala City consisted primarily of key informant interviews (KIIs) with MOE officials and project implementers and partners. In the WH the evaluation team interviewed the LLP regional field staff, and technical staff in each of the five Directorates of the Departmental Offices of Education (DIDEDUCs, Dirección Departamental de Educación), as well as a series of interviews and discussion groups in each municipality, including group discussions with youth participants and leaders, and with pre￾primary and primary teachers from the targeted schools. Uniform questions were used across departments and municipalities to allow coding and analysis by themes and patterns. For Component A, the findings across the three questions pointed to a high level of satisfaction with the extent and the quality of the LLP products and outputs across all institutions and groups. For Evaluation Question (EQ) 1, the team found a broad consensus that the focus for future programs should be at the local level to integrate the use of the various tools and methods in direct support of student learning. Respondents pointed to the role of the Departmental technical staff in implementation and to the emerging role of the coaches and pedagogical advisors through SINAE (Sistema Nacional de Acompañamiento Escolar). For EQ 2, teachers praised the contents of the kit of materials, especially the reading materials for children, but also reported problems in terms of accessibility, language, and links to the LLP methodology for teaching early grade reading and writing in a bilingual intercultural context. At the same time that they seek additional tools, they also are focused on the need for coaching and guidance in the use of the materials. 5 EQ 3 addressed the question of sustainability of the LLP university-based in-service training programs designed to strengthen the human resource base in the education system as a whole. Again, there was a general consensus that the programs are excellent and important, but the universities do not have the financial resources to continue to provide the services without USAID subsidies. Component B for OSY was an innovative activity for the USAID/Guatemala Health and Education Office and involved engagement activities with rural youth as well as work with the MOE General Directorate for non-school education (DIGEEX, Dirección General Extra-Escolar). A significant contribution of this program has been to identify and draw attention to a segment of the rural population that had been largely unrecognized and invisible, the majority of whom are young women living with their families, many with children of their own. The young participants identified strong personal gains from the activities including increased self-esteem, a sense of purpose, friendships, and access to education and skills for employment and entrepreneurship. The demand for youth programs has expanded over the past five years. The challenge for future programs is to develop community- and municipal-level leaders and institutions to respond to the demand and maintain the momentum. The final EQ (6) concerns the utility and support for a major research and monitoring effort by J&A to develop a set of indicators and tests to measure the quality and utility of the MOE alternative (non-school) education program. The components of the alternative education program are functional literacy, writing, numeracy, and financial literacy, defined in terms of the requirements of the local labor market in the WH. Based on these skill sets, learning and quality standards were identified and adopted for each functional area. The next step, which is not yet complete, is to develop a system to measure, monitor, and set benchmarks for the achievement of these standards. The LLP is now in the process of collecting a large data base to finalize the items to be included in the monitoring tool and to establish benchmarks for levels of achievement. The task is seen as important (although not fully understood) for both the MOE in curriculum development and certification, and for the organizations delivering the alternative basic education curriculum to monitor compliance and strengths/weaknesses of their programs. The evaluation team developed a set of recommendations corresponding to the first five evaluation questions, directed to future programming. For Component A, the recommendations focus both on the need to channel the resources and tools for early grade reading in a bilingual intercultural context to the classroom and the students, and on the importance of applying the tools and methods as a package so that the full potential of the model can be realized and tested. They also reiterate the need not only for continuous learning and human resource development within the educational system but also for continuous guidance and support to build on the potential available through the reading materials, tools, and training. For Component B, the focus of the recommendations is on building sustainability into local youth programming through internalizing the demonstration of demand as well as the response in terms of increased opportunities. 6 INTRODUCTION USAID/Guatemala LLP (project title in Spanish, Leer y Aprender) began in March 2014 and will end in January 2020. The project has two major components, which are implemented separately. Component A, Learning to Read, the larger of the two, is focused on early grade reading and writing in a bilingual intercultural context. Component B, Education for Employment and Lifelong Learning, provides programming for out-of-school youth (OSY) ages 15-24 years, in rural settings, focused on education and workforce skills. The project is implemented in five departments of the WH, Quetzaltenango, San Marcos, Huehuetenango, Totonicapan, and Quiche, and directed to 12 municipalities within these departments. The prime contractor is J&A. The Mission completed a midterm formative evaluation of the project in 2016, which resulted in a TEC increase in 2017, to: (1) expand professional development opportunities (namely, Mayan language training for teachers and leadership training for school directors), (2) develop and distribute learning materials in Mayan languages and Spanish, (3) enhance parents’ support for reading activities at home, (4) expand alternative basic education opportunities for OSY, (5) expand the project’s research agenda, and (6) expand the gender equity and female empowerment integration in the project. Accordingly, this final performance evaluation gives particular attention to activities after the TEC and to three discrete evaluation questions defined for each component, rather than the broad assessment performed in the midterm performance evaluation. The final evaluation took place between July and September 2018, and was conducted by SSG Advisors, d/b/a. Resonance, under the LAC Education Technical Support Services Contract AID-OAA-M-14-00024. The team consisted of the following five individuals: team leader, two education specialists for Components A and B, one translator/coordinator and an analyst/research assistant. EVALUATION PURPOSE AND EVALUATION QUESTIONS According to the Statement of Work (SOW), the purpose of the evaluation is to provide answers to key questions that will facilitate a technically strong program closeout and inform future USAID/Guatemala education programming. The evaluation is intended to build on and integrate the findings from the mid-term performance evaluation and to assess key aspects of Resonance Evaluation Team 7 the LLP that were incorporated as a result of the mid-term. The LLP has a different approach for each component. These approaches take into consideration the Ministry of Education’s (MOE) structure and capabilities. In this sense, this final performance evaluation is part of a collective learning agenda. The evaluation results are intended to inform late-term adjustments to the project as well as the design of new activities. In addition, the results will be used to promote discussions among USAID/Guatemala, MOE, and other implementing partners about strategies, best practices, and sustainability. The evaluation is structured around six key evaluation questions, listed below—three to address each of two components. COMPONENT A: LEARNING TO READ 1. At what level (national – policy, departmental – education offices, municipal – education commissions and schools) do key stakeholders think USAID should focus its reading interventions? What are the opportunities and challenges of working at each level? 2. What is the perception of the teachers who received materials from LLP of the usefulness and relevance of the materials kit, and the tools to assess the linguistic profile of the students? 3. How likely is it that the participating universities will continue to implement in-service teacher training programs without USAID support? (considering resource needs, capacity, and willingness) a. What are the factors that promote or hinder the long-term sustainability of the LLP implemented in-service professional development programs for teachers and MOE professional staff? b. How could a follow-on activity be more sustainable? What are potential models which could be implemented to achieve similar objectives of teacher training? COMPONENT B: EDUCATION FOR EMPLOYMENT AND LIFELONG LEARNING 4. In what ways have the youth engagement activities been beneficial to youth participants and their communities? 5. How can USAID work most effectively with the OMJ? (considering that several USAID interventions work with them) 6. What lessons can be learned from the development, validation, and implementation of the tests for functional skills in reading, writing and mathematics for OSY? a. What is most relevant for the MOE? b. What is most relevant for youth-serving organizations? 8 PROJECT BACKGROUND The LLP was designed to build on two previous USAID education programs (Education Standards Research Program, 2005-2009, and Education Reform in the Classroom, 2009-2014), which contributed to the model of EBI, as well as a long history of partnership and collaboration with the MOE. The regional concentration of the current project, however, is a function of the shift in USAID/Guatemala strategy to increase impact by concentrating development activities only in the poorest and most vulnerable part of the country. The concentration of activities in five departments (and 12 municipalities) of the WH is consistent with the regional focus of the USAID/Guatemala Country Development Cooperation Strategy (CDCS) for 2012-2019, and the development of the WH Integrated Program to address the root causes of poverty and chronic malnutrition. The objective of the LLP is to improve education quality and increase access for underserved populations, i.e., indigenous children and OSY in the WH. The project focuses on the three Goals of the USAID Education Strategy: (1) Improved reading skills for children in early grades; (2) Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals; (3) Increased equitable access to education in crisis and conflict environments. The two components of the project, Component A: Learning to Read, and Component B: Education for Employment and Lifelong Learning have been implemented separately. The methodology for early grade reading and writing under Component A is grounded in bilingual intercultural education, which has been espoused by USAID with the MOE since the 1980s, as appropriate for a region in which the mother tongue of more than half the population is not Spanish. In the five departments served by the LLP, the two principal Mayan languages are Mam and K’iche’. The principal technical assistance activities of the LLP for Component A are increasing teacher effectiveness, improving the classroom learning environment, fostering effective first and second language acquisition and reading, and expanding parents’, communities’ and stakeholders’ participation in student learning. The primary measure of effectiveness has been outcomes in reading and writing for children in pre-primary and primary grades one to three. While the Component A activities and approach build on the lessons of a series of USAID/Guatemala projects with the MOE, the focus on OSY in Component B is new, including the definition and identification of the target group as youth ages 15 to 24 years who have dropped out of school. One of the results of Component B has been to give visibility to the needs and potential of these young women and men. The activities have focused on development of soft skills for employment, life planning, and civic participation as well as presenting opportunities for alternative education and training in applied skills for employment or entrepreneurship. ChildFund International, an NGO (non-governmental organization), 9 which has worked in Guatemala for many years with infants, children, and youth through its program of sponsorship, is the primary sub-contractor responsible for implementation of Component B According to J&A, 2 the areas of concentration since the 2017 TEC have been: (1) Consolidation of the activities to support implementation of the model for early grade reading and writing in a bilingual and intercultural context at the central and departmental levels of the MOE; (2) Distribution of studies about the experience and results of the model; (3) Support for the availability, use and distribution of the reading materials in Mam, K’iche’, and Spanish online, and use of the Bloom software; (4) Support for strengthening the in-service b-learning university programs, the training for alternative education, and the university consortium; (5) Consolidation of the activities with parents and communities with the MOE at the central and departmental levels; (6) Support to the implementation of SINAE; and (7) Continuation and growth of activities with DIGEEX and alternative education. EVALUATION METHODS AND LIMITATIONS The USAID/Guatemala SOW for the evaluation stated explicitly that the content for the evaluation should be limited to the six evaluation questions, and that the information sought was primarily qualitative. The quantitative analysis carried out in the earlier mid-term performance evaluation provided the foundation and context for the current evaluation without any need to repeat it. Further, the analysis was to focus primarily on the period since the mid￾term evaluation and the TEC amendment, and on options for future programming. As a result, the methodology for this evaluation is built on three primary sources of information: project documentation, KIIs with implementers and partners, and focus/discussion groups with beneficiaries. The documents consulted included recent LLP quarterly, annual, and monitoring reports submitted to USAID, background and research studies produced by the project team, and contextual information about the USAID strategy and programming in Guatemala, particularly in education. The list of documents consulted is included in Annex III. DATA COLLECTION The evaluation team collected primary qualitative data in Guatemala City and in the five target departments of the WH (Quetzaltenango, San Marcos, Huehuetenango, Totonicapan, and El Quiche.) The community level activities of the LLP were concentrated in 12 municipalities in the five departments. The evaluation team conducted interviews in seven of the twelve. The 2 Overview presentation to the evaluation team, 13 August 2018, 10 municipalities were selected purposively to include both Mam and K’iche’ communities, and large and small municipalities in terms of population and number of schools. Accessibility and travel times also were considerations, as well as last minute changes due to the annual fair in one case, and election unrest in another. The seven municipalities were: Concepcion Tutuapa in San Marcos, San Juan Ostuncalco in Quetzaltenango, Chiantla in Huehuetenango, Totonicapan, San Bartolo Aguas Calientes, and Santa Maria Chiquimula in Totonicapan, and Joyabaj in El Quiche. Interviews were conducted in the DIDEDUCs. The actual number of people included in the discussions in the five departments varied but the intent was to meet with the Director and technical staff, including the representative of DIGEEX. The same interviews and discussion groups were scheduled in each municipality to provide comparable data across the region. The interviews for Component A included a discussion group with teachers who had access to the kit of materials, and in some cases, had participated in the LLP short-course training. These groups varied in size from five to ten persons. Key informant interviews were conducted with a member of the municipal education commission, and a supervisor/CTA (Coordinador Técnico Administrativo). For Component B, group discussions were held with youth participants and with youth leaders. Some of the participant groups exceeded 20 people. KIIs focused on members of the municipal youth support network, and staff of the OMJ. Participants in the teacher and youth discussion groups were recruited by the LLP technical personnel in the municipality and invited to travel to the municipal cabecera for the meetings. Resonance covered the cost of their transportation and provided a snack to participants. The team was forced to adopt this method of selection for the groups because of the short timeframe for data collection in the municipalities. Some of the participants traveled a considerable distance from their villages to attend the group discussions. KIIs in Guatemala City included implementing partners (J&A and ChildFund), the central offices of the MOE (including the Minister, three Vice Ministers, and the Directors of the General Directorates), representatives of the universities delivering the LLP in-service training programs, sub-grantees (ACUDE, CEIBA, Population Council), LLP strategic partner organizations/projects, and related USAID projects (PUENTES, Nexos Locales). The complete list of people and organizations, and the tally of the number of interviews completed and individuals contacted are included in Annex III. ANALYSIS The analytic process involved construction of one or more tables for each evaluation question, recording the responses to each question/topic in the interview/focus group protocol, in order to identify patterns and frequency of responses. A second important part of the analytic process was a weekly team discussion of findings and uncertainties associated with each question. The final documentation of findings and conclusions was done by the education specialist for each component. 11 LIMITATIONS As noted, the evaluation team relied on the LLP field personnel to inform and invite potential participants for the discussion groups with teachers and youth. Few other options were available given the tight schedule for fieldwork. Participants also were asked to travel from their communities to the municipal center. Both factors may have introduced unknown biases into the group composition, although this effect may have been mitigated by the fact that the team sought to meet with direct beneficiaries of the programs. Some unintentional and unknown bias also may have been introduced in the selection of the sample of municipalities. The best way to avoid that possibility would have been to include all 12 sites but this was not an option. A second important factor to recognize is that the evaluation content is limited to the six evaluation questions identified in the SOW. These questions directed the team to collect perceptions and interpretation from project beneficiaries and participants. The findings reported are based on the interviews and focus groups. In some cases, the perceptions of the participants and stakeholders may differ from the actions reported by the implementers. 12 EVALUATION QUESTIONS: FINDINGS AND CONCLUSIONS COMPONENT A: LEARNING TO READ EQ 1: At what level (national – policy, departmental – education offices, municipal – education commissions and schools) do key stakeholders think USAID should focus its reading interventions? What are the opportunities and challenges of working at each level? Implementation of Component A of the LLP has involved a dual approach with both bottom-up and top-down components. The project has focused on training and equipping the technical staff in the DIDEDUC to carry out their role to implement the EBI model for early grade reading in the rural schools. At the same time, the project worked with central MOE officials to improve the model, develop materials and tools, and strengthen the human resource capacity of the sector as a whole. The LLP has concentrated on implementation of the model in 12 targeted municipalities and selected rural schools in those areas in part so that they could track and measure the effect of the methodology in the classroom, in terms of the children’s learning in reading and writing in both their mother tongue and in Spanish. These results were then fed back to the central offices to move the process forward toward broader implementation.3 The evaluation team questioned officials at the national and departmental levels, and indirectly at the municipal level, for their assessment of the strengths and weaknesses of the LLP approach and thoughts on the focus for future USAID activities in early grade reading and writing in a bilingual and intercultural context. FINDINGS: Department Directorates of Education (DIDEDUCs): In all departments except El Quiche4 the evaluators met with departmental technical staff as a group. In all cases, the officials pointed to the importance of implementation at the local level, including the municipality, district, schools, and classrooms. At the same time, they pointed to the operational role of the DIDEDUC as the avenue to reach the local level. The supervisors (CTAs) also emphasized the need for 3 It is important to note that the LLP did not work directly at the local level, in the schools and classrooms, but focused instead on building the capacity of the departmental technical staff to carry out this intended role and avoid creating a parallel LLP structure. 4 In El Quiche only the Deputy Director and the Departmental representative for non-school education (DIGEEX) were available to meet with the team. In Totonicapan, the team met separately with the technical staff and with the Director. The Departmental Director was not available in Huehuetenango. Students at Aldea Paxboch School 13 direct guidance and coaching in the schools. Several cited the importance of the emerging role of SINAE. Central Ministry of Education: In response to a direct question, officials in the MOE at the national level also pointed to the need for change at the local level in the classroom, but did not discuss the process to achieve it. Teachers: Teachers in the focus groups in the target municipalities echoed the same point made by the departmental staff. The focus should be local, but with support from the DIDEDUC technical staff, supervisors (CTAs), and, in the future, SINAE. Teachers also identified the need for support from the municipality and parents. Municipal Education Commissions: Members of the commissions felt that their role should be expanded in terms of giving more support and emphasis to education in the community, and especially for building support among parents. OPPORTUNITIES AND CHALLENGES:5 The broad picture of opportunities and challenges that emerged from the interviews and discussion groups in the departmental offices and in the municipalities is that the LLP integrated approach to early grade reading in a bilingual intercultural context has put essential components (i.e., human resource development, reading materials, work with parents, strengthening the DIDEDUCs) in place for effective implementation of EBI and the national curriculum. At the same time, these essential components have not yet come together at the local level and in the classroom to effect broad changes in student learning, teaching methods, or local ownership. Departmental: The presence of trained personnel in the DIDEDUCs to implement the model, and their commitment to early grade bilingual reading and writing provide the groundwork and an opportunity to support implementation at the local level. The clearest testimonials for this approach were heard in the departmental offices, where staff had attended university training programs, participated in the preparation and distribution of reading materials, and interacted with LLP technical staff on a regular basis. The LLP provides training and technical assistance to the departmental technical staff to strengthen their role to support the municipalities, schools, and teachers. The LLP has not provided technical assistance or materials directly in the schools. 5 Opportunities and challenges are included in EQ 1 because it was requested as part of the question outlined in the original implementation letter. 14 In interviews, the DIDEDUC staff also identified challenges they face in transferring this methodology and tools to the local level. The challenges include frequent turnover in personnel, including individuals who have had scholarships for the master’s degree programs as they near retirement, and scarcity of resources/funds, especially for travel. Another challenge cited in three departments was the effect of the hierarchical top-down structure of communication in the MOE on the technical comprehension and adoption of the model/methodology on the part of the DIDEDUC technical staff, and their transfer of the methodology to the local level. The DIDEDUCs receive separate instructions from the MOE central offices and the LLP field staff. At the same time, the DIDEDUC staff are not asked by either the central MOE or the LLP to participate in the conceptualization of the components of the model for early grade reading in a bilingual intercultural context or to provide feedback. As a result, the DIDEDUC technical staff tend to see the tasks associated with the EBI approach to early grade reading simply as additional items on their “to do” list. A second effect of this structure is that individual technical officers in the DIDEDUC do not tend to understand or visualize the model for early grade reading in EBI as a package (or a model), but rather in terms of the separate parts/tasks that are their personal responsibilities. Municipality and Local: The potential opportunities at the local level are based on the initial steps the project has taken to affect implementation of the EBI model for early grade reading in the municipalities, communities and schools. Nearly a thousand (n=982) teachers have finished or are in the process of training in the model through the short-course6 (diplomado) offered by the Universidad del Valle at ten locations throughout the region. The kit of materials for early grade reading and writing in bilingual and intercultural settings is now available in the targeted municipalities and schools, including reading materials in Mam, K’iche’, and Spanish, and teachers’ guides. Teachers also pointed to the link the project has made between the classroom reading activities and the USAID-supported activities for reading in the community such as the mini-community libraries and the workshops with parents. Teachers, CTAs, and members of the municipal education commissions also cited multiple challenges at the local level, with consistent patterns across the municipalities and interviewees. • While respondents acknowledged the value of increased attention to reading in the communities, they also commented on the continuing burden of parental lack of support for the school and particularly for the bilingual and intercultural approach to reading and education. Teachers and CTAs pointed to the role of the municipality in meeting this challenge but the commitment and resources in the municipal government are often 6 The term short-course may be a misleading translation of the term diplomado since the course covers a 10-month period and has 10 modules. Interview with CTA in Totonicapan. 15 weak and burdened by politics. Involvement in education, in support for the Municipal Education Commission, and specifically in support for reading seems to depend on the idiosyncratic initiative of the mayor or individual council members. • Only a minority of the teachers in the region have attended the UVG short-courses. While they personally may implement the early-grade reading model in their classes, few examples were offered of transfer of this knowledge to fellow teachers. • In the municipalities and schools in Totonicapan, teachers and directors have also been exposed to the CRS PRODESSA (Catholic Relief Services Proyecto de Desarrollo Santiago) model for bilingual education, which has provided tools, training, and guidance directly to the classroom. Teachers do not seem to have a clear sense of the differences between the LLP and the PRODESSA model and their selection between these two options for materials, methods, or training often seems to be based on criteria that have little relation to teaching strategy, such as availability, incentives, or familiarity. • The most consistent challenge cited by the teachers and school directors is the absence of continuous support and guidance in the methodology. Suggested responses to this challenge include SINAE and learning communities. CONCLUSIONS: • The LLP has worked with the central MOE to improve the integrated approach to EBI and early grade reading. Respondents at all levels praise the quality of each component. • The LLP has sustained a commitment to early grade reading in a bilingual intercultural context in the WH, which is recognized within the MOE and valued in the region. • According to participants, the project has provided valuable technical assistance and training to the DIDEDUC technical staff. • The coverage of the individual components at the school level has not been sufficient to generate sustainability without continuous follow-up and guidance. Further, at both the departmental and the local levels, few individuals visualize the model as a whole. • The LLP assistance at the departmental level has not yet resulted in effective transfer of the model to the classroom due to challenges at both the departmental and local levels. The focus on training the DIDEDUC staff has not been adequately accompanied by attention to the mechanisms of transfer to the teachers and the classrooms. • Some respondents discussed a role for the municipal education commission in encouraging attention to early grade reading and writing at the local level and in support for the schools, but there is no clear model or vision of what this role should be. RECOMMENDATIONS: In both the top-down and the bottom-up aspects of the LLP the various components of the model have been connected to distinct and dispersed actors and decision-makers, obfuscating the vision and implementation of the model as a whole. It is recommended that in the future, USAID focus its intervention for early grade reading on bringing the various components together as a unified model at the departmental level through the trained technical team, and at the local level through the SINAE coaches (pedagogical administrators). The coaches and 16 technical staff should focus on the local educational community to implement concrete classroom strategies based on the various components and methodology of the model as a whole, (e.g., connecting teaching methods to the reading materials and diagnostic tools.) Moreover, USAID should attend to the challenges in administrative and technical coordination at all levels, including the central MOE, which have resulted in differential integration of various elements and components of the model and uneven support of different sectors of the educational community. EQ 2. What is the perception of the teachers who received materials from LLP of the usefulness and relevance of the materials kit and the tools to assess the linguistic profile of the students? The development and distribution of materials to support EBI reading and writing in the early grades has been a central focus of the LLP. Lack of reading materials, especially in the mother tongue, has been identified as a significant constraint to early grade reading instruction and to meeting the objectives for reading time in the classroom. Lack of reading materials also contributes to low reading achievement scores. (See also the LLP Midterm Performance Evaluation Report.) Two types of materials have been developed – teaching guides for teachers (based on the EBI approach to early grade reading), and reading materials in Mam, K’iche’ and Spanish for the students, at each grade level from pre-primary through third grade. Teachers, parents, and departmental technical personnel have been engaged in production of stories and books in Mayan languages, which are culturally appropriate and emerge from the communities themselves. Teachers and departmental technical staff are particularly enthusiastic about the use of the Bloom software to create books, and a regional group has been created to continue this process. In two departments, the technical staff discussed the Bloom software as a tool to easily generate additional books in the Mayan languages based in the local culture and traditions. The software provides a template for new books that is designed specifically for use by individuals with limited computer skills. The new regional group of DIDEDUC staff (mentioned in meetings in two departments) will work to continue the process of increasing the quantity of reading material available in the schools. The individuals involved explained that the group is just beginning to work together. They have received training and demonstrations on use of the software from the LLP staff and are planning to include teachers in the tasks. Under the agreement between USAID and the MOE, USAID develops the materials, and the MOE reviews, approves, reproduces and distributes them. In the past this agreement has been LLP Materials Kit 17 effective, and some of the materials produced under the LLP (e.g., the “big books,” megalibros, for the youngest children) have been distributed widely. On the other hand, under the current Guatemalan administration the review process has been slow, and the evaluation team was told by senior officials in the MOE that budget constraints may delay further immediate distribution. The continuity of the kit and sequencing in the overall EBI early grade reading model are distorted when the Ministry picks and chooses among individual items. Under the TEC increase, funds were made available to the LLP to accelerate the reproduction and distribution of kits to all of the target schools in the 12 municipalities. In addition, J&A has produced all of the items in digital and print form. The teaching guides are available on the MOE platform. Audio and digital reading materials are available online (and in some cases on USBs) so that teachers can download them for use in the classroom.7 FINDINGS: The evaluation team met with teachers in each municipality from schools that had received the kit of materials. The discussions focused on when and how the teachers are using the materials in their classes and teaching, and their assessment of the relevance of the materials. • Teachers in all the groups were aware of the kit of materials, but utilization varied. In some groups, teachers reported that they are using the reading materials on a regular basis, especially in pre-primary and first grade. In two groups, teachers said they have used the materials in the kit for reading/book fairs with great success. On the other hand, teachers from another school said they had received the kit recently, with only a brief introduction (two hours) from the departmental technical specialist and have no idea how to use many of the items, especially the games. • The tool cited most often was the megalibros (big books), which reportedly engage and animate the children, and in two groups, the teachers said they have sparked interest among the children in reading in Mam/K’iche’. Teachers commented particularly on the value of the illustrations in the big books. • In Totonicapan, one group of teachers noted that other materials8 also are available in the school. Without adequate guidance on how to use the LLP tools, teachers said they simply pick among the various items to fit the situation. • Even in municipalities where nearly all children speak either Mam or K’iche’ as their first language, teachers report difficulty in using the early grade reading tools based on the EBI model. Some teachers are monolingual in Spanish, others speak but do not read or write the Mayan language, and others simply do not understand the methods for teaching reading and writing across two languages. • Teachers who are monolingual or teachers in monolingual (Spanish) schools cannot use the bilingual materials or approach. In almost all groups, teachers commented on the problem of multiple dialects within the Mayan language groups, so that the Mam used in San Marcos is not appropriate for the students in Huehuetenango. 7 The audio materials also are available to reach children with disabilities although the evaluation team did not encounter any discussion or recognition of this option. 8 Presumably these are from PRODESSA although the teachers did not discuss them by name. 18 • Across all groups, teachers discussed the limited availability of the tools. Only one kit was sent to each school and it is kept in the director’s office. Teachers need to request an item each time they want to use it. As a result, it is difficult to plan lessons around the items in the kit on a continuous basis. In one group, a teacher said the school director has placed restrictions on use. • The project materials and training are available only for teachers of pre-primary and primary grades one to three. Since the school’s cover grades one to six, teachers complain of a lack of continuity, especially since they frequently shift grades from year to year. On two occasions, teachers also discussed the difficulty of using the materials, which are designed by grade, in multi-grade classrooms. CONCLUSIONS: • Teachers generally praised the materials in the kit, especially the reading materials for the children, but did not discuss the utility of the kit as a package or as tools for implementing the MOE and LLP approach to early grade reading in a bilingual intercultural setting. Only two teachers in two different discussion groups, who had taken the UVG short-course, discussed use of the reading materials in relation to the methods presented in the course. • Overall the discussions of the materials did not reflect systematic utilization linked to a specific methodology or objective. The most consistent use appeared to be in pre￾primary and first grade. The delivery of the materials without intensive orientation or on-going support and guidance has reduced their utility. • Issues in delivery also echo longstanding concerns about equipping classrooms, in particular the problem of having only one kit per school. Some teachers said the materials arrived late in the year, and access is controlled by the director, who is personally responsible for ensuring the materials stay in the school in good condition. Diagnostic Tool for the linguistic profile One of the key tools available to pre-primary and first grade teachers is the tool and method to assess the linguistic profile of each child relative to his/her use and knowledge of Mam/K’iche’ and Spanish. Pre-primary and first grade teachers in five of the seven discussion groups said that they are using this tool with incoming students. These teachers reported that the tool is easy to use and understand but complained that it is time-consuming since it is administered individually. There was a general agreement that the results are helpful because they show the teacher the capabilities of the children in the group, primarily so that they know how to communicate with them effectively. Teachers also generally noted a shift over time toward more use of Spanish by the children. On the other hand, there was no discussion in any of the groups about the utility of the diagnostic tool for preparing a teaching strategy or links to the EBI model for early grade reading. 19 RECOMMENDATIONS: The teachers voiced a positive perception of the materials from the LLP and some teachers are using items like the megalibros in their classrooms. But, the kit is not being used in a systematic way as part of the EBI model for early grade reading and writing. It is recommended that, in the future, USAID should deliver the materials through SINAE, in conjunction with the pedagogical guidance and support to be provided by the coaches. Further, USAID should pursue the review and approval process of the materials in the MOE so that copies of the kits (or of sets of tools) are distributed to each primary school teacher (as opposed to one per school) and can be used on a daily basis as an integral part of their teaching strategy. Likewise, the coaches from SINAE should assist teachers not only in the application but also in the use of the linguistic profile assessment tool as part of the EBI model. EQ 3. How likely is it that the participating universities will continue to implement in-service teacher training programs without USAID support? (considering resource-needs, capacity, and willingness) a. What are the factors that promote or hinder the long-term sustainability of the LLP-implemented in-service professional development programs for teachers and MOE professional staff? b. How could a follow-on activity be more sustainable? What are potential models which could be implemented to achieve similar objectives of teacher training? The LLP strategy for human resource development was designed in collaboration with the MOE and was based on programs for in-service university-level training of teachers and other education professionals carried out by previous USAID projects and other donors. The specific focus of USAID programming has been on early grade reading and writing in a bilingual intercultural context. The purpose of the strategy for human resource development under Component A of the LLP has been to strengthen the capacity of classroom teachers as well as of supervisors, coaches, and technical specialists at the departmental level of the MOE. To respond to this question, the evaluation team met with the coordinators of the LLP￾supported programs in the three universities implementing the programs:9 • Universidad del Valle de Guatemala (UVG) for the short-course (diplomado) in bilingual intercultural early grade reading; • Universidad Panamericana de Guatemala (UPANA) for the bachelor’s and master’s degree programs in bilingual intercultural early grade reading; and 9 Note that the evaluation team did not collect any information on the USAC course for alternative education except for a brief discussion with the director of DIGEEX. 20 • Universidad de San Carlos de Guatemala (USAC), the national university, for a master’s degree program in leadership and pedagogical coaching/guidance to strengthen educational quality. In all three cases, in response to a direct question about whether and how the program could continue without USAID support, the coordinators said that the universities cannot continue the programs without grants from USAID or other donors. The costs, primarily for faculty, cannot be covered by internal resources. In addition, teachers require scholarships to participate in the bachelor’s and master’s specialized programs. Teachers’ salaries are insufficient to cover the cost of any in-service degree program. At the same time, all three universities said they would like to continue to offer the programs. The universities and USAID have made a substantial investment in human resource development under the LLP in preparation of curriculum, materials and platforms, technological equipment, infrastructure throughout the region, and recruiting and training faculty and tutors. All the pieces are in place to build on this investment. For UVG, the short-course is seen as a part of their service to the community and a commitment to quality education in the bilingual intercultural setting. UPANA cites the importance of offering a unique original degree in early grade reading in a bilingual intercultural context and the ability to respond to a perceived and incipient demand for these skills. USAC, as a public university, cites its commitment and requirement for building in-service teacher training to complement its basic teacher training and certification program. Both the initial teacher training program (FID, Formación Inicial Docente) and the in-service program (PADEP, Programa Academico de Profesionalizacion Docente) are funded by the MOE10. a. Factors that promote or hinder long-term sustainability The responses to questions about what may hinder, or support sustainability of the programs rest in part on factors related to supply and demand. As noted above, the universities feel they have an important service to offer to teachers and education professionals and that the structure is in place to deliver these courses. No one, in the universities, in interviews in the MOE, or in discussions with teachers, voiced opposition to the approach or to the content of the programs. The B-learning approach, relying on the platforms and distance learning, and periodic in-person sessions with instructors and tutors, is acceptable. Sustainability in terms of the supply side of the equation seems to rest primarily on methods to cut costs or to obtain a reliable (internal) source of funding. The conversation with the University Consortium suggested several alternatives for meeting the short-term gap in demand and cost, including having the university subsidize the new program with profits from a more lucrative established program. Potentially innovative methods for inter-university collaboration in offering these degrees also were mentioned but will require further discussion. 10 There is no direct relationship between the LLP in-service programs and PADEP and none has been suggested for the future. They are separate programs. FID and PADEP were referenced by USAC in this context as evidence of USAC involvement and commitment to human resource development in education. 21 In the discussion of both supply and demand it may be useful to note that the initiation of the courses and participation may have been compromised by the political context in the country in 2015-16, with the forced resignation of the vice-president and then the president, followed by resignations of key ministers, including the Minister of Education, and other officials down to the departmental level. The process was accompanied by considerable civil unrest, demonstrations, and disruption of transportation and daily activities. The situation was not resolved until after the inauguration of the new president in 2016. This situation may have caused delays in establishing formal working relationships among the MOE, the universities, and the LLP. It also may have affected participation and drop-outs from the courses, particularly the UVG short-course. At the same time, no one in any of the interviews discussed this context as a factor in their participation in the project activities. Through the meetings with technical staff in the departmental education offices, CTAs, and in the discussion groups with teachers, the team also spoke with individuals who had participated in the short-courses and the bachelor’s and master’s programs. These conversations provided some insights (largely anecdotal) into the perceived value and application of the training by the beneficiaries, and factors that may affect demand. • In two of the departmental offices and among the CTAs, those who have received master’s degrees (primarily at USAC) are enthusiastic about the experience and active in attempting to apply what they have learned. An important factor in demand for the master’s programs is the potential for employment in the MOE and especially in SINAE. • The response among teachers who had attended the short-course on bilingual intercultural early grade reading was mixed. Some were excited by the content and described efforts to use the methods in their classes. Generally, however, participants focused on the personal costs involved in participation, a lack of organization during the first two cohorts of the short-course, and the high drop-out rate.11 • In a number of municipalities, teachers drew comparisons between the UVG short - course and other courses that are available through PADEP (and other programs like PRODESSA), suggesting that they can choose among them. Others said they are unlikely to do any courses. These responses appear to downplay the content of the courses per se as a factor in demand, and to indicate lack of a clear vision or plan for self￾improvement or success in teaching. • Factors (outside course content) mentioned as related to demand include: - Teachers are not given leave during work hours for the course. The in-person sessions are held after the schools close for the day. Some teachers have second jobs and home responsibilities that make these evening sessions burdensome in terms of time, transportation costs, and often skipping a meal.12 11 The coordinator from UVG also discussed the problems in organization and drop-out, especially in the first cohort of teachers. The specific problems mentioned across the interviews included problems with the computers, with the meeting sites, with excessive assignments, and with the tutors. She said that the logistical and personnel problems have been corrected, and some re-design of the curriculum is still underway. To date, the current cohort of 360 students has seen a much lower drop-out rate. 12 According to J&A, the official policy of the MOE and USAID requires that all professional development activities be organized so that they do not hamper the students’ opportunity to have a teacher in regular school hours. That 22 - The virtual aspects of the courses also may be a deterrent in some cases. The UVG coordinator noted a lack of familiarity with computer technology among short￾course participants. She said that about 20 percent have computers, but some who entered the course did not even have email. Most participants rely on Internet cafes, but teachers complained about the costs and unreliability of these on-line connections and Internet. - The courses are not available to contract teachers or to monolingual teachers. Weak facilities in reading/writing in the Mayan language also may be a deterrent for the master’s programs. These requirements are based on the uncertain tenure of the contract teachers, and on the MOE regulation that, in bilingual schools, teachers of pre-primary, and primary grades 1 to 3 should be bilingual in order to meet the EBI standards. - The short-courses are only open to teachers in pre-primary, and grades 1 to 3. Teachers frequently shift across grades (from pre-primary through grade 6) from year to year. - A more general problem, across all the programs, is the absence of incentives for completing the course or the degree, in terms of compensation, promotion, or broader opportunities. On the other hand, some in the departmental offices have suggested that scholarships for the master’s program should not be open to teachers/supervisors/other professionals who are approaching retirement, whereby the benefits accrue only to the individual and not to the education system. - There is broad agreement across the universities and the teachers that teachers’ salaries are too low for them to participate in these programs without scholarships, and in some cases without additional subsidies. CONCLUSION: At this point, there is very little possibility that the programs in both the private and public universities will continue without USAID support. Under the LLP, in-service training programs have been developed and delivered to initial cohorts. A rigorous assessment of the actual and potential effect of this training for individual participants, for the delivery of bilingual intercultural primary education, and for early-grade reading would help set the parameters for both the supply and demand for each program and for the package. RECOMMENDATIONS: Given the scarce possibilities in the immediate future for sustainability of these programs, USAID should consider continuing its support for these activities, at least in the short-term. This support should be coupled with strengthening the links to the other aspects of the EBI model for early-grade reading, especially the follow-up with graduates and the guidance/coaching in the classroom. said, teachers noted the schedule as a specific disadvantage of this short-course compared to others that meet during standard work hours. 23 USAID should conduct an assessment to clarify the demand for each course and the potential links to other in-service training programs including those offered through the university consortium. The assessment should also include investigation into the link between the training (particularly the short-course) and other components of the EBI model for early grade reading, and the results in the classroom. Finally, further investigation is needed of the potential incentives (including financial) for pursuing in-service training. COMPONENT B: EDUCATION FOR EMPLOYMENT AND LIFELONG LEARNING EQ 4: In what ways have the youth engagement activities been beneficial to youth participants and their communities? Unlike Component A of the LLP, which was built on decades of USAID collaboration with the MOE and a focus on EBI, the smaller Component B was a new area of engagement for USAID in 2014. It involved identifying and defining a segment of the rural population in the WH that has been relatively invisible and ignored, namely, young people (ages 15 to 24 years) who had dropped out-of-school, many in the early primary grades. They are too old to return to school and find few options local employment. The majority of the group in the local population and in the LLP are young women, sometimes with children of their own. The young women are further constrained by family responsibilities and by their parents’ unwillingness to allow them to move around freely, to socialize, or to participate in activities outside the house. 13 The young men tend to work for sub-standard daily wages in agriculture or other unskilled work. As young men, they are less constrained than the young women and often migrate seasonally for work in agriculture, especially coffee and sugar. Migration to urban areas and/or out of the country is often desired, but costly. The two components of the LLP have been implemented separately, except both components have targeted the same municipalities. 14 The LLP strategy for OSY includes five processes: (1) 13 Both the youth and the project staff reiterated that girls in this region may have children in their early teens. Some young women also work for sub-standard wages cleaning or cooking in other homes. 14 Component B is implemented in eleven of the twelve municipalities. In spite of intense efforts, ChildFund was not successful in recruiting a sufficient number of participants in Concepcion Chirquirichapa (Quetzaltenango) to merit development of the project activities. Only 25 youth expressed interest, apparently because of the large number (both male and female) who work outside the municipality. Youth focus group participants in Concepción Tutuapa, San Marcos. 24 the reduction of barriers, (2) establishment of spaces/situations for dialogue, (3) the formation of youth support networks and formation or strengthening the Municipal Office for Youth, (4) human resource development through Alternative Basic Education (non-school), and (5) improving the quality of the alternative education system. In the project, youth were organized into local committees in 150 communities, and encountered points where the youth could meet, receive training in soft skills and community participation, and participate in education and skills training programs were identified, often with the support of the municipality. About 313 youth (210 women) completed a training-of-trainers (formador de formadores) course focused on life skills and planning, and in turn, provided training to other youth in their communities. The evaluation team met with youth participants in the seven municipalities included in the sample. At the request of the evaluation team, the LLP local technical staff assisted in identifying and recruiting the youth to attend the discussions. The evaluation team reimbursed participants for transportation costs to the municipal center for the focus groups. The youth were asked to discuss six key questions: (1) What were the positive aspects of your participation in the project? (2) What aspects have been the most difficult to achieve? (3) Have you met resistance or rejection from adults in the community because the young people are organizing and working together? (4) How have you been treated by the mayor, the municipal council, and the OMJ? Do you think change is possible? Do you have an action plan? (5) What actions or projects do you plan to pursue? (6) Will your committee continue to meet after the project ends? FINDINGS: When asked about the positive effects of the project for them, they responded first in terms of the impact it has had on them as individuals: • Greater self-confidence, less fear of speaking in public and expressing ideas, self-esteem; • A change in aspirations and a vision of the future, rejection of dangerous activities, more thought about when to marry and have children; • A sense of personal rights and responsibilities, discovery of personal talents, and new skills to generate income; • Made friends and joined others in forming groups across communities, improved communications with parents and other adults. Organized and delivered training to other youth in their communities; and • Learned new skills that they have used to generate income through micro-businesses. When asked about their relationships within the community, adults, municipal officials, actions as a group, and projects, they cited: • Recognition and appreciation of the abilities of the youth and of their potential to act as leaders; 25 • Participation in the COCODES and the COMUDES; 15 • Increased acceptance of the participation of women and recognition of their interests and needs; and • In three municipalities, youth leadership in mini-projects (introduction of clean water in a community in Chiantla, an awareness campaign in San Juan Ostuncalco, “No cambies tus sueños por un bebe,” and supporting small community productive projects in Sta. Maria Chiquimula.) The problems cited, in response to questions (2) and (6) are of two types. On the one hand, in terms of their activities and especially their plans for marriage, the young people continue to be frustrated by the restrictions and expectations imposed on them by their parents and family. They also noted a problem in generating local activities and commitment, particularly among young men, who often are more focused on migration out of the community to find opportunities than on expanding options in the community. The second set of problems they cite are challenges to the sustainability and growth of the program. In several municipalities the youth discussion groups talked about a problem with turnover among the project technical staff and the fact that they often live outside the communities, so they are not around at crucial times. Youth leaders also cited specific concerns about next steps: • Concerns about how to move forward without the support of the project because of their limited experience in raising issues and acting in relation to local authorities; • Need for resources to carry out productive projects; • Need for guidance on how to organize and legally register and operate civic organizations: • Requirement for continuing guidance and support to participate effectively in the COMUDES; and • Need for continuous guidance and support in developing methods for organizing and working as groups. CONCLUSIONS: • The LLP has identified a clear need in this segment of the population for recognition, organization, acceptance, and participation. This need may be particularly clear for the young women who have less mobility than the young men and may be caring for young children. The actions of the project to counter these barriers and open up new opportunities are widely praised. • In the focus groups, young people also praised the opportunities to continue their education and to learn new skills to generate income. At the same time, as youth, their 15 COCODE: community development council; COMUDE: municipal development council 26 timeframe is short and they gave little indication of future plans or expectations of new options in the community. For many, migration is a dream but not necessarily a reality. 16 • The change in the communities and in the lives of the young people is real but tenuous. As a new activity, Component B initiated new actions and programs to increase the visibility of the OSY population and to expose and overcome the barriers they face. The participants and emerging leaders would like to build on this experience but are uncertain about how to do it. EQ5 also directly addresses issues related to the OMJ as a mechanism to build sustainability and continuity. RECOMMENDATIONS: While praising the positive effects of the program for the youth who have participated, the young leaders voiced concerns about continuing resistance within their families and communities. They recognize that it will be difficult to recruit more youth and sustain the programs without outside technical and financial support. The recommendations come from the youth leaders themselves and are focused on the steps needed to internalize the activities, to make them sustainable as part of the fabric of the communities. Three options are offered for consideration: (1) A significant barrier for OSY, particularly young women, is their own parents. A first step is to carry out a continuing series of workshops with the adults in the community. The LLP initiated intergenerational conversations as a prototype for these workshops. The OSY ask that these dialogues continue at the community level to generate open discussion of the situation and roles of the youth, the problems they face, and possible solutions to these problems. (2) Identify and train young leaders in each community to become tutors and to organize one or more groups to pursue alternative education classes. These leaders should be linked to organizations (NGO, government) offering alternative basic education and/or technical training. In this way, building on the processes underway and the tools developed through the training-of-trainers (formador de formadores) activity, the project could nurture sustainable and, to the extent possible, permanent leadership within the community to help fill the gap left by the departure of outside technical support and to generate internal energy for change.17 (3) A third action would be to strengthen these young leaders with other life skills and tools to serve as mentors or counselors in other aspects of community life affecting the OSY, such as: sports or entertainment activities, conflict resolution, building alliances for productive projects, prevention of violence in the household, gender￾based violence, and alcoholism, and building youth networks across communities. 16 The references to going to the US are unlikely to be reliable since the evaluators introduced themselves as a team from USAID. 17 This approach is currently being developed by the Population Council with young women in 40 communities. Another potential avenue to pursue would be to consider negotiating with the Mayan community leaders to endorse these activities and to examine the role of tutor as a way to meet the required year of community service. 27 Skills in these areas could continue to serve the tutors themselves and their communities as they move into adult roles. EQ 5: How can USAID work most effectively with the Municipal Youth Offices? (considering that several USAID interventions work with them) According to the Municipal Code, the Municipal Commissions and Offices of Youth are intended to ensure the rights of youth as citizens and their participation in decisions. Although the Municipal Youth Commission has been included in the code since 2002 as Commission 9, the form and function of the Commission varies widely. Commission 9 includes families, children, adolescents, youth, and senior adults. In the past, the USAID-funded project, Nexos Locales, worked with 43 municipalities in the WH to strengthen the OMJ institutionally although it is no longer engaged in this activity. Only five LLP municipalities were served by the Nexos Locales project. In interviews, the municipal officials and OMJ staff were clear about differences among the various USAID-funded programs and the opportunities provided. In response to this question, the evaluation team visited the OMJ18 in the seven sample municipalities and interviewed the staff. Of the eleven municipalities in which the LLP is working with OSY, only one, Chiantla, provides a clear model of how the OMJ should operate. Of the seven offices visited, only two are dedicated only to youth. The other five are Offices of Children, Adolescents, and Youth (one also includes the elderly.) In most cases, the offices were created through the initiative of UNICEF and Save the Children and are therefore oriented primarily toward protection rather than initiatives related to rights and participation. Further, due to the diversity of the population segments to be served, the actions and activities are diverse, discrete, and lack continuity and sustainability. FINDINGS: The configuration for the seven offices is shown in the following table. In all cases, the municipality provides the office space (including utilities and security) and, where present, the staff person. In one municipality, the staff person for the Office of Children, Adolescents, and Youth also directs the municipal library and is responsible for municipal purchases (Compras Municipales). In this case, the Office of Children, Adolescents and Youth is located in the library. MUNICIPAL OFFICES IN VISITED MUNICIPALITIES Mam Region LLP Municipalities Name of the office in the sample of municipalities Departments Staff Huehuetenango Chiantla Ofic Municipal de Juventud 1 18 The municipal offices for youth services have different names, staffing, and functions across the municipalities. In some cases, the office includes children, adolescents and youth. In others, the title of the office includes the term “Protection” of children, adolescents, and youth. The tem OMJ is used here to represent this spectrum of titles. 28 San Marcos Concepción Tutuapa Ofic Mun Juventud (en formación) 0 Quetzaltenango San Juan Ostuncalco Ofic Municipal de Niñez, Adolescencia, Juventud 1 K´iche´ Region Departments LLP Municipalities Totonicapán Totonicapán Ofic Municipal de Juventud 1 Santa María Chiquimula Ofic Municipal de Protección a Niñez, Adolescen, Juventud 2 San Bartolo Aguas Calientes Ofic Mun Niñez, Adolescencia, Juventud 0.33 Quiché Joyabaj Ofic Municipal de Protección a Niñez, Adolescen, Juventud 3 7 municipalities 7 municipal offices 8.33 Only two of the OMJ visited for the evaluation have received program funds for youth from the municipal government. Most operate with funds from donors and both national and international NGOs. They also coordinate with other governmental agencies such as the Ministries of Health, Labor, Education, and Culture and Sports. In general, the OMJ has been viewed as a location or a means to operate activities rather than as an objective in and of itself. The staff reported that in almost all municipalities, the mayors and the members of the municipal councils have few connections to the offices. This report is made despite the fact that each mayor signed a Memorandum of Understanding (MOU) with the project agreeing to support youth activities. The mayors do not provide funds, attend events or activities, or incorporate the operational plans of the offices into the operational plans for the municipality as a whole. The profile given by the people interviewed is consistent with the conventional view of the mayors and councils as motivated primarily by infrastructure and construction rather than social concerns. In addition, neither the mayor nor the municipal council has a legal obligation to maintain and support the OMJ with support from the annual budget. This situation is in contrast to the legal obligation attached to the Municipal Women’s Office due to a 2010 change in the Municipal Code (Decreto 22-2010), primarily as a result of advocacy by women’s organizations. With one exception, the staff person in the OMJ has minimal experience with youth issues and programs, or with the municipality (although s/he may have ties with the mayor). They tend to respond rather than to lead and say they would like to have additional training and technical assistance. One of the responsibilities of the OMJ staff is coaching and guiding the youth named as representatives to the Municipal Development Councils (COMUDEs). Youth have been named to participate as voting members of the COMUDE in three of the seven municipalities, Chiantla, San Juan Ostuncalco, and Sta. Maria Chiquimula. In two municipalities, according to OMJ staff, the presence of the young men in the COMUDE is positive. In the third, the young person filling that role has not attended regularly and has not been effective. Participation in these 29 meetings is very challenging for the youth members since as many as 100 or more adults may attend, many skeptical of youth involvement. The staff of the OMJ identified a series of difficulties in continuing and sustaining the participation of the young people within the municipal structures: • A lack of resources/funds to continue to recruit and mobilize the youth groups; • A lack of knowledge about the basic municipal laws that affect the functioning of the office, such as the Municipal Code or the regulations for the development councils (i.e., COCODE, COMUDE); • Rivalries for services and leadership may develop among the youth, especially between those who are in school and the OSY; and • The need for continuous recruitment and formation of new groups and being responsive to the concerns of the emerging youth population. CONCLUSIONS: POTENTIAL ACTIONS FOR USAID TO WORK WITH THE OMJ • Work directly to engage the mayor and the municipal council to actively support the youth issue. Collaborate with members of the municipal council to define the concrete purpose, functions, scope, and responsibilities of the OMJ. • Promote exchanges to learn from the experiences, achievements, and limitations experienced in other municipalities, such as the recent visits to Chiantla by officials from other LLP municipalities. • Building on the concept of the Youth Network, set up a coordinated action group of organizations and projects involved in youth programs in the municipality to support the operation of the OMJ. • Assist in development of the Annual Operational Plan (POA) for the OMJ, with the Municipal Planning Office, to establish budgets, indicators, goals, and periodic progress reports on OMJ activities. • Promote a modification of the Municipal Code to provide a legal foundation for the Municipal Youth Commission and Office, similar to the code for the Municipal Office for Women. • Prepare an administrative manual for the OMJ, adapted to the communities and population of each municipality. The recent LLP publication of a guide for operation of an OMJ could provide background for the manual. • Consider recruiting former residents of the municipality who are interested in “giving back” to their home community to join the local youth support network. RECOMMENDATIONS: • Prepare a short-course (Diplomado de Formacion) for staff of the Municipal Offices of Children and Youth. The course could be designed in collaboration with the Nexos Locales activity or other USAID programs supporting municipal governments, to focus 30 on effective administration of these offices and procedures to access and utilize funds available to them. The purpose would be to prepare these staff to become allies of the mayor and the municipal councils by giving them the tools to acquire and administer the resources from the Departmental Development Councils (CODEDEs), which are available (and often untapped) for this purpose. They would learn to present projects in the required format for successful submission to the SIAF (Sistema Integrdo de Administracion Financiera) and the SICOIN (Sistema Integrado de Contabilidad), and to execute the designated budgets. • In addition, since the mayors and the municipal councils give priority to investments in public works and infrastructure, the OMJ staff could be encouraged to present proposals for investments in public works directed to the needs of children and youth such as schools, libraries, parks and sports arenas, and then to manage the implementation of these projects. • Participation in the short-course might be encouraged by also developing a prize or declaration for mayors and municipalities who best serve the needs of children and youth. This could be done in conjunction with SEGEPLAN (Secretaría de Planificación y Programación de la Presidencia), ANAM (Asociación Nacional de Municipalidades) CONJUVE (Consejo Nacional de la Juventud) and INAP (Instituto Nacional de Admistración Pública.) • Promote a legal modification of the Municipal Code, with collaboration of other organizations, to incorporate a Municipal Youth Commission and an OMJ, to attend to youth. EQ 6: What lessons can be learned from the development, validation, and implementation of the tests for functional skills in reading, writing and mathematics for OSY? a. What is most relevant for the MOE? b. What is most relevant for youth-serving organizations? The high proportion of OSY in the WH points to the importance of the alternative basic education program developed and managed by the MOE DIGEEX, with LLP support. The alternative basic education system is intended to provide OSY (and other adults) with the functional skills in reading, writing, numeracy, and financial literacy needed to earn a living, to fully participate in the community, and to advance to other levels of education. In collaboration with DIGEEX, the LLP identified the skills required for the labor market of the WH, through a series of interviews with local employers. Based on these skill sets, learning and quality standards were identified and adopted for alternative basic education programs in each functional area. The next step, which is currently underway, is to develop a system to measure, monitor, and set benchmarks for the achievement of these standards. The LLP is now in the process of collecting a large data base to finalize the items to be included in the monitoring tool and to establish benchmarks for levels of achievement. The task is important for both the MOE in 31 curriculum development and certification, and for the NGOs like CEIBA and ACUDE that are delivering the alternative basic education curriculum to monitor the compliance and strengths/weaknesses of their programs. According to J&A, this type of test does not presently exist in Guatemala. It requires an extended complex methodological operation, and is geared specifically to the labor market in the WH. After identifying the skill sets and the standards, the LLP developed and validated items to reflect achievement of skills according to the standards. These items are now being administered to a large sample of students to finalize the content and scale. The tests (pruebas) were administered to youth in ACUDE and CEIBA education programs in 2015, 2016, and 2017. By the end of the current year, J&A expects to have a large enough test pool to validate and finalize the scales and benchmarks. In the meantime, some general summary statistics on the results of these pruebas have been published by the project and reported to the implementing organizations. The MOE at the central level (DIGEEX and the Vice Minister for Quality) recognizes the value and utility of the pruebas as a base for monitoring alternative education achievement, which is linked to realistic, day-to-day application of learning rather than on academic standards. The demonstrated utility of learning is important not only to strengthening the local labor market and employment but also to demonstrate concrete reasons to the OSY for continuing to study. At the same time, while they support the general purpose of the pruebas, the officials at the central level say they do not really understand the process or exactly how it will be applied in the future. The evaluation team found no information about or knowledge of the process or of the pruebas in the discussion groups with the youth (although two young women said they had taken the test at some point). DIGEEX staff in the departmental education offices also have no knowledge about the testing process or its purpose. ACUDE and CEIBA, organizations providing alternative basic education programs in the WH and participating in the pruebas, have received some feedback from J&A about the results. One of the two said that the feedback had been useful in examining their teaching methods in relation to particular topics. The other said that the brief introduction to the findings was not sufficient for them to revise their current methodologies. Both organizations said they are interested in learning more and using the tests in the future to improve student achievements. They also noted that they have not had any negative reactions from students about taking the tests, recognizing that they are not used to judge the success of individual students. The primary lesson at this stage in the process is that it appears to be a promising and important methodology and should continue through to its conclusion. The evaluation team has no recommendation on this topic except that the exercise should be carried through to its conclusion and put into practice by the Ministry and particularly DIGEEX, and the organizations implementing the alternative basic education curriculum. The responses to questions about the pruebas, voicing interest and support but lack of information and understanding, also suggest that once the validation process is complete, transparency and full discussion of the process and its interpretation will be essential for full implementation. 32 CONCLUSION In conducting the evaluation and meeting with individuals touched by the LLP activities at all levels of the education hierarchy, as well as with youth and youth advocates in the highland communities, the general appreciation and praise for the quality of the various components of the project are clear and unanimous. This is a program that has done what it set out to do and has done it well. Looking ahead, however, the evaluation team has identified gaps and points of vulnerability that will affect future plans. For Component A, as discussed in evaluation questions 1 and 2, the primary issue is that, at this point, the various parts and components of the model for early grade reading in a bilingual intercultural context do not appear to come together to form a coherent and observable whole at either the local or the central level of the education system. The disjuncture is clearest in conversations with classroom teachers. The discussion groups demonstrated that teachers are aware that early grade reading is a priority and that attention needs to be given to the requirements and needs of children in bilingual settings. The reading materials are attractive, popular, and spark the interest of Mam- or K’iche’- speaking children. Yet for many of the teachers, the opportunity to attend the short-course in early grade reading is just one of various options, assessed as much in terms of incentives and costs as in terms of the subject matter. Teachers apparently do not make the connection between the reading materials and tools, the reading fairs, and the in-service training, a situation that is exacerbated by the uneven presence and timing of the various components in the schools. Across all groups, the teachers lamented the absence of classroom guidance and support, or more generally, the absence of interest in the classroom from the community and from the Ministry. The team observed a similar disjuncture at the departmental level and in the central MOE. Each component of the model and its advocates operates separately in terms of the characteristics of that particular component rather than in terms of an overall model. The sense from the interviews is that decisions about reading materials, in-service training, or teacher’s manuals are made about discrete items rather than about their contribution to the model as a whole. For Component B, testimonials from the OSY participating in the LLP activities demonstrate a clear impact on them as individuals and on their identity within the rural population. The greatest concern for these programs is sustainability, expressed not only by the youth leaders and participants but also by the technical staff and tutors who have managed these programs in the communities. To this point, the programs have been brought into the communities from the outside by the LLP and various NGOs. The challenge for the future is to incorporate these programs into the fabric of the communities themselves so that the activities become self￾generating 33 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK Statement of Work Final Performance Evaluation Lifelong Learning Project (Guatemala) I. Purpose of the Evaluation: The purpose of this final performance evaluation is to provide answers to key questions that will facilitate a technically strong program closeout and inform future USAID Guatemala education programming. This evaluation will assess key aspects of the Lifelong Learning Project (LLP) that were incorporated as a result of the mid-term evaluation. II. Background: In March 2014, USAID/Guatemala awarded the LLP to Juarez and Associates. The objective of LLP is to improve education quality and increase access for underserved populations, i.e. indigenous children and out-of-school youth (OSY) in the Western Highlands (WH) region, (specifically in Huehuetenango, Quetzaltenango, Quiché, San Marcos, and Totonicapán). The project provides technical assistance to the Ministry of Education (MOE) to improve the quality and equity of students’ reading outcomes. Technical assistance is focused on increasing teacher effectiveness; improving classroom learning environments; fostering effective first and second language (Mayan languages and Spanish) acquisition and reading; extending learning opportunities to underserved populations, especially girls and indigenous groups; and expanding parents’, communities’ and stakeholders’ participation in student learning. Reading outcomes are being improved through the MOE structure reaching as many schools as possible. On the other hand, the project seeks to enable OSY to achieve greater economic self-reliance and higher levels of civic engagement and educational achievement. LLP is working to provide Guatemalan OSY with increased education and civic engagement opportunities with the aim of increasing stability and economic growth in the WH. LLP directly supports the USAID Education Strategy Goals: ● Goal 1: Improved reading skills for children in early grades. ● Goal 2: Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals. ● Goal 3: Increased equitable access to education in crisis and conflict environments. 34 The project also supports USAID/Guatemala’s CDCS 2012-2017, Development Objective 2: “Improved Levels of Economic Growth and Social Development in the WH.” Intermediate Result 2.3: “Education quality and access improved.” Target beneficiaries are the historically underserved and excluded populations in the rural areas of Guatemala’s WH. LLP has two components: each with its own set of results and sub-results. ● Component A: Learning to Read ● Component B: Education for Employment and Lifelong Learning The Objectives of LLP under Component A are to foster pertinent learning environments by providing relevant opportunities to learn for children in primary public schools. As more than quarter of Guatemalan children are failing first grade, interventions in early grade reading in bilingual contexts (Mayan languages and Spanish) are sorely needed. USAID’s improved reading approach revolves around five goals. These are: 1) More time devoted to teaching reading, 2) Better techniques for teaching reading, 3) More texts in the hands of learners, 4) Teaching in the mother tongue (a language learners speak and understand), and 5) Testing learners’ reading progress. Target beneficiaries are primary students from grades 1 to 3. Under Component B, LLP targets youth education and employment. This intervention also emphasizes the critical need to develop basic literacy and numeracy (with reading as a foundational skill), but targets a different group, i.e., OSY (15 to 24 years old) in selected rural areas and adds a focus on providing these youth with education and workforce skills. There is an important need to focus on this growing young population with limited educational and employment opportunities Education options under Component B are tailored to different pathways, local social capital, and labor market assessments. The provision of education and workforce opportunities for youth is a preventive measure to contribute to a conflict and violence free environment. This intervention also provides valuable feedback and a learning agenda to Government of Guatemala (GOG) plans and interventions targeting OSY. Both components work at the department and national level but focus their interventions primarily in 12 municipalities in five departments of the WH: Department Municipality Quiché San Pedro Jocopilas Joyabaj San Marcos Concepción Tutuapa Totonicapán Totonicapán San Bartolo Aguas Calientes Momostenango Santa Lucía La Reforma Santa María Chiquimula Quetzaltenango Concepción Chiquirichapa San Juan Ostuncalco Huehuetenango Chiantla 35 Santa Barbara In 2017, the project received a TEC increase to: 1) expand professional development opportunities for principals, teachers and MOE staff; 2) develop and distribute more learning materials in Mayan languages and Spanish, including audio books, which can be used to support low-vision and blind students; 3) enhance parents’ support to improve and increase reading activities at home; 4) expand alternative basic opportunities for OSY in the WH; 5) improve and expand the project’s research agenda; and 6) improve and expand the gender equity and female empowerment integration in the project addressing issues of gender-based violence. The TEC increase was developed considering relevant recommendations from the mid-term performance evaluation conducted by a third party. A key feature of this TEC increase was the development and expansion of in-service professional development degree and certificate programs for teachers in the project’s target municipalities. A list of the programs, implementers and other details can be found as Attachment #1. In addition, the MOE has a unit in charge of providing alternative basic education opportunities for OSY. This directorate (DIGEEX) has several programs implemented partially through civil society organizations. All programs have low coverage and there were no quality standards to assess their results. The GOG has a National Youth Policy that prioritizes youth as a key element of the country’s development agenda. However, this intention is not consistent with the actual resources and efforts invested to promote youth development and participation. Youth education and the situation of OSY in Guatemala has received considerable attention due to the humanitarian crisis declared by the U.S Government when in 2014 more than 50,000 unaccompanied migrant children and youth were apprehended at the border in their way to the United States. These unaccompanied children and youth coming from Mexico and Central America took the journey in search of better economic opportunities. Addressing the challenges they face and creating conditions for their development has become a Mission wide priority for USAID/Guatemala. The LLP is part of this effort and it is being observed closely since it is creating a path to improve the quality of alternative education and support services for youth. This project is developing creative approaches to overcome education and employment barriers specific to rural/indigenous OSY. III. Scope of Work The evaluation team should consider the context in which the project is implemented to interpret and analyze data. The LLP’s work has a different approach for each component. These approaches take into consideration the MOE’s structure and capabilities. In this sense, this final performance evaluation is part of a collective learning agenda. The evaluation results will inform late-term adjustments to the project, as well as the design of new activities. In addition, the results will be used to promote discussions among USAID/Guatemala, MOE, and other implementing partners about strategies, best practices, and sustainability. 3.1 Key Evaluation Questions 36 For Component A 1. At what level (national - policy, departmental - education offices, municipal - education commissions and schools) do key stakeholders think USAID should focus its reading interventions? What are the opportunities and challenges of working at each level? 2. What is the perception of the teachers who received materials from LLP of the usefulness and relevance of the materials kit (Attachment #2) and the tools to assess the linguistic profile of the students? 3. How likely is it that the participating universities will continue to implement the programs without USAID support? (considering resource-needs, capacity, and willingness) a. What are the factors that promote or hinder the long-term sustainability of the LLP￾implemented in-service professional development programs for teachers and MOE’s professional staff? b. How could a follow-on activity be more sustainable? What are potential models which could be implemented to achieve similar objectives of teacher training? For Component B 4. In what ways have the youth engagement activities been beneficial to youth participants and their communities? 5. How can USAID work most effectively with the Municipal Youth Offices? (considering that several USAID interventions work with them) 6. What lessons can be learned from the development, validation and implementation of the tests for functional skills in reading, writing and mathematics for OSY? a. What is most relevant for the MOE? b. What is most relevant for youth-serving organizations? 3.2 Anticipated Data Collection Methods To meet the objectives of the performance evaluation and successfully answer the evaluation questions above, the evaluator should perform at least the following tasks: Desktop Research and Document Review: Review all relevant implementer reports including the Monitoring and Evaluation reports, work plans, and quarterly performance reports as well as documents on the Guatemalan context. In addition, reviewing the contract and studies done by the LLP are key to understand the intervention logic. A number of key documents are listed at the end of this SOW. Semi-structured Key-Informant Interviews or Focus Groups: Interview project staff and key stakeholders. A list of key informants will be provided by USAID/Guatemala, but the evaluation 37 team should not necessarily consider that list complete or sufficient for data saturation. Departmental and municipal level actors should be included as key informants. The evaluators may interview any other relevant individuals necessary to answer the above research questions. Analysis: Analyze qualitative and quantitative data obtained during field work and secondary analysis. This may include qualitative analysis and simple descriptive statistics. All project databases will be available to the evaluation team. Evaluators are expected to discuss with the activity manager the type of analysis that will best fit this performance evaluation. The evaluator should propose additional data collection and analysis methods, if needed. IV. Applicable Documents USAID/Guatemala will make the following documents available to the evaluation team upon signature of the award: ● LLP’s Section C of the Contract and the TEC increase modification ● Mid-term evaluation reports ● Relevant deliverables such as LLP strategic documents, work plans, and reports ● List of key informants ● List of project-affected schools and contact information by municipality ● USAID Education Strategy ● USAID/Guatemala Country Development Cooperation Strategy (CDCS), Public Version. ● USAID Evaluation Policy V. Technical Requirements USAID/Guatemala envisions the evaluation being conducted by a team that will perform a combination of desk review and in-person data collection in Guatemala. The Contractor shall propose any team members or team composition necessary to accomplish the tasks as long as the team meets the requirements listed below. In addition, USAID/Guatemala’s staff may accompany the team on its fieldwork and key informant interviews, and participate in team discussions on analysis and conclusions. 5.1 Evaluation Team The team should meet the following minimum requirements: ● All team members must be proficient at reading and speaking Spanish, including specialized vocabulary related to public education. ● At least one native English speaker, with the ability to communicate technical information clearly and concisely in written form. ● Together the team must have experience in primary education, development and tracking of reading skills, bilingual education (indigenous languages and transition to second language), institutional strengthening, youth education, workforce development, technical and vocational education, and teacher training. 38 ● The team leader must have a minimum of 10 years of experience in projects’ evaluation and at least five years of experience in evaluation of education projects. ● Team should include at least one local consultant with at least five years of experience working in the education sector in Guatemala. ● The Local consultant(s) must not has/have had a contractual relationship (i.e., consultant, employee) with Juárez and Associates and ChildFund under the LLP during the six months prior to the date the technical proposal for this evaluation is submitted to USAID. VI. Technical Direction The activity manager for the performance of this evaluation will be Ligia María Alfaro from the PPSO. VII. Deliverables ● Draft Work Plan and Methodology: Must be submitted in English (electronic copy only) no later than five days after the signature of the modification. The work plan shall describe the proposed evaluation design (data collection and analysis methods) and all activities required to implement the evaluation and related deliverables, including: - Sequencing and timeframes for document review, field work, analysis, major briefings or other deliverables. - Research design and evaluation methods. - A plan for quantitative and qualitative data analysis. - A proposed schedule of interviews and focus groups including a description of the criteria for selecting locations and respondents (e.g., random or purposive). - Draft data collection instruments, including questionnaires. ● Final Work Plan and Methodology: USAID will provide feedback to the draft work plan within 7 working days. The evaluators will incorporate or respond to comments and submit a final version 5 working days after the feedback from USAID has been sent. ● Up to four progress meetings with summary progress reports: Meetings may include other USAID staff. Progress reports will be submitted electronically and in English beforehand. ● Presentation of preliminary results to USAID and implementer: Approximately four weeks after the work plan approval and prior to the departure of any international consultants. ● Draft version of the evaluation report: Must be submitted electronically in English within five working days of the preliminary presentation. USAID will comment within one week. ● Final version of evaluation report: The final report should be no more than 25 pages, should start with an Executive Summary in English and Spanish which can be distributed as a solitary document to partners and MOE. The final report must comply with the 39 USAID Evaluation Policy as it relates to performance evaluations, and should use the criteria for quality evaluation reports listed in Appendix I of the Evaluation Policy. ● Presentation of the evaluation report to USAID and implementer: A presentation of the final report to relevant USAID/Guatemala and Juárez and Associates’ staff at USAID’s offices. ● Submit final report to the Development Experience Clearinghouse (DEC): The final approved report must be submitted directly to the DEC at USAID within one month of approval. VIII. Timeline USAID expects that this activity be completed in approximately ten weeks. The final report needs to be approved no later than August 20, 2018. 40 ANNEX II: DATA COLLECTION INSTRUMENTS LLP Final Evaluation August 2018 ENTREVISTA MINISTRO DE EDUCACION 1. Introducción objetivo de la evaluación – propósito, tiempo, audiencia – refiérase a la carta. 2. Preguntar su experiencia/impresiones del proyecto (mencionar los dos componentes) – valor de todas las actividades y logros. 3. Preguntar su análisis sobre la importancia de los componentes y la planificación del Ministerio en todos estos temas. - Enfoque en aspectos bilingües de educación - Educación para jóvenes fuera de la escuela - Evaluaciones y pruebas - Problemas para la población rural (comparado con la población urbana) 4. En su opinión, de qué manera podría USAID apoyar o encajar en los programas y planes dentro del Ministerio, de ahora en adelante? 5. Otras áreas prioritarias para el Ministerio de Educación, mirando hacia el futuro. 41 LLP Final Evaluation August 2018 ENTREVISTA VICEMINISTERIOS Y DIRECCIONES GENERALES MINISTERIO DE EDUCACIÓN 0. Introducción a la evaluación y el propósito de la reunión. 1. ¿Cuál ha sido la experiencia del Vice despacho/Dirección General con el Proyecto LEA? 2. ¿Cuáles son sus impresiones acerca de los resultados del Proyecto LEA? ¿Cuáles actividades del Proyecto han sido las más efectivas? ¿Qué barreras o dificultades se han encontrado? 3. ¿En qué nivel de la estructura de la educación de Guatemala se puede ver el mayor impacto o efecto de un Proyecto como LEA en educación bilingüe e intercultural en la escuela primaria? 4. ¿Cómo garantiza el Vice despacho/Dirección General la sostenibilidad de los componentes del Proyecto LEA en la estructura técnica del sector oficial? 5. ¿Cuáles son las prioridades hacia el futuro, especialmente para el año entrante para educación bilingüe en primaria, educación extraescolar, capacitación de docentes y otros profesionales en educación. Se pueden identificar algunas prioridades o desafíos específicos para áreas rurales y la zona del altiplano occidental? 6. De manera prioritaria ¿Qué estrategias y acciones debe asumir el Vice despacho/Dirección General del Proyecto LEA? 7. ¿Qué ideas o recomendaciones en relación a las actividades del Proyecto LEA ha promovido y qué puede proponer hacia el futuro para otros proyectos similares o para nuevos proyectos que impulse la Misión de USAID? 42 LLP Final Evaluation August 2018 ENTREVISTA CONSORCIO DE UNIVERSIDADES POR LA EDUCACIÓN 0. Objetivo: Cómo ven la demanda de la educación universitaria para la formación de recursos humanos particularmente para docentes en servicio. 1. ¿Existe interés en su universidad a través de la Facultad de Educación para la formación de recursos humanos en apoyo a la educación escolar del país? 2. Considerando los resultados de la evaluación que realiza el MINEDUC en las áreas de lectoescritura y matemáticas ¿Existe interés en su universidad a través de la Facultad de Educación para formar recursos humanos en lectoescritura en idiomas nacionales y matemáticas en todos los niveles educativos? 3. En el plan estratégico de su universidad ¿Están incluidos las alternativas de solución para formar humanos en lectoescritura en idiomas nacionales y matemáticas en todos los niveles educativos? 4. Si la respuesta es positiva ¿Con qué recursos consideran implementar estas alternativas de solución? 5. ¿Han identificado demanda para implementar las alternativas de solución que desean ofertar en la formación de recursos humanos en apoyo a la educación escolar del país? 43 LLP Final Evaluation August 2018 ENTREVISTA DECANOS DE EDUCATION DE LAS UNIVERSIDADES PRINCIPALES 1. Basado en su experiencia con el proyecto LEA, cómo se evalúan los programas de LEA de educación professional con docentes y supervisores en el sistema público de educación (diplomados/ licenciaturas /maestrias)? - En el caso de esta universidad, ¿Hay indicadores del valor positivo del programa para los docentes y otros profesionales que ya participaban? - ¿Se han visto beneficios para la universidad o para la facultad de educación? 2. ¿Qué problemas o deficiencias se han encontrado? Hasta qué punto, se han resuelto estos problemas o deficiencias? 3. ¿Cuáles barreras en términos de recursos ha encontrado la universidad? – o va a encontrar si continua con el programa (por ejemplo, recursos como personal calificado, materiales, costos, reclutamiento de docentes, espacio físico)? ¿No permitirá eso poder expandirla y ofrecerla en otras sedes de la Universidad?. Actualmente ya cuentan con el diseño de la carrera y una prueba de su aplicación en una región del país. ¿Qué dificultaría hacerlo?" En cuanto a las barreras que la universidad ha encontrado o va a encontrar si el programa sigue: 4. Asumiendo que USAID no puede continuar a proveer subvenciones de apoyo al programa, se puede identificar otras posibles fuentes de recursos para poder continuar con el programa o para superar estas barreras? Discusión y ejemplos. - Basado en su experiencia, hay demanda suficiente para este tipo de educación continua o profesional para sostener el programa hacia el futuro? (Hay barreras o condiciones que puedan limitar esta demanda efectiva?) 5. Se interesa esta universidad en continuar a ofrecer esta clase de educación profesional? Bajo qué circunstancias o condicionalidades? De punto de vista de la 44 LLP Final Evaluation August 2018 ENTREVISTA TÉCNICOS DE JUÁREZ. TEMA: MATERIALES Y FORMACIÓN DE RECURSOS 0. Propósito de la reunión. 1. Descripción de los procesos del componente 1 2. Cuál es el proceso de elaboración de los materiales que integran el kit de materiales. ¿Cómo lo decidieron? ¿Cuál es el futuro del kit? ¿Cómo se va a irradiar el uso del kit? 3. Qué mecanismos hay para la utilización del kit de materiales en la escuela? 4. Qué relación tienen los materiales del kit con la formación de recursos humanos. 5. ¿Cuáles son los materiales del Proyecto LEA con más impacto en la lectura del nivel preprimario y del primer ciclo de la primaria? ¿Cuáles son los indicadores? ¿Contemplan cambios? ¿Qué informe hay sobre utilización? 6. ¿Cuáles son sus recomendaciones al Ministerio de Educación para que asuma y haga una masiva reproducción de los mejores materiales del Proyecto? 7. ¿Qué le falta al Ministerio de Educación en cuanto al uso efectivo de los materiales disponibles para el fomento de la lectura en el nivel inicial? 8. ¿Cuáles son los aportes efectivos y relevantes del Consorcio de universidades a la formación del recurso humano del Ministerio de Educación? ¿En qué colabora cada Universidad? 9. Ideas/ recomendaciones en relación a las actividades del proyecto LEA hacia el futuro. 45 LLP Final Evaluation August 2018 Entrevista: Juarez Strategic Partners, Empresarios, UNESCO, IREX, 1. Descripción del apoyo a la educación. 2. Relación con los esfuerzos de leer y aprender en la educación guatemalteca. 3. ¿Qué acciones y estrategias ejecutan de manera complementaria? ¿Con quiénes? 4. ¿Cuáles son los aportes de mayor impacto al MINEDUC? 6. ¿Cuál es la continuidad que ustedes consideran para apoyar los esfuerzos de lectura en la educación guatemalteca? 46 LLP Final Evaluation August 2018 ENTREVISTA: CHILD FUND 1. La relación con JUÁREZ y ASOCIADOS para el componente de jóvenes, ¿cómo ha sido desde su punto de vista? 2. ¿Cómo han experimentado la reacción y respuestas de los jóvenes? 3. ¿Cómo ha sido la relación con las autoridades municipales? ¿Cómo han sido sus experiencias con los alcaldes, concejos municipales y personas responsables de la oficina? 4. Continuidad, sostenibilidad y permanencia de las organizaciones de jóvenes promovidas. 5. ¿Cómo han experimentado la reacción de los adultos ante los jóvenes organizados? ¿Ayudan, interfieren, colaboran, están celosos? 6. Las redes de socios a nivel municipal o a nivel comunitario, ¿cómo han funcionado y cuáles son los aciertos y desaciertos de su funcionamiento? 7. ¿Cuáles son sus aprendizajes de este proyecto? ¿Qué cambiaría para mejorar? ¿Qué propone para otros proyectos similares? 8. ¿Cuáles serían sus consejos para los siguientes implementadores de proyectos con jóvenes? 47 LLP Final Evaluation August 2018 ENTREVISTA: PROYECTO PUENTES 1. Describir el Proyecto PUENTES en términos generales. Su inicio y su nivel de avance. 2. ¿Cómo se relaciona con el Proyecto LEER y APRENDER? ¿Cómo se relacionan con LEA? 3. ¿Qué contribución hacen con los socios (CEIBA, FUNDAP, INTECAP y otros), si son similares a los de LEA? 4. Monto presupuestario aproximado. 5. ¿Existe un sistema de recolección de datos y seguimiento de los jóvenes? 6. En cuáles municipios trabajan, con qué criterios de selección y prioridades. 7. Diferencia en el reclutamiento de jóvenes mujeres. ¿Se requieren esfuerzos especiales para incorporar a los jóvenes hombres? 8. Cómo es la relación con las autoridades municipales? Cómo interactúan con las oficinas de niñez, juventud, o mujer? ¿Cuantos municipios cubren? 9. SOSTENIBILIDAD FUTURA de este esfuerzo. Como dejar capacidad instalada en el país? ¿Cómo alcanzar una cobertura a nivel país? 10. Qué solución y resultados están funcionando bien. Barreras que han encontrado. Sus opiniones y sugerencias. 48 LLP Final Evaluation August 2018 ENTREVISTA EVALUACIÓN SOBRE PRUEBAS FUNCIONALES DE LECTURA Y MATEMATICAS PREGUNTAS para Asociación GRUPO CEIBA y Asociación para la Cultura y el Desarrollo ACUDE OBJETIVOS: 1. Conocer lo que identifica el equipo directivo sobre intereses y necesidades prioritarias de los jóvenes. 2. Establecer si los programas educativos resuelven necesidades e intereses de los jóvenes 3. Determinar si las pruebas de matemática, lectura y finanzas funcionales, miden capacidades reales en los JFE. 4. Conocer si los resultados de las pruebas han modificado algún aspecto de la entrega educativa. 5. Conocer reacciones de los empleadores y padres por los estudios académicos de los jóvenes. PREGUNTAS GENERADORAS 1. ¿Cuáles son los aspectos más positivos que destacar del trabajo de convocatoria, reunión, acompañamiento, formación, motivación y retención de los JFE a la formación y a otras actividades? 2. Obtener títulos y diplomas, ¿es un interés de los jóvenes? ¿Es suficiente estímulo para inscribirse? 3. ¿Desde cuándo aplican las pruebas? ¿Cómo las ven y sienten los jóvenes? 4. Para que los JFE tomaran las pruebas de lectura y matemáticas funcionales, ¿fue necesaria una coordinación especial, o las tomaron sin objeción ni dificultad? 5. ¿Han realizado las pruebas una o varias veces? ¿Las tomaron antes, durante y después de su inscripción? 6. ¿Ha habido agrado o desagrado por las pruebas? ¿Cómo las consideran ellos, sencillas y fáciles o complejas y difíciles? ¿Han pedido conocer los resultados individuales de estas pruebas? 7. ¿Las pruebas a los jóvenes, ayudaron a la retención o causaron deserción? ¿Cómo ha sido la retención y permanencia de los JFE en los programas de formación académica? 49 LLP Final Evaluation August 2018 JUAREZ – QUETZALTENANGO 1. Queremos entender la forma de trabajar – como empezaron aquí en la región? Los contactos y actividades principales? Como ha cambiado su modo de trabajar durante los cinco años? Que es el plan para la salida (cuando cierre el Proyecto LEA)? 2. En su opinión, pensando en Componentes A y B, en qué nivel ha tenido más éxito o impacto – al nivel Departamental, municipal, o en las aldeas? Cuáles son las contribuciones principales y los menos exitosos? (por qué) 3. Pensando en el Componente A (lectoescritura), cuáles son sus impresiones de la distribución, utilidad, y aplicación actual del kit de materiales? ¿Qué impacto ha visto? ¿Cuál es su visión de la aplicación de estos materiales hacia el futuro? 4. En cuanto a la capacitación de los docentes, cuáles son sus observaciones del impacto de la participación de docentes con los programas de las universidades del diplomado y maestría – con los docentes propios, las escuelas y comunidades, el municipio, y al nivel departamental? Como aprecia la importancia de estas ofertas hacia el futuro (e.g., hay interés, utilidad)? Pensando hacia el futuro, que importancia tienen las becas? ¿Hay otros factores importantes en cuanto a la demanda para esta capacitación? 5. Enfocando en el componente B de los JFE, como aprecia la demanda de los jóvenes para educarse y para la participación cívica? ¿Qué diferencias hay entre las jóvenes mujeres y hombres en estos dos aspectos? 6. Cuáles son sus impresiones del impacto de estas actividades para los/las jóvenes y para las comunidades? Cuales han sido los problemas o barreras principales? A qué se debe las diferencias en resultados entre las comunidades y municipalidades? 7. En los dos componentes del Proyecto LEA, Hay varias entidades (ONGs, gobierno, donantes, grupos locales) trabajando en la región en educación bilingüe de lectoescritura y con JFE: - Hay coordinación entre las entidades? Como (ejemplos) y a qué nivel? (e.g., a nivel de Proyecto, municipio, comunidad, individual?) - Como aprecia el rol ahora y hacia el futuro de la municipalidad en coordinación y en manejo propio de los programas de lectoescritura y de JFE? - Cuáles son las barreras y ventajas principales y a que se debe las variaciones hoy en día entre las municipalidades? 50 LLP Final Evaluation August 2018 ENTREVISTA A NIVEL DEPARTAMENTAL DIRECTOR DEPARTAMENTAL DE EDUCACION DIRECTOR DEPARTAMENTAL DE EDUCACIÓN BILINGÜE INTEGRAL DIRECTOR DEPARTAMENTAL DE EDUCACION EXTRESCOLAR 0. Introducción a la evaluación y el propósito de la reunión 1. Experiencia con el Proyecto LEA - Consultas y/o acompañamiento en las municipalidades - Revisión y/o distribución de materiales - Utilización de publicaciones y/o resultados de pruebas - Colaboración en cuanto a la preparación y capacitación de docentes - Apropiación y compromiso de parte de los docentes, (preguntar cómo lo han visto, se ha logrado?) - Colaboración o interferencia por parte del Sindicato de Maestros. (por ser un apoyo de USAID, podrían apoyarlo u oponerse según se les tome en cuenta o no) 2. Impresión de los resultados del Proyecto LEA. Dé su punto de vista cuáles actividades del proyecto han sido las más efectivas? ¿Qué barreras o dificultades se han encontrado? Escala o no escala. 3. Ante la poca cobertura de los ciclos básico y diversificado, hay muchos JFE. ¿Qué alternativas ofrece el Ministerio a los jóvenes? ¿Son efectivas? ¿Hay demanda de esos servicios? El proyecto LEA propone alternativas. ¿Cómo las califica? ¿Qué sugiere para el futuro? 4. En su opinión, a qué nivel de la estructura de educación en Guatemala se puede ver el mayor impacto o efecto de un proyecto como LEA?. Específicamente – del programa de lectura y escritura en educación bilingüe] 5. Dé su punto de vista, cuáles son las prioridades hacia el futuro, y especialmente en el año entrante – para lectura en preprimaria y primaria, materiales, formación de recursos humanos, educación bilingüe intercultural. 6. Ideas/ recomendaciones en relación a las actividades del proyecto LEA hacia el futuro. 7. Recomendaciones a USAID para proyectos educativos y proyectos de lectura para el futuro? 8. Enfoque de EBI: ¿Solamente el estudio de la lengua? ¿Estudio de lengua más cultura? 51 LLP Final Evaluation August 2018 ENTREVISTA: NEXOS LOCALES como otro proyecto de USAID con jóvenes. 1. Describir el proyecto NEXOS LOCALES en términos generales y su forma de relacionarse con la juventud de los municipios que cubren. 2. ¿Cuáles son los puntos de coincidencia y coordinación con el proyecto LEA? A partir de qué momento iniciaron esta coordinación? 3. ¿Existe una atención específica hacia JFE o es una atención a toda la juventud, escolarizada y no escolarizada? ¿Es parte del proyecto la formación laboral y la búsqueda de empleo de los jóvenes? 4. ¿Cómo establecen coordinación con los otros proyectos de USAID en el Altiplano, y como ven que las autoridades municipales reciben esta presencia de de diferentes proyectos? ¿No les confunde o no genera interferencias? 5. Las otras organizaciones internacionales, o las gubernamentales nacionales, ¿también forman parte de la coordinación? ¿Se toman en cuenta sus aportes para el trabajo con los municipios? 6. ¿Qué sugerencias o recomendaciones puede hacer a la misión USAID para futuros proyectos a nivel municipal? 52 LLP Final Evaluation August 2018 ENTREVISTA SUPERVISOR TECNICO ADMINISTRATIVO (CTA) 1. ¿Cuál es su apreciación sobre el desarrollo de la educación bilingüe intercultural y la lectura en el aula con el apoyo del proyecto LEA? 2. ¿Qué usos tiene el kit de materiales del Proyecto LEA en el aprendizaje de la lectura y escritura en L1 y L2? - ¿Cuáles son sus observaciones sobre la utilización de los materiales incluyendo guías para los docentes, libros para los estudiantes, y el diagnóstico del perfil lingüístico? - ¿Usted como supervisor participo en alguna capacitación u orientación relacionado a los materiales? 3. ¿Cuál es el uso futuro del kit de materiales educativos bajo la responsabilidad del MINEDUC? 4. ¿Cuál es su apreciación sobre la participación de los/las padres de familias en el aprendizaje y desarrollo de la lectura y escritura de las niñas y de los niños en su escuela? 5. ¿Cómo visualizan el futuro de la lectura y de la escritura en L1 y en L2 sin el apoyo del Proyecto LEA? ¿Ustedes están dispuestos a asumir el compromiso para la continuidad del esfuerzo por la lectura bilingüe en su escuela? 6. ¿Según su experiencia ¿en qué nivel debe USAID enfocar su apoyo en los próximos años? 7. El Proyecto LEA es de USAID ¿Qué recomendaciones le hace a USAID para los próximos proyectos de educación y de lectura? 53 LLP Final Evaluation August 2018 ENTREVISTA: COMISION DE EDUCACION MUNICIPAL 1. ¿Cuál es la apreciación del Concejo Municipal sobre el desarrollo de la educación bilingüe intercultural en el aula con el apoyo del proyecto LEA? ¿Qué apoyo ha proporcionado la Municipalidad para el aprendizaje de la lectura y escritura en L1 y en L2 en los primeros grados? 2. ¿Cuál es su apreciación sobre la participación de las familias (padres y madres de familia) en el aprendizaje y desarrollo de la lectura y escritura de las niñas y de los niños en su escuela? 3. ¿Cuál es su apreciación sobre los materiales del Proyecto LEA para la enseñanza de lectoescritura bilingüe en los primeros grados? Como se utilizan? ¿Se ha visto algún efecto nuevo debido a la presencia de estos materiales? 4. ¿Cuál es su apreciación de las oportunidades para los docentes en términos de capacitación vía diplomados u otros cursos que ha ofrecido el Proyecto LEA? 5. ¿Cómo visualizan el futuro de la lectura y de la escritura en L1 y en L2 sin el apoyo del Proyecto LEA? ¿La Comisión está dispuesta a asumir el compromiso para la continuidad del esfuerzo por la lectura en su escuela? 6. Según su experiencia ¿En qué nivel debe USAID enfocar su apoyo en los próximos años? 7. El Proyecto LEA es de USAID ¿Qué recomendaciones le hace a USAID para los próximos proyectos de educación y de lectura? 54 LLP Final Evaluation August 2018 ENTREVISTA: OFICINAS MUNICIPALES DE NIÑEZ, ADOLESCENCIA Y JUVENTUD OBJETIVOS 1. Conocer la forma en que está inscrita la oficina en la estructura municipal. Historia y nivel de integración por parte del Alcalde y la Corporación Municipal. 2. Establecer la forma como conoce y coordina con el proyecto LEA. 3. Determinar las alianzas establecidas y estrategias de trabajo. 4. Comprender si existe sostenibilidad en las actividades con JFE, al finalizar LEA. PREGUNTAS GENERADORAS 1. Nombre, fecha de creación y fines de la oficina de niñez, adolescencia, juventud del municipio. 2. ¿Cuál es el interés e importancia que el Alcalde y los miembros de la Corporación Municipal dan a la oficina? ¿Participan en sus actividades? ¿Han aprobado fondos del presupuesto municipal? 3. ¿Cómo ven la participación de los JFE en las actividades de la oficina? ¿Están satisfechos con la forma como lo hacen? ¿Hay algún aspecto que pueda ser mejorado en esta presencia y participación? 4. ¿Qué nivel de sostenibilidad tiene la oficina? ¿Continuará con el cambio de autoridades en el año 2020? ¿Piensan que pueden continuar con las actividades? 5. ¿Hay otras organizaciones, gubernamentales y no gubernamentales, apoyando el trabajo de la oficina? ¿Están coordinadas estas acciones y presencia? 6. ¿Cuáles son sus sugerencias y recomendaciones para mejorar la participación de JFE y de la cooperación internacional para la atención de los jóvenes? 55 LLP Final Evaluation August 2018 PREGUNTAS: Grupo Focal Docentes 1. ¿Cuál es su apreciación sobre el desarrollo de la educación bilingüe intercultural en el aula con el apoyo del proyecto LEA? - Enfocando en lectoescritura bilingüe en los primeros grados - Ha participado en alguna capacitación sobre lectoescritura bilingüe? (diplomado, otro?) 2. ¿Qué usos tiene el kit de materiales del Proyecto LEA en el aprendizaje de la lectura y escritura en L1 y L2? - En su aula ¿Ha usado las guías para el/la docente, libros o megalibros para los estudiantes? - Ha utilizado el diagnóstico del perfil lingüista? - En su apreciación, que impacto han tenido estos materiales? Ha visto algún impacto en cuanto a lectura, facilidad con L1 y/o L2? Ha visto algún efecto en el comportamiento de los estudiantes en el aula? - En su experiencia, que es la percepción de los/las padres de familia de los materiales y la lectoescritura bilingüe? - ¿Usted ha recomendado estos materiales a colegas en alguna ocasión? - ¿Cómo aprecia el acompañamiento hoy en día en cuanto a los materiales y la enseñanza de lectoescritura bilingüe en los primeros grados? 3. ¿Cuál es el uso futuro del kit de materiales educativos bajo la responsabilidad del MINEDUC? 4. ¿Cuál es su apreciación sobre la participación de las familias en el aprendizaje y desarrollo de la lectura y escritura de las niñas y de los niños en su escuela? 5. ¿Cómo visualizan el futuro de la lectura y de la escritura en L1 y en L2 sin el apoyo del Proyecto LEA? ¿Ustedes están dispuestos a asumir el compromiso para la continuidad del esfuerzo por la lectura en su escuela? 6. El Proyecto LEA es de USAID ¿Qué recomendaciones le hace a USAID para los próximos proyectos de educación y de lectura? 56 LLP Final Evaluation August 2018 PREGUNTAS: GRUPO DE REDES DE SOCIOS ASESORES DE LOS JÓVENES QUE INTEGRAN COMITÉS o REDES OBJETIVOS: 1. Conocer lo que propone los asesores de los jóvenes para apoyarles en organizarse en comités. 2. Identificar dificultades que tienen los jóvenes para organizarse y tener espacios de participación en sus comunidades y municipios. 3. Determinar cómo coordinan con autoridades municipales o departamentales. 4. Conocer reacciones de los adultos hacia la organización de jóvenes. PREGUNTAS GENERADORAS 1. ¿Qué áreas del Proyecto consideran las más importantes y por qué? 2. ¿Qué áreas requieren más tiempo de formación? ¿Algún área necesita menos tiempo, se puede reducir o se puede suprimir? 3. ¿Cómo observan el liderazgo de los jóvenes? ¿Existen riesgos en ese liderazgo? 4. ¿Tendrán continuidad los comités y las redes al finalizar el Proyecto Leer y Aprender? ¿Por qué? 5. Los padres, madres, adultos en general, ¿Ven positivamente que los jóvenes se organicen o tienen recelos, suspicacia, o desconfianza al verlos organizados? 6. Las pruebas de matemáticas y lectura, ¿considera que son adecuadas? ¿son fáciles o difíciles para los jóvenes? ¿Les ayuda a conocer sus resultados? 57 ANNEX III: SOURCES OF INFORMATION BIBLIOGRAPHY 1. DevTech Systems, Inc, Midterm Evaluation Final Report, December 2016. 2. Juárez & Associates, Inc, Segundo Informe Trimestral. April 2018. 3. Juarez & Asspcoates. Inc. Informe Anual AF 2017. Cuarto Informe Trimestral. October 2017. 4. Juarez & Associates, Inc, Requerimiento A.3.1, September 2014 5. Juarez & Associates, Inc, Requerimiento A.1.1 Estrategia de Implementacion EBI. September 2014. 6. Juarez & Associates, Inc, Requerimiento A.2.2 Estrategia Global de Formación. September 2014. 7. Juarez & Associates, Inc, Updated M&E Plan, January 2018 8. Juarez & Associates, Inc. Sistematizacion de la Estrategia de Reduccion de Barreras de Acceso a la Educacion y Formacion Laboral de Jovenes Fuera de la Escuela. July 2018. 9. Juarez & Associates, Inc. Orientaciones Tecnicas para el Funcionamiento de las Oficinas Municipales de la Juventud (OMJ). August 2018. 10. Nexos Locales (DAI). Ruta Tecnica, Juridica y Politica para la Implementacion de Oficinas Municipales de la Juventud. November 2017 11. Plan Intenacional (sub-contratista de Juarez & Associates in LLP) Informe Integrado de Estudios de Capital Social en los Municipios de Intervencion. May 2014. 12. USAID, Diagnostico Participativo de Jóvenes. Resumen del Reporte Final. May 2015 13. USAID, Education. Opportunity Through Learning. Education Strategy. February 2015. 14. USAID, Guatemala Country Development Cooperation Strategy, 2012-2016. March 2012. 15. USAID, Contract Section C effective as of modification 7, without date. 16. USAID, Outline Component B, without date. 17. USAID, Results Framework Component A, without date. 18. USAID, Results Framework Component B, without date. 58 LIST OF PERSONS INTERVIEWED RESUMEN TOTALIDAD DE PARTICIPANTES 258 PERSONAS Mujeres 138 115 jóvenes fuera de la escuela Mujeres: 75 Hombres: 40 50 Docentes Mujeres: 27 Hombres 120 Hombres: 23 METODOLOGIA Componente A 9 grupos focales docentes: 50 docentes Componente A y B 15 grupos focales: 165 participantes Componente B 7 grupos focales jóvenes fuera de la escuela: 115 Componente B 5 grupos focales de jóvenes líderes y miembros de comités Componente A y B 93 entrevistas individuales y colectivas Total: 21 grupos focales 59 CIUDAD DE GUATEMALA Location Date Name Position/ Organization Male/Female Component Guatemala City 08/13/18 Fernando Rubio Director Leer y Aprender/Juarez & Associates Male A-B Guatemala City 08/13/18 Gabriela Núñez Subdirectora Leer y Aprender/Juarez & Associates Female A-B Guatemala City 08/14/18 Liza Hernandez Directora de Estrategia y Alianzas Proyecto PUENTES/ World Vision Female B Guatemala City 08/14/18 Raquel Montenegro Especialista en Español Female A Guatemala City 08/14/18 Sophia Maldonado Especialista en educación Female A Justo Mactzul Especialista en EBI Male A Guatemala City 08/14/18 Julio Estrada Subdirector Proyecto PUENTES/World Vision Male B Guatemala City 08/14/18 Lesley Rosales Especialista de Monitoreo y Evaluación Leer y Aprender/Juarez & Associates Female A-B Guatemala City 08/14/18 Rodrigo Bustos Director/Child Fund Male B Guatemala City 08/14/18 Fermín Cojon Gerente Subcomponente B3 Leer y Aprender/Child Fund Male B Guatemala City 08/14/18 Verónica Spross Directora Ejecutiva/Empresarios por la Educación Female B Guatemala City 08/16/18 Lucia Verdugo Oficial de Educación/ UNESCO Female B Guatemala City 08/17/18 Tania Eguizábal Directora Maestría en Lectoescritura Para Ambientes Bilingües Interculturales/UPANA Female A Guatemala City 08/17/18 Héctor Alejandro Canto Mejía Viceministro Técnico de Educación/Ministerio de Educación Male B Guatemala City 08/17/18 Daniel Domingo López Viceministro de Educación Bilingüe Intercultural/Ministerio de Educación Male A Guatemala City 08/30/18 Oscar Hugo López Ministro de Educación Male A Guatemala City 08/30/18 Barton Pogue Oficial de Educación USAID Male A-B Guatemala City 08/30/18 Laura Villegas Oficial de educación USAID Male A-B Guatemala City 08/31/18 Cecilia Garces Population Council Female B 60 Guatemala City 08/31/18 Claudia Polillo Sandra Toc ACUDE Female B Guatemala City 08/31/18 Todd Amani IREX Male B Guatemala City 08/31/18 Miguel Angel Bayardo Universidad de San Carlos Male A Guatemala City 09/03/18 Wendy Rodriguez DIGEEX Female B Guatemala City 09/03/18 Aroldo David Noriega DIGEEX Male B Guatemala City 09/04/18 Miguel Angel Franco, Nidia Gerogis Orozco, Marcela Pereira, Consorcio de Universidades Representante de Landívar, ITSMO, UPANA, GALILEO y Mariano. 4 females And 1 Male: 5 participants A Guatemala City 09/04/18 Samuel Puac DIGECUR Male A Guatemala City 09/05/18 Ana María Hernández Directora DIGECADE Female A Guatemala City 09/05/18 José Moreno Viceministro DIGECADE Male A Guatemala City 09/06/18 Lic. Carlos Coz DIGEBI Male A Total: 16 Females and 17 Males: 33 participants 61 DEPARTAMENTO QUETZALTENANGO Location Date Name Position/ Organization Male/Female Component Ciudad Quetzaltenango 08/20/18 Salome Gonzalez y Gonzalez, Coordinador de evaluación educativa/Dirección departamental de Educación Male A Ciudad Quetzaltenango 08/20/18 Juan José Colón, Dirección departamental de Educación Male A Ciudad Quetzaltenango 08/20/18 Catarina Chay Ordoñez, Coordinadora de Entrega Educativa/ Dirección departamental de Educación Female A Ciudad Quetzaltenango 08/20/18 Gladis García, Coordinadora de preprimaria y Formación Docente/ Dirección departamental de Educación Female A Ciudad Quetzaltenango 08/20/18 Susana Escobar. Dirección Departamental de Educación Female A Ciudad Quetzaltenango 08/20/18 Leslie Staackman Coordinadora General/ Universidad del Valle Sololá diplomados de LEA Female A Ciudad Quetzaltenango 08/20/18 Ventura Salanic Felipe Orozco Dewi Rosal Hipólito Hernandez Diana Ventura Julio Cesar Walter Entrevista colectiva Juarez & Associates (Kit de materiales) Coordinador Regional Acompañante Pedagógico Mam) Coodinadora Regional Subcomponente B1 Acompañante pedagógico Kiche, Coordinadora Regional Subcomponente A3 Encargado de tema de inserción laboral con jóvenes. Encargado de Participación Ciudadana con jóvenes. 5 Males y 2 Females: 7 participants A-B Total: 8 Males y 6 Females: 14 participants 62 DEPARTAMENTO DE SAN MARCOS Y MUNICIPIO DE CONCEPCIÓN TUTUAPA Location Date Name Position/ Organization Male/Female Component Departamento de San Marcos 08/21/18 Alba Orozco Gómez, Merlín González Saturnino Juan Verdugo Nelson Méndez Bernardo Samuel Orozco Feliciano Rodolfo García Marroquín Dirección Departamental de Educación de San Marcos. Jefe Departamento Calidad Educativa. Formador de sección pedagógica (preprimaria y primaria) Sub director técnico bilingüe intercultural Jefe departamento entrega educativa Coordinador formación docente 1 Female, 7 Females: 8 participants A-B San Marcos, Concepción Tutuapa 08/21/18 Grupo de Jóvenes fuera de la escuela 9 Males, 4 Females: 13 B San Marcos, Concepción Tutuapa 08/21/18 Grupo de docentes y directores 7 Females, 6 Males:13 B San Marcos, Concepción Tutuapa 08/21/18 Profa. Dina Sofía Lainez Nolasco Oficina Municipal Dirección Municipal de la Mujer Female B San Marcos, Concepción Tutuapa 08/21/18 Samuel Luis Reynoso Concejal de comisión de educación Male A San Marcos, Concepción Tutuapa 08/21/18 Arturo Méndez Centro de Atención Integral ( CAI) Male B San Marcos, Concepción Tutuapa 08/21/18 Selvin Marroquín Centro de Atención Permanente (CAP) Female B San Marcos, Concepción Tutuapa 08/21/18 Lic. Misael Morales Coordinador Tecnico Administrativo Male A San Marcos, Concepción Tutuapa 08/21/18 Emilza Simón Secretaria de la comisión municipal de educación Female A Total: 15 Females and 25 Males: total: 40 63 MUNICIPIO DE SAN JUAN OSTUNCALCO Location Date Name Position/ Organization Male/Female Component Quetzaltenango, San Juan Ostuncalco 08/22/18 Audel Ríos Asociación para la Cultura y el Desarrollo (ACUDE ) Male B Quetzaltenango, San Juan Ostuncalco 08/22/18 Grupo focal de Jóvenes Fuera de Escuela 14 Females and 1 Male: 15 Participants B Quetzaltenango, San Juan Ostuncalco 08/22/18 Miriam Yolanda Diaz Vasquez, Elsa Morroquin Vicente Olivia Romero Gomez Irma Delgado Grupo focal Docentes de la Escuela la Victoria y Aldea Los Romeros. 4 Females B Quetzaltenango, San Juan Ostuncalco 08/22/18 Grupo de jóvenes líderes y miembros de comités B Quetzaltenango, San Juan Ostuncalco 08/22/18 Aura Rojas Oficina Municipal de Niñez adolescencia y Juventud OMNAJ Female B Quetzaltenango, San Juan Ostuncalco 08/22/18 Lic. Ruperto Meza Coordinador Técnico Administrativo Male A Quetzaltenango, San Juan Ostuncalco 08/22/18 Patricia Rosales Presidenta de la comisión de educación Male A Total: 19 Females and 4 Males: 23 64 DEPARTAMENTO DE HUEHUETENANGO Y MUNICIPIO DE CHIANTLA Location Date Name Position/ Organization Male/Female Component Departamento de Huehuetenango 08/23/18 Aroldo Sosa Román Domingo Díaz Dirección Departamental de Educación. Coordinador de DIGEEX Analista Depto de asistencia pedagógica Enlace de proyecto leer y aprender 2 Males A-B Huehuetenango, Chiantla 08/23/18 Grupo focal de Jóvenes Fuera de Escuela 8 Females y 10 Males: 18 Participants B Huehuetenango, Chiantla 08/23/18 Grupo de jóvenes líderes y miembros de comités B Huehuetenango, Chiantla 08/23/18 Reynaldo Luis Pérez Juan Carlos Jiménez Luis Marisol López, Alex Ramos, Sara Yolanda Vásquez María Magdalena Gómez Pérez, Arnoldo Hernández Grupo focal Docentes de la Escuela la Chochal y Cinco Arrollos. 6 Males and 5 Females: 11 participants A Huehuetenango, Chiantla 08/23/18 Lizandro Chavéz Oficina Municipal de la Juventud Male B Huehuetenango, Chiantla 08/23/18 Patricia López Dirección Municipal de la Mujer Female B Huehuetenango, Chiantla 08/23/18 Melvin Granados Fe y Alegría Male B Huehuetenango, Chiantla 08/23/18 Joel Isaí López Carillo Técnico de educación Male A Huehuetenango, Chiantla 08/23/18 Lic. Juan Carlos Tebalán Coordinador Técnico Administrativo Male A Total: 14 Females and 22 Males: 36 DEPARTAMENTO DE TOTONICAPAN Y MUNICIPIO 65 Location Date Name Position/ Organization Male/Female Component Departamento de Totonicapán 08/24/18 Hipólito Rosales, Higinio Us López, Juan Basilio Tahay, María Dominga Pu Tax Dirección Departamental de Educación de Totonicapán, Directora Departamental Asistente de Subdirección, Jefe Secc. De Asistencia Pedagógica. Subdirector Técnico Pedagógico 3 Males and 1 female: 4 participants A-B Municipio de Totonicapán 08/24/18 Eddin García Oficina Municipal de la Juventud Male B Municipio de Totonicapán 08/24/18 Grupo focal de Jóvenes Fuera de Escuela 19 Females and 4 Males: 23 participants B Municipio de Totonicapán 08/24/18 Grupo de jóvenes líderes y miembros de comités B Municipio de Totonicapán 08/24/18 Elvia Paulina Poncio, Margarito Chaclán, Patricia Norato, Félix Talé, Director Toribio. Grupo focal Docentes de la Escuela Chuipachec 2 Females and 3 Males: 5 participants A Municipio de Totonicapán 08/24/18 Rebeca Itzep Alfaro Coordinadora Técnico Administrativo Female A Municipio de Totonicapán 08/24/18 Leonel Ajcum, Mirna Chaclán, Tobias Tzoc Grupo focal Red de Socios: OSAR Juvenil, CDRO, Jóvenes dejando huella 1 Male and 2 Males: 3 participants B Municipio de Totonicapán 08/24/18 Salomón García Comisión Municipal Male B Municipio de Totonicapán 08/24/18 Tomas Tzul 31705326 SESAN Female B Total: 25 Females and 14 Males: 39 66 MUNICIPIO DE SAN BARTOLO AGUAS CALIENTES Location Date Name Position/ Organization Male/Female Component Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 Grupo focal de Jóvenes Fuera de Escuela 10 Females B Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 Grupo de jóvenes líderes y miembros de comités B Totonicapán, Municipio de San Bartolo Aguas Calientes Sergio Champet, Odilia Arguta, Donal Perez, German Herrera. Grupo focal Docentes de la Escuela Paxboch 1 Female and 3 Males A Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 Greysi Tzarax Oficina Municipal de Niñez, adolescencia y Juventud Female B Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 José Feliciano Perez Champet Coordinar Tecnico Administrativo distritral y tambien el representante de la comisión municipal. Male A Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 Santos Bonifacio Ajanel Consejal de Municipalidad Male B Totonicapán, Municipio de San Bartolo Aguas Calientes 08/27/18 Elisa Melina Choxom SOSEP Female B Total: 13 Females and 5 Males: 18 67 MUNICIPIO DE SANTA MARIA CHIQUIMULA Location Date Name Position/ Organization Male/Female Component Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Grupo focal de Jóvenes Fuera de Escuela 9 Females and 7 Males: 16 participants B Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Grupo de jóvenes líderes y miembros de comités B Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Walter Tumax, Miguel Tun, Melchor Tzunux, José Tzunux, Vitoria Tojin, Mario Reyes, Esperanza Ponce, Eugenia Pu. Grupo focal Docentes de la Escuela Oficial Urbana Mixta “Mario Méndez Montenegro" y Escuela de Paraje Chijoj Aldea el Rancho. 3 Females and 5 males: 8 participants A Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Marcos Chacaj Coordinador oficina de la niñez y juventud Male B Totonicapán, Municipio de Santa María Chiquimula 08/28/18 María Eugenia Tzunun Ministerio de trabajo Female B Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Tomás Tzunux Director de la Escuela Oficial Urbana Mixta “Mario Méndez Montenegro" Male A Totonicapán, Municipio de Santa María Chiquimula 08/28/18 Mauricio Osorio Coordinador Técnico Administrativo Male A Total: 13 Females and 15 Male: 28 participants. 68 DEPARTAMENTO DE QUICHÉ Y MUNICIPIO DE JOYABAJ Location Date Name Position/ Organization Male/Female Component Departamento de Quiché 08/29/18 Jose Eduardo Ajanel Ventura, Angelina Lux. Elizabeth Chuc. Dirección de Departamental de Educación de Quiché Director de Centro educación extraescolar voluntariamente. Asistente de Coordinación de DIGEEX, Coordinadora de Educación Escolar. 2 Females and 1 Male: 3 participants. A-B El Quiché, Municipio de Joyabaj 08/29/18 Grupo focal de Jóvenes Fuera de Escuela 9 Females and 8 Males: 17 participants B El Quiché, Municipio de Joyabaj 08/29/18 Grupo de jóvenes líderes y miembros de comités B El Quiché, Municipio de Joyabaj 08/29/18 Vilma Lorena Baten, Lilia González, Carmen Mejicanos, Francisca y Enma Grupo focal Docentes de la Escuela Caserio El Temal. 5 Females A El Quiché, Municipio de Joyabaj 08/29/18 Carolina Canil Morales Oficina municipal de la Niñez y la Juventud Female B El Quiché, Municipio de Joyabaj 08/29/18 Lic. Clemente Sanchez Coordinador Técnico Administrativo Male A Total: 17 Females and 10 Males: 27 participants 69 ANNEX IV: DISCLOSURE OF ANY CONFLICTS OF INTEREST 70 71 72 73 74 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Juárez y Asociados, Inc. Proyecto Leer y Aprender Lifelong Learning Project 12139 National Boulevard • Los Angeles California 90064 • USA • +1 (310) 478-0826 • F +1 (310) 479-1863 Avenida Reforma 6-64, Zona 9, Plaza Corporativa Reforma • Guatemala • T (502) 2-390-6700 • F (502) 2-390-6731 Guatemala, December 7, 2018 Ligia Maria Alfaro Planning and Program Support Office USAID/Guatemala Dear Ms. Alfaro: Juarez & Associates appreciate the opportunity to participate in the presentation of the Final Performance Evaluation Report of the Lifelong Learning Project conducted on November 29th. We acknowledge the importance of the recommendations provided to the Mission in the report. We will like to add some clarifications to contribute to the purpose of the final evaluation: “the purpose of the evaluation is to provide answers to key questions that will facilitate a technically strong program closeout and inform future USAID / Guatemala education programming”. These clarifications contain some details related to the current contract under which LLL project works, the technical design of the project, and contextual and political factors, under which the project implemented in 2015. Clarifications to the Final Performance Evaluation of the USAID/Guatemala Lifelong Learning Project In accordance with the design of the project, the different interventions of Component A, particularly those related to professional development programs through universities, the operationalization of the Sinae, the development of educational materials and other elements of the model of literacy in bilingual and intercultural environments and support to the Ministry of Education (MoE), were developed and implemented according to the rhythm of differentiated implementation that the Ministry of Education has required. This led to some of the interventions being done in parallel and others in a sequential manner. Additionally, the set of elements of the model is interrelated to achieve the effect of improving the learning of boys and girls. Therefore, the model must be understood as the reference framework of an intervention project, where each of the actors involved at the central, departmental, municipal and local levels plays a specific role without being required to dominate the integrality of the model or have knowledge of all interventions related to the model. The important thing is that they recognize and participate in the levels that correspond to each them. The focus of the Learning to Read component has been to provide technical advice to the Ministry of Education, which is responsible for the delivery of services. Therefore, we emphasize the distinction between the functions that correspond to the project and those that are the responsibility of the Ministry of Education, both at central, departmental and local levels. The project carries out the technical delivery and accompaniment of the processes to the departmental teams of the Ministry of Education, but the continuous support and guidance regarding the methodology, appropriation and monitoring of the processes at the level of schools, principals and teachers is a function of the Departmental Directorates of Education. With the purpose of achieving greater effectiveness in terms of the implementation of the literacy model in bilingual and intercultural environments, the project has recommended that the MoE locate bilingual teachers with experience in the first grades; however, this is an administrative matter that concerns the MoE to attend to it. Since there is not yet a normative that supports it (as the current law does not permit authorities to mandate grades in which teachers must teach) , this decision relies in the school principal and/or the appointed teacher, and there is little that the departmental directorates can do to enforce it. The changes of ministerial administration and priorities of the Ministry of Education made the provision of technical assistance more complex, and so it was the possibility of giving continuity to some of the processes. This is the case particularly with priority given to the reading and reproduction of some of the educational materials that are part of the literacy model. However, the project worked in collaboration with the Ministry in the production of educational materials to support the teaching of literacy, responding to the priorities raised by the new administration. Such is the case of the books targeting teachers to teach them how to teach early grade literacy and reading comprehension, anthologies, among others. The central MoE and particularly the Educational Materials Management Group at the regional level have participated in the design, elaboration and validation of materials. The Academy of Mayan Languages of Guatemala has provided linguistic certification in Mam and K'iche '. Conclusions of Question 2 in the evaluation report mentioned that the project delivered only one kit of materials per school, so it seems that the project should have provided all teachers with materials. The contract states that this must be the responsibility of the Ministry of Education. According to the amendment No. 7 of the contract AID-502-C-140001 Section C between USAID and Juarez & Associates, Inc., page 11 of 34 indicates "USAID does not anticipate funding for the printing and distribution of materials" and the project is requested do not substitute the material endowment function that the Mineduc must carry out. The project, as part of the TEC increase, reproduced a certain amount of materials to support the implementation of the literacy model, as recommended by the midterm evaluation. Further, we want to underscore that, although one kit of materials was delivered per school, the number of copies for some of the materials in the delivered kit varied according to the number of sections per grade, that is, according to the size of the school. The kit included the same resources for all the schools, differentiating by mother tongue in Mam and K'iche' and by whether the material was addressed to all teachers, only to the directors, or only to pre-primary and first-grade teachers, or to the teachers of second and third grade. The information on the number of teachers in the schools is provided by the Ministry of Education, which is not broken down according to the grades they serve. The training in using resources in the kit was also differentiated, as described. With respect to the findings presented in question 2 on the difficulty of applying the methodology and the use of materials for the teaching of literacy, the project developed and made explicit in the training of the departmental teams the application of tools such as linguistic profile, the transfer processes from L1 to L2, and methodological guidelines in Spanish for monolingual teachers on the use of materials in Mayan languages for literacy learning. Also, in relation to another of the findings on the fact that the materials made in the Mam used in San Marcos are not appropriate for Huehuetenango, it is necessary to indicate that the approach taken by the project is the one approved by the Academy of Mayan Languages of Guatemala, which promotes the standardization of the written language. This explained the effort by the LLL project to obtain linguistic certification from the Academy of Mayan Languages of Guatemala, by local law the entity with regulatory authority in these matters. The report points out, in the opportunities and challenges section, that despite the acknowledgment of the value given to reading in the communities by the informants, the lack of support for the school and the bilingual and intercultural approach of reading and the education is insufficient. Although we mostly agree with the previous statement, since 2016 USAID Lifelong Learning project, has systematically monitored and evaluated the progress of all activities in the intervention areas, 216 communities, not the more than one thousand communities in the 12 intervention municipalities, given the available resources. From this initial phase, fathers and mothers experience positive changes, and positive reactions about the "classroom for parents" have been given by teachers and school principals. Through interviews, leaders of these groups, especially mothers, have manifested the change in reading performance of their sons and daughters and their constant involvement at home in learning to read. Likewise, the fathers and mothers interviewed from the different groups where the intervention was carried out expressed their support for the learning of literacy in maternal and Spanish Mayan language. Regarding question 3, the report raises as a general problem the absence of incentives for those who complete the professional development programs . It is the MoE that should provide the incentives. The report mentions in the recommendations the need to conduct research on potential incentives. In 2013, USAID / Education Reform in the Classroom made a policy recommendation regarding incentives to teachers graduating from these types of programs, based on an incentive study carried out by Anthony Dewees, PhD. This proposal that has not been implemented by the two ministerial administrations that have led the Ministry of Education since then. It is necessary to clarify that all the participants of the professional development programs are permanent employees of the Ministry of Education. Therefore, obtaining a job in the Ministry of Education is not an incentive. The exception is when graduates participate in opposition competitions to work in Sinae, which is attractive only for those who have salaries lower than those established to work in the Sinae. About Question 4 on component B Education for employment and lifelong learning it is necessary to indicate that the original design of the project did not include the issue of sustainability as one of its purposes. Its purposes were to work on the design of an intervention that promotes alternative basic education, job training and the participation of outside of school youth. USAID | Guatemala continues to support these efforts through the Puentes Project. Regarding the approach of question 6 on functional tests, we emphasize that its primary purpose is to evaluate the functional competences of the young people who participate in the training programs of the project. The development process continues, and this effort will continue in the Mineduc over the next few years. To conclude, it is important to note that there have been contextual factors that have influenced the strategic decisions of the project, as was the case in 2015 of the resignation of the education minister and the collapse of the government with the imprisonment of the president and vice president, among other high level government officials. This caused a situation of distrust in public institutions, having for example an effect on the high dropout of teachers of the Literacy Diploma Program in the intervention area of the project. Again, we are grateful for the opportunity to comment on this report, and we commend it. Sincerely, Fernando Rubio Director