Final Evaluation IDEA Project, SC/USDA September 2018 Prepared by ADOC (Associación de Desarollo Organizacional Comunitaria) and Kristin Rosekrans with the support of Mario Moreno FY 2014 USDA McGovern-Dole IDEA Guatemala Project Final Evaluation Program: McGovern Dole Agreement Number: FFE-520-2014/043-00 Funding Year: Fiscal Year 2014 Implemented by: Save the Children Evaluation Authored by: Associación de Desarollo Organizacional Comunitaria (ADOC) Kristin Rosekrans Mario Moreno DISCLAIMER: This evaluation was conducted by an independent third party firm. The views expressed in this report do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this report can be made available upon request by emailing FAS.monitoring.evaluation@usda.gov. 2 Table of Contents List of Tables, Figures, and Graphs……………………………………………………………………………………………………………….3 Acronyms and Abbreviations………….……………...……………….…………………………………………………...………………………4 I. Executive Summary…………………………………………………..…………………………...…………………………………………….6 II. Context……...……………………………………………………………..………………………………………………………….………..…….8 III. Project Description ................................................................................................................................................................................9 IV. Project Evaluation Design………………………………………………………………………………….………………………………..12 V. Impact of Strategic Objective 1: Improve literacy levels in school age children………………………………………23 VI. Impact of Intermediate Results of Strategic Objective 1: Support to Improve Reading….................................…36 VII. Impact of Strategic Objective 2: Increase in Health and Dietary Practices................................................................. 48 VIII. External and Internal Factors that Affected the Results..................................................................................................... 59 IV. Relevance, Efficiency and Efficacy................................................................................................................................................. 64 X. Sustainability and Scalability……...…………………………………………...……………………………………………....................74 XI. Conclusions………………….……………………………………………………………………………….…………………………………….81 XII. Recommendations………………………………………………..……………………………………….....................................................84 XIII. Best Practices………………………………….…...………………………………………………………………………………………….....88 XIV. Annexes………………………………………….………………………………………………………………………………………………….91 Annex 1. Reliability of literacy skills subtests Annex 2: Literacy Boost Test (Spanish and K'iche’) Annex 3. Explanation of the statistical models Annex 4. Distribution of Non-Readers, Beginners, and Readers Annex 5. Statistical results of literacy abilities and associated covariates Annex 6. Literacy abilities by sex and area 3 List of Tables, Figures, and Graphs Table 1. Initial student registration for second and third grades, according to municipality Table 2. Distribution of sample across municipalities Table 3. School sample Table 4. Teachers’ sample Table 5. Sample of other key informants Table 6. Number of schools / percentage of students that participated in the study, by municipality and sex Table 7. Percentage of students that participated in the study, by intervention area and sex Table 8. Percentage of teachers that participated in the study, by sex Table 9. Number of establishments evaluated and percentage they represent Table 10. Number of people from MINEDUC that participated Table 11. Number of participants in focus groups Table 12. Intervention and evaluations that were carried out Table 13. Summary of second grade Spanish reading abilities by year and area Table 14. Summary of second grade K’iche’ reading abilities by year in area 2 Table 15. Summary of Ixil reading abilities by year in area 1 Table 16. Summary of third graders Spanish reading abilities by year in area 2 Table 17. Summary of third grade K’iche’ reading abilities by year in area 2 Table 18. Number of teachers//assistants from schools trained or certified as a result of USDA assistance Table 19. Number of textbooks and other learning materials provided as a result of USDA assistance Table 20. Observation of the use of methodology and tools in the classroom Table 21. School feeding indicators Table 22. Types of printed materials in areas 1 and 2 in the midterms and final evaluations Table 23. Reading practices in the home Table 24. Reading practices in the community Table 25. Number of PTA/similar school management structures supported as result of USDA assistance Table 26. Situation of school kitchens Table 27. Number of students that receive deworming medication Table 28. Storage conditions Table 29. Access to water Table 30. Situation of school warehouses Table 31. Situation of sanitation services Table 32. Children’s’ hand washing practices Table 33. Situation of the school kitchens Table 34. Number of public-private partnerships formed as a result of USDA assistance Table 35. Organizations that provide support in area 1 and area 2 Figure 1. Process for qualitative data collection Figure 2. Analysis of qualitative data Figure 3. Distribution of second grade students’ Spanish reading comprehension levels Figure 4. Distribution of second grade students’ Spanish reading comprehension levels by sex Figure 5. Distribution of second grade students’ K’iche’ reading comprehension levels Figure 6. Distribution of second grade students’ Ixil reading comprehension levels Figure 7. Distribution of third grade students’ Spanish reading comprehension levels Figure 8. Mean Spanish abilities comparison between intervened areas across time Graph 1. Second grade Spanish reading abilities in area Graph 2. Second grade Spanish reading abilities in area 1 Graph 3. Second grade Spanish reading comprehension in by area and sex Graph 4. Second grade K’iche’ reading abilities in area 2 Graph 5. Second grade Ixil reading abilities in area 1 Graph 6. Third grade Spanish reading abilities in area 2 Graph 7. Third grade K’iche’ reading abilities in area 2 Graph 8. Existence of Book Banks and implementation of Reading Buddies and Reading Camps Graph 9. Percentage of children that pay attention or not; teacher perception Graph 10. Percentage of children that pay attention or not; student perception Graph 11. Percentage of Children that said they had breakfast in the morning Graph 12. Student responses about school providing food Graph 13. Number of times per day that children are hungry; student/self-perception Graph 14. Other organizations that provide support Graph 15. Type of report received by other sources 4 Acronyms and Abbreviations ADOC Asociación de Desarrollo Organizacional y Comunitario Area 1 Nebaj, Chajul and Cotzal Area 2 Cunen, Uspantán and Sacapulas BL Baseline COCODE Community Development Council CODEDE Departmental Development Council CSB Corn Soy Blend CTA Technical Administrative Coordinator ECD Early Childhood Education FE Final Evaluation GF Focus Group IDEA Investment for Educational Development in the Highlands INCAP Institute of Nutrition of Central America and Panama INE National Statistics Institute L1 Primary Language L2 Secondary Language MT Mid Term MAD Minimum Acceptable Diet MGD McGovern-Dole Food for Education Program MESAN Better Healthy Schools MOE Ministry of Education MOH Ministry of Health MTE Midterm Evaluation MUW Most Used Words M&E Monitoring and Evaluation NGO Non-Governmental Organization PTA Parent-Teacher Association PP Percentage point Q Symbol for Guatemalan currency (Quetzales) SAN Food and Nutritional Security SC Save the Children SESAN Secretariat for Food and Nutrition Security 5 SEGEPLAN Presidential Secretariat for Planning and Programming SO Strategic Objective TDR Terms of Reference TM Metric Tons UNDAF UN Development Assistance Framework UNICEF United Nations International Children's Emergency Fund USAID United States Agency for International Development USDA United States Department of Agriculture wcpm Words correct per minute 6 I. Executive Summary The final evaluation of Save the Children’s four-year program, Investment for Educational Development in the Highlands Project (IDEA), used a mixed methods design to determine the effects of the project since its implementation, taking into account the baseline and midterm evaluations to conduct a comparative analysis. It aimed to answer the questions: (1) What results and impact has the project produced?, (2) How effective and efficient has the project been in meeting it objectives?, (3) How relevant is the project to the country’s context?, (4) To what extent are the project’s strategies sustainable, scalable, and replicable?, (5) What enabling and conditional factors (both internal and external) have supported or limited the project’s capacity to achieve its main results?, and (6) What recommendations stem from the evaluation results to ensure the achievement of the project's objectives and project sustainability? Overall, the final evaluation of the IDEA project revealed many positive results, including changes from the time of the midterm evaluation. There are a few areas in which there was little or no improvement, but the majority of interventions have shown a positive impact. The trend for the percentage of second graders that can read with comprehension in all three languages shows an increase in “beginners” and a decrease in both “readers” and “non-readers”. The percentage of children that can read with comprehension in Spanish showed an increase from the midterm to the final evaluation in the “beginners” category from 36% to 44%, a decrease of from 20% to 16% in the “readers” category, and a decrease of in the percentage of “non-readers” from 20% to 16%. Changes in the percentages of second grade students that can read with comprehension K’iche’ were more pronounced: the category of “beginners” increased by 32 percentage points from the midterm to the final evaluation (to 65%), while “non-readers” decreased from 53.2% to 33.8%. Ixil also showed a relatively large increase in the percentage of “beginners” (12% to 36%) and decrease in readers (28% to 9%). In comparison to second grade levels, in third grade the percentage of “readers” shows a slight increase of 20% to 24% percent from the midterm to the final evaluation as well as an increase from 56% to 59% in the “beginners” group. Thus, even though percentages in the “readers” category don’t show an increase, the decrease in “non-readers” for both grades, both geographical areas, and all three languages reflect a positive trend towards students developing reading comprehension. Regarding specific literacy abilities, there was an overall (statistically significant) improvement from the baseline to the final evaluation for reading comprehension in second grade in all three languages and both geographical areas and an overall improvement from the baseline to the final evaluation in all K’iche’ and Ixil literacy abilities. Furthermore, for all literacy abilities in both languages (except for “% of correct letters” in K’iche’) increases were statistically significant. Additionally, as with second grade Spanish reading abilities, area 1 reflects a decrease in some of the literacy components between the midterm to the final evaluation, while in area 2 almost all of the components show a continual improvement over time (2015- 2018). Reading comprehension is the only literacy ability that reflected a statistically significant difference between boys and girls over time in both areas, with boys performing better. Furthermore, the covariate “having read to someone in the community” was positively associated with literacy skills in all languages. All third grade literacy abilities in Spanish and K’iche’ reflect improvement from the midterm to the final evaluation. Additionally, the positive change was statistically significant for third grade Spanish and K’iche’ “% of correct letters” as well as for reading comprehension for third grade K’iche’ literacy. Additionally, covariates that were positively associated with third grade literacy abilities were “having read to someone in the community” and “someone helps you study”, as well as “had breakfast in the morning” for K’iche’. Implications of the literacy skills outcomes are that the intervention to improve teachers’ knowledge and skills and the materials provided for them should be examined to see if more emphasis should be placed on the development of each of the literacy abilities that did not show improvement, or as much improvement. Additionally, further inquiry should be done on the difference in comprehension between boys and girls. Finally, because reading to someone in the community was positively associated with all literacy abilities, this should be documented as a best practice and reinforced for all three languages. 7 During 2016 and 2017, a total of 1,485 (2016: 802 y 2017: 685) teachers and 189 principals were trained or certified. Teachers expressed that they have learned new techniques and strategies that they are putting into practice, in some cases giving specific examples. Parents commented on observing increased learning and engagement of their children, particularly with reading and writing. While the data suggest that teachers are putting into practice new approaches for literacy development, data also suggest that this isn’t the case with all teachers. Mechanisms should be put into place to document teacher practices in a systematic way and target training interventions according to specific needs. School attendance has improved over the duration of project implementation. Though there are likely various reasons for this, as indicated by the comments of the teachers and parents, the food provided (and type of food provided) has been very influential in improving school attendance as well as helping students to pay more attention during class. The evaluation results revealed that the existence of book banks has increased from the midterm evaluation to the final evaluation, while the implementation of reading buddies and reading camps are nuanced. This points to the need to reinforce the implementation of these practices in some areas and schools. The findings show that the conditions of infrastructure in regards to school hygiene are acceptable in many aspects, such as schools having separate toilets for girls and boys and having a wash basin or some sort of washing station. A substantial challenge remains in regards to schools having the materials for proper hygiene such as soap for washing hands. However, in area 1 (schools located in Nebaj, Cotzal and Chajul) it was observed that students did wash their hands before and after eating (presumably with soap provided by the teacher); in area 2 (schools located in Sacapulas, Cunén and Uspantán), data showed that the percentage of students observed washing their hands before and after meals was much lower. In the midterm evaluation, it was found that 21,147 children had received deworming and micronutrients as a result of Save the Children’s actions with public health stakeholders. Other actions promoted by the project that aim at preventive health are food and nutrition training received by parents. The final evaluation showed that there are 121,653 children that have received deworming and micronutrients as a result of Save the Children’s actions with public health stakeholders. The project promotes sustainability as a cross cutting theme and to date has carried out various actions in the communities and schools as well as with municipal governments and authorities from the Ministries of Education and Health in the department of El Quiché. However, challenges with sustainability remain. The parents and community leaders have not identified other stakeholders with whom they can coordinate so that the actions of the IDEA project can continue and they also lack management and negotiation abilities. For the mothers’ group, sustainability is less visible given that the women have limited local decision making participation and their schooling level is very low. The municipal governments hire teachers and personnel for food preparation in some schools and also contribute with transportation of food and materials to the schools. For the sustainability of the IDEA project’s interventions, it will be important that communities are proactive in seeking out this support over time. For both education and health, the approach of Save the Children working closely with government and community stakeholders has made the project highly relevant for the current context of Guatemala. There are various aspects of the project that can be scaled-up and replicated, including expanding into other geographical areas that receive less donor support. The most successful actions that could be taken to a national scale are: a) the guide for parents used for sensitizing and including them in the educational community to seek the wellbeing of their children, b) the Literacy Boost methodology that can be implemented at the municipal, departmental, and national levels by first to third grade teachers as well as fourth through sixth grade teachers, and c) the Literacy Boost diploma program to professionalize teachers in literacy development, both in Spanish and in local languages. II. Context 8 The department of El Quiché is located in Guatemala’s northwestern region. It borders México to the North, the departments of Chimaltenango and Sololá to the South, the departments of Alta Verapaz and Baja Verapaz to the East, and the departments of Totonicapán and Huehuetenango to the West. It has a territorial extension of 8,372 km2 . 1 Of the total population, 51.4% are women and 48.6% are men. At the national level, the percentage of the population that identify as indigenous is 40%, while for the department of El Quiché, it is 88.6%. Most of the population (68.8%) lives in rural areas. In 2013, the department’s median age was 15 years old, which is 2 years younger than the national average of 17 years of age. 2 The illiteracy rate for the population age 15 and over is 27.2%.3 The IDEA project (Investment for Educational Development in the Highlands, in English) is being carried out in the following municipalities in El Quiché: Chajul, Cotzal, Cunén, Nebaj, Sacapulas, and Uspantán. A. Educational Context In 2014, Guatemala had a national third-grade achievement rate of 49.9% and a national sixth-grade achievement rate of 40.4%. The second regional comparative and explanatory study (SERCE, in Spanish), published by UNESCO in 2008, placed Guatemala as one of the last Latin American and Caribbean countries in the ranking for Mathematics and Reading tests performances.4 At the beginning of the school year (January, 2018), there were a total of 168,187 children enrolled in primary school in El Quiché. Out of these, 48% (80,612) were girls and 52% (87,575) were boys. Additionally, 13% of the students live in urban areas, while the majority (87%) live in rural areas. Quiché also has 12,667 repeating students (13.3%), of which 44% are girls and 56% are boys. Out of the 1,574 establishments that were operating in 2016, 97% were public and only 3% were private. Regarding the initial student registration in the target municipalities, in 2016, Chajul had 8,587 students; Cotzal, 5,904; Cunén, 7,973; Nebaj, 12,515, Sacapulas, 8,951, and San Miguel Uspantán, 12,721 students.5 The following table shows the initial enrollment data per grade for each of the municipalities. Table 1. Initial student registration for second and third grades, according to municipality Municipality Total Second grade Third grade Total Male Female Total Male Female Chajul 8,587 1,698 851 847 1,518 750 768 Cunén 7,973 1,583 813 770 1,269 632 637 San Juan Cotzal 5,904 1,193 637 556 1,008 521 487 Nebaj 12,515 2,132 1,128 1,004 2,104 1,122 982 San Miguel Uspantán 12,721 2,525 1,344 1,181 2,311 1,189 1,122 Sacapulas 8,951 1,722 910 812 1,589 794 795 Source: Prepared by the authors with data from the 2016 Statistical Yearbook of Education of Guatemala 1 Guatemalan Ministry of Economy. 2015. Department of El Quiché. Retrieved from http://dae.mineco.gob.gt/mapainteractivo/index.php?controller=crm&action=detalles&id=14 2 National Institute of Statistics. 2014. Departmental characterization. Quiché. Retrieved from https://www.ine.gob.gt/sistema/uploads/2015/07/20/8lyJY70TduTA2BpXw8p4mpswppmvOyKX.pdf 3 Guatemalan Ministry of Economy. 2015. Department of Quiché. Retrieved from http://dae.mineco.gob.gt/mapainteractivo/index.php?controller=crm&action=detalles&id=14 4 UNESCO. 2008. SERCE study 5 Ministry of Education Planning Directorate. 2016. Statistical yearbook of education in Guatemala. Retrieved from http://estadistica.mineduc.gob.gt/anuario/2016/data/Resultado.htm?Nivel=43&Depto=00&Pob=0&Inf=1&Sector=0&Desgloce=0&Sub mit=Mostrar+cuadro 9 Regarding the department of El Quiché, net enrollment rate indicators for 2016 were 68.6% for boys and 67.7% for girls. The graduation rate for that year was slightly higher for girls (84.9%) than for boys (82.4%). Finally, the drop out was higher in boys (20.1%) than in girls (17.5%). Regarding the 2014 reading achievement rates, in El Quiché it was 28.8% for third grade and 20.16% for sixth grade. At the municipal level, Chajul had a rate of 8.7% for third grade and 3.45% for sixth grade; Cunén had 0% for both grades; Nebaj had 10.2% for third grade and 3.2% for sixth; Sacapulas 53.1% for third grade and 9.7% for sixth; Cotzal had 5.9% for third grade and 0% for sixth; and Uspantán had 6.7% for third grade and 3.2% for sixth.6 B. Project Background Save the Children Federation, Inc. (SC) is implementing a four-year program, Investment for Educational Development in the Highlands Project (Inversión para el Desarrollo Educativo en Occidente; IDEA in Spanish), which was designed to improve literacy levels of school-age children and increase use of Health and Dietary practices for nutrition and health for in primary school-age children in the department of El Quiché, Guatemala. SC uses donated commodities and funds provided by the United States Department of Agriculture (USDA), through the McGovern Dole International Food Program for Education and Child Nutrition, with support from the Ministry of Education at the national, departmental, and municipal levels. The project began in October 2014 and its activities are expected to end on January 30, 2019. By the end of the project, IDEA will have impacted 43,000 children from 283 schools across six municipalities in El Quiché. During the first year of implementation (2015) SC operated in three municipalities in the Ixil area (Nebaj, Chajul and Cotzal). Starting in the second year (2016), the project expanded to the northern area of El Quiché, covering the municipalities of Uspantán, Sacapulas and Cunén. In these six municipalities, the project distributes an average of 15 daily school meals per month for 9.5 months of the year. III. Project Description A. Organizations and Beneficiaries Save the Children (SC) in Guatemala is implementing the IDEA Project, which was designed to improve the literacy levels of school age children and to increase and enhance the use of best practices in nutrition and health in primary school-age children in the department of El Quiché. SC uses donated commodities and funds provided by USDA’s Foreign Agricultural Service (FAS) for project implementation. The Project was designed to benefit 40.000 children in 320 schools of six municipalities of El Quiché. However, the project reached 40.000 with fewer schools (277 schools). Last year USDA approved a cost extension, thus SC has reached 43,000 children and 283 schools. During the first year (starting in January, 2015), SC focused IDEA project interventions in the Ixil area, or area 1, (composed of three municipalities: Nebaj, Chajul and Cotzal). Starting in the second year (January, 2016), the project expanded to the northern area of El Quiché (area 2), covering the municipalities of Uspantán, Sacapulas and Cunén. These areas are amongst the most vulnerable in the country, as access to basic education, health, and nutrition services is extremely limited. 6 General Directorate of Educational Research and Evaluation (Dirección General de Evaluación e Investigación Educativa - DIGEDUCA). Evaluation results yearbook. 10 B. Expected Results and Indicators The project’s strategic objectives are: SO1: Improve literacy levels in school age children ● Increase children’s reading and comprehension abilities in Spanish, Ixil and K’iche’ by improving support from parents and the community for reading activities, increasing adequate access to reading materials and school supplies and supporting the efforts of Ministry of Education (MOE) staff and teachers in facilitating instruction for reading activities in the classroom ● Improve children’s school attendance and attention by addressing the issue of short-term hunger through providing school feeding (breakfasts) ● Improve children’s school attendance by implementing community sensitization activities to inform about the benefits of education and to increase children’s enrollment in school SO 2: Increase the use of food and health practices ● Increase knowledge of health and hygiene, nutrition practices, and good food preparation and storage practices, and provide tools and equipment for food preparation and storage ● Improve the physical infrastructure of schools to support positive behavior changes in health for children by building and rehabilitating school infrastructure, including kitchens, storage areas, latrines and hand washing stations The project carries out the following activities: ● Distribution of/support in: materials and school supplies, strengthening teachers’ capacity and knowledge, preparation of books and supplementary reading materials, reading camps, reading buddies, reading marathons, delivery of foods for school, enrollment campaigns, training of trainers for community activities, reading awareness workshops for members of the community, and reading fairs ● Capacity building at the local, regional, and national levels ● Training in food preparation and storage ● Distribution of deworming medications, vitamins, and minerals ● Construction/rehabilitation of kitchens and infrastructure, latrines and hand washing stations, improved stoves, and trainings for the parent-teacher associations (PTA’s) The IDEA project has longstanding collaborative relationships with the Ministry of Education (MOE), the Ministry of Health (MOH), local NGOs, and PTAs at the community level. Coordination with the MOE and MOH is as follows: ● Departmental Education District: Ensure that the interventions proposed by the IDEA Project are included within the annual operating plan and that they are implemented according to what was established. This includes the participation of quality educational services providers in administrative processes and in training activities, and guaranteeing that the required supplies 11 and materials are provided without interruption in the intervention areas (Nebaj, Chajul, Cotzal, Uspantán, Cunén and Sacapulas) during project implementation. ● Health Area Bureaus in El Quiché and the Ixil areas: Will ensure that the interventions proposed in the IDEA Project are included in annual operating plans (delivery of deworming medicines, vitamins and minerals) ● Coordination with the Municipal Districts: Support the implementation of project interventions by ensuring adequate delivery of high quality and culturally appropriate educational and health and nutrition services ● Community Organization: Ensure that the proposed interventions take place at the community level to increase nutrition, education, and health services by involving of leaders and parents IDEA seeks to improve student attendance and attention by means of better nutrition and health habits and by increasing access to school-provided food. The intervention aimed to improve reading abilities and practices (referred to as the “Literacy Boost” methodology) supports the quality of instruction inside and out of school by means of training for teachers, community reading activities, creation of materials, reading evaluations, and literacy sensitization activities at the community level. The IDEA Project seeks to improve students’ reading comprehension so that by the end of the project, at least 50% of the children in second grade can read and understand the textbooks for their academic grade￾level in Spanish. When compared to a prior evaluation of their abilities, it is expected that this will mean that at least twice as many children will be able to read and understand what they read. C. Key Project Partners and Stakeholders Save the Children implements the IDEA Project in conjunction with the MOE, which is its primary partner. This partnership is based on the Cooperation Agreement signed in April 2015, in which each organization’s individual as well as shared responsibilities are outlined. SC coordinates the planning process and implementation of the interventions with six directorates of the MOE: DICONIME (National and International Agreements for the Ministry of Education Directorate); DIGEBI (Bilingual and Intercultural Education Directorate); DIGECADE (Education Quality Management Directorate); DIPLAN (Education Planning Directorate); DIGEFOCE (Educational Community Strengthening Directorate); DIDEDUC (Departmental Education Directorate). Additionally, it coordinates the nutrition and health interventions with the MOH through the two Health Area Bureaus of Ixil and Santa Cruz del Quiché, and it is part of the Departmental Commission for Healthy Schools lead by the DIDEDUC and MOH. In order to ensure the success of the project and to promote institutionalization of the interventions, SC works closely with representatives of the educational community of the 283 schools where the IDEA Project is implemented. This includes community leaders, COCODE, and religious leaders. With regard to key, fundamental aspects for building sustainability into the activities, SC has established cooperation and partnership mechanisms with the public-private sector through: Asociación Chajulense, Bank of Central America, Colgate Palmolive, two Municipalities (Cunén and Uspantán), University of San Carlos Guatemala, and the Equine Association. 12 IV. Project Evaluation Design A. Evaluation Questions ● What enabling and conditional factors (both internal and external) have supported or limited the project’s capacity to achieve its main results? ● What recommendations stem from the evaluation results to ensure achievement of both the project's objectives and project sustainability? Relevance questions: ● To what extent is the program aligned with the policies, programs and priorities of the Guatemalan government? ● To what extent was the program aligned with the priorities of the USDA McGovern Dole Food for Education program and other multilateral development strategies or relevant donors (for example, Guatemala's 2015-2019 UNDAF)? ● What is the level of agreement between the stakeholders with regards to project strategy and the roles of the different actors? ● To what extent does the project complement the policies of the Ministry of Education (MINEDUC), municipalities, and community leaders? ● How relevant and appropriate are the strategic goals, objectives and plans in the economic, cultural, political, current and emerging context? Effectiveness questions: ● How have different activities contributed to achieving the established objectives? ● What are the administrative strengths and weaknesses of this project (including programmatic, human and financial resources)? ● To what extent was the project successfully coordinated in collaboration with key stakeholders, including the USDA, the Government of Guatemala, other NGOs (PCI and CRS), as well as other international organizations such as UNICEF, FAO, the World Food Programme and the private sector? ● To what extent was the teacher training effective in improving the knowledge and effectiveness of the teachers? Efficiency questions: ● How efficient were the strategies in terms of financial and human resources, when evaluated against project results? What could be improved? ● How efficient was the monitoring system in meeting the needs and requirements of the project? ● Were there alternatives that would achieve the same results with less funding/human resources? Impact / results questions: 13 ● When combining health and nutrition interventions, as well as Literacy Boost program, have the effects on the development of children's reading, seen in the midterm evaluation, been maintained in the last two years in both intervention areas? ● How do student literacy results in the Northern Quiché area compare with those collected in the midterm evaluation when the intervention had just begun? ● How has the culture of reading in communities changed because of Literacy Boost activities? ● What health and nutrition practices have changed after the project intervention? ● What are the changes in student attendance and attention as a result of the project? ● What main positive and/or negative effects are attributed to the intervention of the project according to the different decision makers? Sustainability questions: ● To what extent did the project plan and advance towards sustainability? ● To what extent has project continuation been planned with activities, local appropriation, and development of sustainable partnerships (with both public and private stakeholders)? ● What are the main enablers and barriers that stakeholders identify which can influence project sustainability? Successfully scaling up the project ● To what extent are project activities expected be scaled up and replicated throughout Guatemala? B. Methodology The project has a broad Performance Monitoring and Evaluation plan that complies with FAD regulations (7DFR part 1599.13). This plan includes a system for monitoring the process and the design of the project’s Baseline (BL), midterm (MTE) and final evaluations (FE). SC planned that the three evaluations would use the same research design and instruments. The project’s selected methodology to collect and measure data for the BL, MTE, and FE is described as follows: a one-on-one oral evaluation comprised of 5 literacy components distributed through 5 sub￾evaluations: knowledge of letters (vowels and consonants), word recognition (reading the most-used words), fluency and accuracy when reading (words read correctly per minute and percentage of the paragraph that is read correctly), and a set of reading comprehension questions related to fluency and accuracy when reading a paragraph. This last part was only administered to students who could read independently (reading comprehension). All instructions were given in the child's mother tongue language. The text to measure reading in the Ixil language was standardized and adjusted to the three language variations that are predominant in the evaluated communities. The student evaluation was accompanied by a survey to collect more qualitative information regarding their socioeconomic situation, educational experience, and literacy environment in the household. It included questions about knowledge and behaviors related to health, nutrition and hygiene. Both the student evaluation and the student survey were conducted in K'iche’ and Ixil, the mother tongue languages of the students in the target municipalities. It also included focus group consultations with teachers, parents, 14 parents' associations, educational authorities, and community leaders on the perception of project activities, the main results, and recommendations for improving project effectiveness. To ensure interviewer reliability, double data entry was carried out for 12% (n=109) of the student evaluations (one interviewer interviewed while the other observed, and both entered their data into the system). Most subtests have excellent reliability; only decoding of words in Spanish and reading comprehension in K'iche' are graded as having good reliability. (See Annex 1: Reliability of literacy skills subtests). C. Sample Selection During the final evaluation, data were collected on both second-grade and third-grade groups from the target areas. In the group of second-grade students, 40 schools were randomly selected from the six municipalities where the program is implemented. The following table shows the sample of the distribution: Table 2. Distribution of sample across municipalities ID Municipality # of schools 2nd-grade sample 3rd-grade sample Area 1 1 COTZAL 27 3 0 2 NEBAJ 71 10 0 3 CHAJUL 39 9 0 Area2 4 USPANTAN 52 4 6 5 SACAPULAS 41 5 8 6 CUNEN 53 9 6 TOTAL 40 20 Source: Database from Final Evaluation From each school, 14 children (7 girls and 7 boys) were evaluated. This sample was compared to the second-grade sample taken in the Mid-term Evaluation (MTE), to see whether program benefits have been maintained over the past two years. In addition, a sample of 160 third-graders randomly selected from 20 schools in the northern area of El Quiché was evaluated and compared to the third-grade sample in this area taken when the program had just begun to see the changes in the children’s literacy skills. The schools were randomly selected using SPSS software. Table 3: School sample ID Municipality # of schools 2nd-grade sample 3rd-grade sample Area 1 1 COTZAL 27 3 0 2 NEBAJ 71 10 0 3 CHAJUL 39 9 0 Area2 4 USPANTAN 52 4 6 5 SACAPULAS 41 5 8 6 CUNEN 53 9 6 TOTAL 40 20 Source: Database from Final Evaluation Table 4. Teachers’ sample 15 No. Municipality 2nd-grade teachers’ sample 3rd-grade teachers’ sample 1 USPANTAN 4 6 2 NEBAJ 10 0 3 CUNEN 9 6 4 COTZAL 3 0 5 CHAJUL 9 0 6 SACAPULAS 5 8 TOTAL 40 20 Source: Database from Final Evaluation Table 5. Sample of other key informants No. Key informant Quantity Method of consultation 1 Mothers and Fathers 59 Focus group 2 Teachers and school principals 76 Survey 3 Teachers and school principals 17 Open-ended interview 4 FOCE District and Technical Coordinators 21 Open-ended interview 5 MINEDUC Departmental Director 1 Open-ended interview 6 Municipal mayors 6 Open-ended interview 7 Department-level MINEDUC authorities 3 Open-ended interview 8 Director of the San Carlos University Center in Quiché 1 Open-ended interview 9 DIGEFOCE Director 1 Open-ended interview 10 Vice-Minister 1 Open-ended interview 11 Life stories 2 According to methodology Source: Database from Final Evaluation D. Document Analysis The ADOC team conducted a document analysis, the majority of which were internal project documents. These included: Evaluation Plan, records and database from the project’s M&E system, baseline survey report, community plans, agreements between SC and partners, notebooks and reports from field visits, and school files, among others. The following key documents were also reviewed: a) Third progress report on achievement of Millennium Development Goals for 2010 prepared by SEGEPLAN b) UNESCO Documents: SERCE Study 2008 c) National Development Plan K’atun: Our Guatemala 2032 d) Document on Educational Challenges in Guatemala, Guatemala Social Programs 2015 e) USAID country strategy for development cooperation f) Fourth National Census measuring height of first-grade students in Guatemala public schools E. Quantitative Data 16 i. Quantitative Data Collection For the quantitative data collection, a total of 809 students were evaluated in the final evaluation (FE). The following table shows the number of schools and the percentage of children evaluated by sex and municipality. Table 6 shows the percentages of students evaluated according to sex and area of intervention. In area 1, 287 students were evaluated and in area 2, 522 were evaluated. For the quantitative analysis, students were evaluated on five reading abilities and their literacy environment. The same methodology, tools and level of difficulty used in the baseline survey (BL) were applied to this final evaluation. (See Annex 2 for the Literacy Boost evaluation instrument). Table 6. Number of schools / percentage of students that participated in the study, by municipality and sex Municipality No. of schools Male Female CHAJUL 9 14.9% 15.0% COTZAL 3 7.1% 8.7% CUNEN 9 31.4% 36.3% NEBAJ 10 14.7% 10.8% SACAPULAS 5 14.9% 12.6% USPANTAN 4 17.0% 16.6% Total 40 Source: Database from Final Evaluation Table 7. Percentage of students that participated in the study, by intervention area and sex Municipality No. of schools Male Female Area 1 COTZAL 3 7.1% 8.7% NEBAJ 10 14.7% 10.8% CHAJUL 9 14.9% 15.0% Area 2 USPANTAN 4 17.0% 16.6% SACAPULAS 5 14.9% 12.6% CUNEN 9 31.4% 36.3% Total 40 Source: Database from Final Evaluation Additionally, a survey was conducted with teachers to understand their perception on the students’ attention span, while school principals were consulted about school conditions, the schedule in which they provide children with food, quantity of new, multi-grade teachers, and organizations that support the schools. In total, 96 teachers completed the survey; 44 were from area 1 and 52 from area 2. Table 8. Percentage of teachers that participated in the study, by sex Sex Area 1 Area 2 Female 43.2% 53.8% Male 56.8% 46.2% Source: Database from Final Evaluation 17 The data collectors also made observations of the different establishments that were evaluated. A total of 39 evaluations were carried out, one for each establishment. The following table shows the number of establishments evaluated in each municipality and the percentage of the total number of schools. Table 9. Number of establishments evaluated and percentage they represent Municipality No. of schools % of observation CHAJUL 8 20.5% COTZAL 4 10.3% CUNEN 10 25.6% NEBAJ 6 15.4% SACAPULAS 5 12.8% USPANTAN 6 15.4% TOTAL 39 100.0% Source: Database from Final Evaluation Data were collected through an observation tool at the schools in the sample. The aspects evaluated were the same as those evaluated in the MTE. The criteria that were established in the MTE were also applied in the FE (for both the education and health components of the project). ii. Quantitative Data Analysis The evaluation employed a quasi-experimental design in which the schools that received the intervention in Year 1 are considered to be the intervention sample, while the schools that received the intervention in Year 2 are taken as the comparison sample. During the baseline study, 35 schools from the Year 1 cohort were randomly selected to receive the intervention. Additionally, of the 133 schools that received the intervention in Year 2, 35 were randomly selected as the control sample. In total, 20 children (10 girls and 10 boys in second grade) were randomly selected. In schools where there were less than 20 children in the classroom, and all children were included in the sample. These children formed the database that was used to track any change in their reading performance in the MTE. During the MTE, the same students that were evaluated in the baseline were evaluated to find the changes in reading performance. At that time, an additional sample of second-grade students was collected from both the intervention and control samples to measure the reading skills of this grade group. In total, 815 third-grade students and 573 second-grade students were evaluated during the MTE. Information was also gathered from 137 teachers and 59 directors from a total of 62 schools in the six target municipalities. In addition, interviews were conducted with teachers to learn about their perception and experience of students' attention in class. Finally, school principals were interviewed to explore the conditions of the school, the level of organization for the provision of school meals, and the organizations that support the schools. For the final evaluation, data was collected from a group of second and third-grade students from the schools that received the interventions. The group of second-grade grade students was randomly selected from 40 schools in the six target municipalities. For each school, a sample of 14 children (7 girls and 7 boys) was selected for evaluation. This sample was compared with the sample selected in the MTE, to see if the benefits of the program have been sustained over the last two years. Additionally, a sample of 160 third￾grade students was randomly selected from 20 schools in Northern Quiché. These schools were compared to the third-grade sample taken in the MTE, when the program had begun, to observe the changes in the children’s reading skills. Summarized statistics were used to analyze student performance in reading tests. In the analysis, the variables of each group were handled as treatments, and tests were performed to demonstrate whether 18 there were any significant differences between the treatment group and the control group. If there was a difference, the differences between means were tested to understand differences between the same treatments, and to determine whether the students in the comparison intervention schools were the same in terms of backgrounds and abilities between groups. This was important in determining whether the intervention for improving reading comprehension contributed to improving the skills of the students that were evaluated. To test the differences obtained in the means of the assigned treatments, and whether they were significant and thus show the comparability of the students' progress, the data obtained in the comparison samples were submitted through "t" tests assuming an unequal variance between the two samples. The previous data were accompanied by descriptive statistics in a summary to analyze the performance of the students in each of the reading tests. In addition, regression models were used to explore the relationships between literacy abilities and characteristics of students and their environment, as defined by SC. Finally, other indicators that may be of interest and were obtained through school observation and the teacher surveys, were described through descriptive statistics. The data collected from schools, teacher surveys, and observations were entered into a digital device that was synchronized daily with the database created for the study. As further discussed in the “Limitations” section, it was extremely difficult to design a robust counterfactual scenario for several reasons that are related to the project’s implementation characteristics. For this reason, several econometrics models were adjusted to try to estimate the differences between groups. To determine and carry out the change analysis, different statistical techniques were applied; the following are highlighted: i) multilevel models; ii) models of difference in differences (exposure to treatment); and iii) regression models with different variables (baseline, intermediate and final evaluations). Multilevel models The multilevel models allowed the differences between Spanish language reading skills amongst second￾grade students in areas 1 and 2 to be determined. For this, the following variables were established: (a) percentage of correct letters in Spanish, (b) percentage of the correct “most used words” (MUW) in Spanish, (c) percentage of correct decoding in Spanish, (d) fluency in Spanish, (e) accuracy in Spanish, and (f) percentage of comprehension in Spanish. Hierarchical linear modeling (HLM) was used to statistically analyze the data structure where students (level-1) were nested within classrooms (level 2). It is important to note that this model is not intended to estimate the effect of the treatment itself, since both groups have received the treatment. What is estimated is the effect of the intensity of the treatment, with a single measurement (final measurement), because the second-grade students in area 1 experienced the intervention longer than second-grade students in area 2. Difference in differences model (exposure to treatment) One characteristic of the difference in differences model with repeated cross measures is that it is not necessary for the treatment and control groups to be assigned randomly. However, it presents the challenge that, during the final evaluation in area 2 (control), it is assumed that the intervention has not produced an effect equal to that of area 1. This represents a model of duration of exposure to the program. Then the impact estimator of the program is unbiased if the effect of the program is solely due to exposure to mentioned program. If the effect varies with the duration of exposure to the program, then the effect of the program after two exposure periods is different from the effect of the program after one exposure period. Regression models, Pre (baseline, midterm evaluation) and Post (final evaluation) 19 The final analysis included a series of models where the baseline and final evaluations for second-graders in areas 1 and 2 were compared separately; the midterm and final evaluations for second-graders in areas 1 and 2 were compared separately; and the midterm and final evaluations of third-graders in area 2 were compared. The models to be adjusted were of the pre- and post-type. Although the students were not the same, the strongest assumption was that the students in baseline (pre) and final evaluation (post) had the same characteristics. Though this assumption is very difficult to sustain, a set of covariates corresponding to the students’ characteristics (for example, socioeconomic status, sex, age, etc.) was included to control for their differences. Additionally, the models were adjusted by estimating robust standard errors for school clustering. The response variables in the adjusted models were the following: (1) percentage of correct letters in Spanish, (2) percentage of correct MUW in Spanish, (3) percentage of correct decoding in Spanish, (4) fluency in Spanish, (5) accuracy in Spanish, and (6) percentage of comprehension in Spanish. With the exception of variable (3), percentage of correct decoding, the same variables were also used in the K'iche’ and Ixil languages. (See annex 3 for a detaled description of the models and their explanations). F. Qualitative Data i. Qualitative Data Collection Interviews were carried out with second and third-grade teachers and principals of the sample schools, and a parent focus group was also carried out in each municipality. This qualitative information served to deepen the explanation of the phenomena that were found with the quantitative data, thus complementing the analysis. The focus groups were made up of twelve participants (the minimum requirement for the methodology is 8 and the maximum is 14) and included mothers, fathers, and volunteers. Through interviews with teachers, data was gathered regarding classroom practices (i.e. methodology and approaches to instruction, time spent on reading activities), perceptions about their students (attention span, health, eating habits), and participation and role of parents in the education of their children. Focus groups with parents provided information about their practices outside of school in terms of reading, health, and eating habits and practices. Interviews were also conducted with key local leaders, SC members, and MINEDUC representatives to obtain information that would answer the evaluation questions related to project sustainability, coherence of project objectives with local and national policies and objectives, and the feasibility of scaling up the project interventions. Facilitators were trained in data collection to ensure the reliability and consistency of the information that was collected. In addition, a pilot was carried out to review the guides and make adjustments in the language, include variables that had not been considered, and review the logic of the approach to the different topics. All interviews and focus groups were recorded and transcribed. Notes were taken to capture verbal information communicated by the participants as well as non-verbal information (i.e. behaviors, gestures, which people share more and which share less in the focus groups). All individuals that were interviewed, as well as focus group participants, were asked for their consent to participate and be recorded, with the understanding that all the information is confidential (as explained in more detail in the next section). 20 Figure 1. Process for qualitative data collection The team supervisor was responsible for checking that the data was collected according to guide requirements, and facilitators held meetings with participants to provide all the necessary information and encourage their participation. The following eight tools were used for the collection of qualitative data: ● Interview guide for focus groups with fathers and mothers ● Interview guide for mayors ● Interview guide for CTA and FOCE technical staff ● Interview guide for the Departmental Director of Education ● Interview guide for the Director of the University Center in Quiché ● Interview guide for teachers and directors ● Interview guide for the Director of DIGEFOCE The tools applied with the focus groups including teachers and principals were validated in the Chiul afternoon session school in Cunén. ii. Qualitative Data Analysis Data analysis began with the structuring of data using the NVivo program, which allows for coding and ordering of data by topics and subtopics and facilitating the management and analysis of qualitative information through transcripts of interviews and focus groups. The themes and categories for coding were pre-defined based on the evaluation questions, the project indicators, and the quantitative data. The rigor and quality criteria used to organize the data were: dependence, credibility, transfer and confirmation. This required that the encoding be: ● Open or first-level, comparing units to create categories and codes ● Axial or second-level, to compare categories that allow interpretation of the data, development of themes or patterns ● Selective, to codify the generation of theories, narratives, and explanations. Facilitator training on Focus Group guide, interviews, observation and field notebook. Revision of methodology, objectives, and logic of the guides. Identification of an adequate location. Organization of the groups, verification of participation criteria, and request of consent. Execution of the focus groups, interviews, observations, and field notebooks. Facilitator meeting to review material Ensure objectives and information required by the guides First transcription in the field 21 Figure 2. Analysis of qualitative data The information from interview and focus group participants was codified to carry out the qualitative data analysis. The following matrix shows the number of participants according to the group and the municipality in which the interviews or focus groups were conducted. The group of parents refers to the mothers and fathers who participated. The municipality group refers to mayors and councilmembers who participated. The group of teachers corresponds to the teachers or school principals who participated. Finally, the MINEDUC group corresponds to departmental coordinators (formerly CTA) or technical staff from the MINEDUC who participated. Table 10. Number of people from MINEDUC that participated No. Group Male Female 1 FOCE Technicians, Ixil area 3 1 2 CTA Ixil area 5 1 3 FOCE Technicians, Quiché area 4 1 4 CTA Quiché area 2 4 Total 14 7 Source: Database from Final Evaluation Support of NVivo program Analysis of Qualitative Data Sorting and categorizing written and audio material CODING Selective Axial, second￾level Open, first-level Data cleanup, recording of field personnel impressions 22 Table 11. Number of participants in focus groups NO. COMMUNITY FATHERS MOTHERS TOTAL TEACHERS / PRINCIPALS (M) TEACHERS / PRINCIPALS (F) TOTAL 1 Baxil Village, Uspantán 0 7 7 1 1 2 2 Chiul Village, Cunén 4 6 10 1 5 6 3 Trapichito Village, Sacapulas 2 13 15 1 2 3 4 Río Azul Village, Nebaj 3 7 10 2 0 2 5 Ilom Canton, Chajul 3 5 8 3 1 4 6 El Mirador Sector, Santa Abelina, Cotzal 2 7 9 0 0 0 TOTAL 14 45 59 8 9 17 Source: Database from Final Evaluation G. Limitations During the study, there were minimal observable limitations. It is important to recognize the sound coordination that the IDEA project has in the area, which facilitated access to the different informants. The lack of electricity in Cunén, where the study operations center was located, resulted in certain difficulties during staff training. Regarding data and the evaluation design over time, there were several limitations. It was extremely difficult to design a robust counterfactual scenario for several reasons that are related to the project’s implementation characteristics. First, area 2 (which was initially the control group) received the intervention, so for the final evaluation, there was no control group. Second, there was a follow-up of sample students only until the midterm evaluation, but not during the final evaluation. In regards to the regression models, the pre- and post-type were adjusted. Although the students were not the same, the strongest assumption was that the students in baseline (pre) and final evaluation (post) had the same characteristics. Though this assumption is very difficult to sustain, a set of covariates corresponding to the students’ characteristics (for example, socioeconomic status, sex, age, etc.) was included to control for their differences. Additionally, the models were adjusted by estimating robust standard errors for school clustering. These adjustments were made to overcome limitations to the degree possible. 23 V. Impact of Strategic Objective 1: Improve Literacy Levels in School Age Children Change Analysis Before presenting the different types of change analysis, it is important to highlight the type of intervention and evaluations that were carried out during the life of the project, and also consider that the project was extended for one year, which led SC to reformulate or readjust the research design. The following table illustrates the process of project implementation. Table 12. Intervention and evaluations that were carried out Area Baseline evaluation: grade and language Midterm evaluation: grade and language Final evaluation: grade and language Area 1 2nd grade, Spanish and Ixil Area 1 was selected for intervention 2nd grade, Spanish 3rd grade, Spanish and Ixil (same students as baseline) At this point, schools in area 1 had already been subjected to intervention. 2nd grade, Spanish and Ixil (different students from baseline and midterm evaluation; as a result of the donor approving a one-year extension of the project, the research design was readjusted) Area 2 2nd grade, Spanish and K’iche’ Area 2 was selected to be the control group 2nd grade, Spanish 3rd grade, Spanish and K’iche’ (same students as baseline) Schools in Area 2 were still the control group, but they were receiving the project interventions (they could no longer be considered the control group) 2nd grade, Spanish and K’iche’ (different students from baseline and midterm evaluation) 3rd grade, Spanish and K’iche’ (different students from baseline and midterm evaluation) At this point, schools in Area 2 had already received the intervention. A. Results i. Percentage of Children that can Read with Comprehension During the final evaluation, students were classified into three categories to inform reading comprehension: “non-Readers”, “beginners” and “readers”. The “non-readers” are defined as students with a fluency of less than five words per minute; “beginners” are defined as those students who do not stop reading but answer less than 80% of the questions correctly; and “readers” are those who read the paragraph and correctly 24 answer 80% or more of the questions related to reading (demonstrating understanding/comprehension). Thus, while comprehension may show gains when measured as a variable (as discussed in the following section), that does not necessarily mean that comprehension will reflect the same gains when calculated to determine the category of students by reading level (“non-readers”, “beginners”, and “readers”). Literacy in second grade students, Spanish For second-grade students’ Spanish reading comprehension levels (Figure 3), the data shows an increase in “Beginners” percentages of 8 percentage points in the final evaluation compared to the midterm. Nevertheless, there is a decrease (4 percentage points) in the “readers” category. Additionally, the final evaluation reflects a decrease (3.5 percentage points) in the group of “non-readers”. The differences between the percentages are statistically significant at 97% level of confidence for the second grade. Figure 3. Distribution of second grade students’ Spanish reading comprehension levels Source: Literacy Boost test results from midterm and final evaluation Regarding differences between girls and boys (Figure 4), in the final evaluation girls show a higher percentage of “readers” (3 percentage points) compared to boys. Additionally, boys still have a higher percentage of “non-readers”. The changes between the MTE and FE are similar for both sexes. In comparison with the midterm evaluation, the percentage of “non-readers” and “readers” decreased for boys and girls, increasing the “beginners”. The differences are statistically significant. (See Annex 4 for detailed analysis of reading category calculations and distributions). 25 Figure 4. Distribution of second grade students’ Spanish reading comprehension levels by sex Source: Literacy Boost test results from midterm and final evaluation Literacy in second grade students, K’iche’ In K'iche' (Figure 5), the reading comprehension percentage of “beginners” increased by 32 percentage points from the midterm to the final evaluation, while “non-readers” decreased from 53.2% to 33.8%, a total of 19.3 percentage points. These differences are statistically significant at 99% level of confidence. Thus, while students in the “readers” category reflects a decline from the MTE to the FE, some progress can be seen with the increase in “beginners” and decrease in “non-readers”. Figure 5. Distribution of second grade students’ K’iche’ reading comprehension levels Source: Literacy Boost test results from midterm and final evaluation Literacy in second grade students, Ixil In Ixil (Figure 6), the reading comprehension percentage of “beginners” increased 21 percentage points and the “readers” decreased by 19 percentage points. However, in this case, there was not a substantial decrease in “non-readers”. The differences are statistically significant at 99% level of confidence. Figure 6. Distribution of second grade students’ Ixil reading comprehension levels 26 Source: Literacy Boost test results from midterm and final evaluation In sum, the overall trend for second graders in all languages is an increase in “beginners” and a decrease in “readers” as well as “non-readers”. This is the most pronounced in the Ixil and K’iche languages, especially in K’iche’. Additionally, the results of the final evaluation demonstrate that the students master the last category, “readers”, in smaller percentages for second grade. This might be related to the teachers’ strike that kept students out of school for 21 days, from April to May 11. The final evaluation was done in two weeks after the strike finished. Literacy in Third grade students, Spanish Figure 7 shows an increase in the percentages of “beginners” and “readers” in the evaluation compared to the midterm evaluation. The differences between the percentages for the third grade Spanish reading comprehension are statistically significant at a confidence level of 95%. Figure 7. Distribution of third grade students’ Spanish reading comprehension levels Source: Literacy Boost test results from midterm and final evaluation In comparison to second grade levels, in third grade the percentage of “readers” shows a slight increase (4 percentage points) from the MTE to the FE. This might be because of the time they have been enrolled in the school system. 27 Finally, even though percentages in the “readers” category don’t show an increase, the decrease in “non￾readers” for both grades, both areas, and all three languages reflects a positive trend. ii. Results for Second Grade Spanish by Literacy Component Spanish Literacy, Second Grade Graph 1. Second grade Spanish reading abilities in area 2 Source: Literacy Boost test results from baseline, midterm, and final evaluation As reflected in the graphs 1 and 2 and in table 13, students’ Spanish reading comprehension abilities in both areas showed continual improvement over time (2015-2018).7 In the area 2, decoding and percentage of correct letters read also showed improvement over the time period 2015-2018. However, in the area 1, the gain in these components (as well as the others) reflected between the baseline and midterm evaluation reflect a slight decline from the midterm to the final evaluation. Graph 2. Second grade Spanish reading abilities in area 1 Source: Literacy Boost test results from baseline, midterm, and final evaluation 7 These results are from using the difference in differences model (treatment exposure). More detail of the models employed and results are provided in the annexes. A common factor across Spanish abilities is the fact that students from area 1 and area 2 showed preexisting differences. Students in area 2 (control group) showed a higher mean than students from area 1. 0 20 40 60 80 100 % Correct letters % Correct MUW % Correct decoding % Fluency % Accuracy % Comprehension Second grade Spanish reading abilities, area 2 2015 2016 2018 0 20 40 60 80 100 % Correct letters % Correct MUW % Correct decoding % Fluency % Accuracy % Comprehension Second grade Spanish reading abilities, area 1 2015 2016 2018 28 Table 13. Summary of second grade Spanish reading abilities by year and area Year Area % Correct letters in Spanish % Correct MUW in Spanish % decoding in Spanish % Fluency in Spanish % Accuracy in Spanish % Comprehension in Spanish 2015 2 71 47 39 9.71 35 19 1 56 28 23 5.03 20 11 2016 2 86 79 67 33.21 69 34 1 82 65 55 31.27 54 36 2018 2 89 78 69 28.6 68 47 1 78 54 49 20.15 48 50 Treatment Effect 2015-2018 .050 -0.024 -0.021 -2.894 -0.029 0.147* Source: Literacy Boost test results from baseline, midterm, and final evaluation Note: Treatment effect 2015-2018 was estimated by using difference and difference model (treatment exposure). Adjusted mean difference 2015-2018 was estimated by using a multiple regression analysis. * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001 Figure 8 presents a different representation of the trend of the mean difference for each Spanish ability between treated areas in the baseline and final evaluation. Figure 8. Mean Spanish abilities comparison between intervened areas across time Source: Literacy Boost test results from baseline and final evaluation 29 A common factor across Spanish abilities is the fact that students from area 1 and area 2 showed preexisting differences. Students in area 2 (control group) reflect a higher mean than students from area 1. There is a parallel trend between areas in graphics A, B, C, D, and E (correct letters, correct MUW, decoding, fluency, and accuracy). This suggests that there was no effect (“treatment effect”) for these components due to the additional year, as confirmed in the respective regression models. On the other hand, graphic F (comprehension) does not show parallel trends between areas across time, which suggests some impact due to the intervention.8 In fact, the respective models show a positive treatment effect for comprehension for both areas. It is important to note that “comprehension” when measured as a literacy component variable differs from how reading comprehension is determined to identify “Beginners”, “Readers”, and “Non-readers”. The latter, as described previously (in the section that discusses these categories), uses different measures; it takes into account “fluency” and “literal comprehension” (ability to retrieve information from the text) as well as “comprehension” (ability to comprehend the text) in order to categorize students’ reading level, rather than just the “comprehension” variable. Thus, to be considered a “Reader” in relation to the indicator “percentage of children that can read with comprehension” is a much higher standard to meet. The fact that area 2 demonstrates more overall improvement across time could be attributed to the quantity of resources and support that area 2 has received from various sources (donors, NGOs, municipal government, and private sector) as well as the more challenging conditions in the area 1. In any case, the results point to the need for further inquiry to understand why more literacy abilities showed a positive trend across time in area 2 than in area 1 in order to implement effective interventions from area 2 in area 1 and have targeted interventions to improve literacy abilities. Additionally, it is important to understand why comprehension showed a statistically significant change while other literacy abilities did not; analyzing the teacher training methodology and materials as well as their actual classroom practices could be a means of gaining such insight. iii. Results for Second Grade Spanish by Literacy Component and Associated Covariates Each component is examined below, as well as how the various covariates are associated with it. (See Annex 5 for detailed statistical results of literacy abilities and associated covariates). Letters read correctly in Spanish (graphic A) The results indicate no treatment effect across areas for the percentage of letters read correctly in Spanish. In other words, the change of percentage of letters read correctly in Spanish, in the area exposed for a longer duration, is not significantly different from the change of percentage of letter read correctly in Spanish, in the area exposed for a shorter duration (β3 = 0.05, p-value = 0.232). On average, the group of students who speak Spanish at home have a higher percentage of letters read correctly in Spanish than the group that does not speak Spanish at home (β8 = 0.06, p-value = 0.042); The difference is 6% in favor of students who speak Spanish at home. Likewise, the group of students that indicated to have read to a member of the community had a higher percentage of letters read correctly, 5% on average, than the group that did not (β13 = 0.051, p-value = 0.000). There were either no or very minimal differences by sex for this variable in both areas. (See Annex 6 for literacy ability results by sex). 8 It is difficult to assert that impact is directly attributable to an intervention based on evaluation results (except for the most rigorous designs such as randomized control experiments). However, due to the different statistical models employed and methods to control for confounding variables, it is likely that the IDEA project had an impact on reading comprehension. 30 Correct MUW in Spanish (graphic B) The results indicate no treatment effect across areas for the percentage of correct MUW in Spanish. In other words, the change of the percentage of correct MUW in Spanish, in the area exposed for a longer duration, is not significantly different from the change of the percentage of correct MUW in Spanish in the area exposed for a shorter duration (β3 = -0.024, p-value = 0.742). The group of students who attended preschool had, on average, a higher percentage of correct MUW in Spanish (5.2%), compared with the group of students who did not study preschool (β6 = 0.052, p-value = 0.046). The group of students who speak Spanish at home had a higher percentage of correct MUW in Spanish than the group that does not speak Spanish at home. The group of students that indicated to have read to a member of the community had a higher percentage of correct MUW in Spanish (10.1% on average) than the group that did not. Finally, the SES had a positive impact of 0.7% on the correct MUW in Spanish percentage, on average, for each level that the SES increases (β15 = 0.007, p-value = 0.018). There were either no or very minimal differences by sex for this variable in both areas. Decoding in Spanish (graphic C) The results indicate no treatment effect across areas for the percentage of correct decoding in Spanish. In other words, the change of the percentage of correct decoding in Spanish, in the area exposed for a longer duration, is not significantly different from the change of the percentage of correct decoding in Spanish in the area exposed for a shorter duration (β3 = -0.021, p-value = 0.783). Additionally, students who were repeating second grade had a lower percentage of correct decoding in Spanish than students who were not repeating that grade (β7 = -0.116, p-value = 000), with the average difference between both groups being 11.6%. Likewise, the group of students who speak Spanish at home had a greater percentage of correct decoding in Spanish than the group that does not speak Spanish at home (β8 = 0.119, p-value = 0.007). On average, the difference between both groups is 11.9%, in favor of students who speak Spanish at home. In addition, the group of students who indicated to have read to a member of the community had a greater percentage of correct decoding in Spanish (9.6%) than the group that did not (β13 = 0.096, p-value = 0.000). Finally, the SES had a positive impact of 0.7% on the correct Spanish decoding, on average, for each level that the SES increases (β15 = 0.007, p-value = 0.03). There were either no or very minimal differences by sex for this variable in both areas. Fluency in Spanish (graphic D) The results indicate no treatment effect across areas for the percentage of fluency in Spanish. In other words, the change of the percentage of fluency in Spanish, in the area exposed for a longer duration (area 1), is not significantly different from the change of the percentage of fluency in the area exposed for a shorter duration (area 2), β3 = -2.894, p-value = 0.533). The group of students that were repeating second grade had a lower score of fluency in Spanish than the group that were not repeating second grade (β7 = -3.317, p-value = 0.002). The mean difference was 3.317 points. The group of students who speak Spanish at home had a higher score of fluency than the group that do not speak Spanish at home (β8 = 7.867, p-value = 0.000). The average difference between both groups was 7.867 points in favor of students who speak Spanish at home, while the group of students who indicated to have read to a member of the community had a higher score of fluency (3,731 points on average) than the group that did not (β13 = 3.731, p-value = 0.001). Finally, the SES had a positive impact of 0.395 points in fluency, on average, for each level that the SES increases (β15 = 0.395, p-valor = 0.004). There were either no or very minimal differences by sex for this variable in both areas. 31 Accuracy in Spanish (graphic E) The model to determine the students’ abilities, evaluating the percentage of accuracy as a dependent variable, showed no treatment effect across areas. In other words, the change of the percentage of accuracy in Spanish, in the area exposed for a longer duration (area 1), is not significantly different from the change of the percentage of accuracy in Spanish in the area exposed for a shorter duration (area 2) (β3 = -0.029, p￾valor = 0.740). Students who were repeating second grade presented lower accuracy compared to students who were not repeating second grade (β7 = -0.092, p-valor = 001); the average difference between these two groups was 9.2%. Students who speak Spanish at home had a greater percentage of accuracy than students who do not speak Spanish at home (β8 = 0.124, p-valor = 0.003). The difference was 12.4% in favor of students who speak Spanish at home. The group of students who have read to a member of the community also had a greater percentage of accuracy, 10.1% on average, than the group that did not (β13 = 0.101, p-valor = 0.000). Finally, the SES had, on average, a positive impact of 0.9% on the percentage of accuracy, for each level that the SES increases (β15 = 0.009, p-valor = 0.013). There were either no or very minimal differences by sex for this variable in both areas. Reading with comprehension in Spanish (graphic F) The model to determine the abilities of the students, evaluated as a dependent variable, reading with comprehension in Spanish, showed a treatment effect of 14.7%. In other words, the percentage change of students from schools exposed to the treatment for longer (area 1) was higher (14.7% on average) than the percentage change of students from schools exposed to the treatment for less (β3 = 0.147, p-value = 0.018). The female group had an overall lower percentage than the male group across time in both areas, as reflected in graph 3; the difference was 3.5%, (β4 = 0.035, p-value = 0.019). In area 1, boys improved from the baseline to the final evaluation 44% while girls improved 36%. Similarly, the group of students who speak Spanish at home had a higher percentage of comprehension in Spanish than the group that does not speak Spanish at home (β8 = 0.262, p-value = 0.000). The difference between the two groups was 26.2%, in favor of students who speak Spanish at home. Likewise, the group of students who indicated to have read to a member of the community had a higher percentage of correct comprehension in Spanish (6.5%) when compared to the group that did not (β13 = 0.065, p-value = 0.000). The group of students who indicated they had breakfast in the morning presented a lower percentage of comprehension in Spanish (6.7% on average).9 Finally, the SES had a positive impact of 1.1% on the percentage of correct comprehension in Spanish for each level that the SES increases (β15 = 0.011, p-value = 0.000). 9 There could be various reasons for this. It may not have been verified whether or not student had breakfast since it is self-reported. Students may have responded that they did not have breakfast expecting to receive food after the survey. In addition, we do not know if students understood the question properly. If the question is measuring students receiving breakfast at school and students think about breakfast at home, the response may not be valid (since the follow up question about where she/he ate was not calculated). The other way around may have occurred. The question was designed to measure whether students had breakfast at home, but students may have thought about having breakfast at school. 32 Graph 3. Second grade Spanish reading comprehension in by area and sex Source: Literacy Boost test results from baseline, midterm, and final evaluation In sum, the Diff & Diff models show evidence that suggests the program intervention had a positive impact on second grade Spanish reading comprehension. Additionally, reading comprehension is the only literacy ability that reflected a statistically significant difference between boys and girls, with boys having a higher level in both areas in each year (except 2016, area 1). In addition, the models suggest that all of the literacy abilities measured are associated with some covariates. Speaking Spanish at home and having read to someone in the community is positively associated with each of them. Having attended preschool is positively associated with MUW. Finally, having repeated second grade is negatively associated with decoding, fluency, and accuracy. Implications are that the intervention to improve teachers’ knowledge and skills and the materials provided for them should be examined to see if more emphasis should be placed on the development of each of the literacy components that did not show a treatment effect over time. Additionally, further inquiry should be done on the difference in comprehension between boys and girls. Finally, because reading to someone in the community was positively associated with all components, this is a practice that should be documented as a best practice and reinforced for Spanish and the other languages (K’iche’ and Ixil). iv. Results for Second Grade K’iche’ and Ixil Reading Abilities Results in K’iche’, second grade, area 2 As illustrated in Graph 4 and Table 14, second grade K’iche’ literacy abilities in the area 2, a positive trend across time was shown for “correct MUV”, “fluency”, “accuracy” and “comprehension”; each of them showed a statistically significant improvement. Additionally, “% of correct letters” increased overall (from baseline to the final evaluation), though it was not statistically significant. Additionally, the models indicate a positive difference between groups of children for the covariates “read to someone in the community” and “went to reading camp”. 21 17 11 1 38 3 35 37 49 45 55 46 0 10 20 30 40 50 60 Male Female Male Female Male Female Male Female Male Female Male Female Area 2 Area 1 Area 2 Area 1 Area 2 Area 1 2015 2016 2018 Percentage Year Comprehension in Spanish, by area and sex 33 Graph 4. Second grade K’iche’ reading abilities in area 2 Source: Literacy Boost test results from baseline, midterm, and final evaluation Table 14. Summary of second grade K’iche’ reading abilities by year, area 2 Source: Literacy Boost test results from baseline, midterm, and final evaluation Results in Ixil, second grade, area 1 In second grade, all Ixil abilities show an increasing trend from the baseline to the final evaluation (see 5). The change for all of the literacy components in Ixil abilities across time is notable In fact, all of the abilities had a statistically significant increase from the baseline to the final evaluation. Results show that reading skills were higher for the group of children in the FE in all the models. The differences range between 24% (“correct letters”) and 45% (“comprehension”). Likewise, children, on average, demonstrated a higher fluency score (13%) in the final evaluation than the children from the baseline, even though there was a decline from the midterm to the final evaluation. The models also indicate a positive difference between groups of children for the covariate “read to someone in the community and a negative association between groups for “repeating second grade”. Year Area % Correct letters in K'iche' % Correct MUW in K'iche' % Fluency in K'iche' % Accuracy in K'iche' % Comprehension in K'iche' 2015 2 68 24 4.09 17 02 2016 2 63 52 17.07 42 11 2018 2 70 62 17.07 47 18 Adjusted Mean difference 2015-2018 0.008 0.311*** 12.856*** 0.237** 0.131*** 0 20 40 60 80 % Correct letters % Correct MUW % Fluency % Accuracy % Comprehension Second grade K'iche' reading abilities, area 2 2015 2016 2018 34 Graph 5. Second grade Ixil reading abilities in area 1 Source: Literacy Boost test results from baseline, midterm, and final evaluation Table 15. Summary of second grade Ixil reading abilities by year, area 1 Source: Literacy Boost test results from baseline, midterm, and final evaluation In sum, second grade K’iche’ and Ixil literacy abilities showed a notable improvement from the baseline to the final evaluation in all components. The increase in the percentages of each literacy ability for both languages (except for “% of correct letters” in K’iche’) was statistically significant. Additionally, as with second grade Spanish reading abilities, area 1 reflects a decrease in some of the literacy components between the midterm to the final evaluation, while in area 2 almost all of the components show a continual improvement over time (2015-2018). Furthermore, as in the case of second grade Spanish literacy abilities, “having read to someone in the community” also reflected a positive association for K’iche’ and Ixil literacy abilities, while “repeating second grade” was negatively associated for literacy abilities in both languages, as it was for Spanish literacy abilities. Implications are that further inquiry should be done into why there was a decline in literacy abilities for in both languages (Ixil and Spanish) from the midterm to the final evaluation in area 1. At the same time, the overall (statistically significant) improvement from the baseline to the final evaluation for reading comprehension in second grade reading abilities in all three languages and both areas is notable and should be highlighted as a major achievement, as should the overall improvement from the baseline to the final evaluation in K’iche’ and Ixil literacy abilities. Year Area % Correct letters in Ixil % Correct MUW in Ixil % Fluency in Ixil % Accuracy in Ixil % Comprehension in Ixil 2015 1 35 9 1.90 9 1 2016 1 54 42 22.94 39 30 2018 1 59 39 14.88 39 46 Adjusted Mean difference 2015-2018 0.238*** 0.293*** 14.010*** 0.281*** 0.437*** 0 10 20 30 40 50 60 70 % Correct letter % Correct MUW % Fluency % Accuracy % Comprehension Second grade Ixil reading abilities, area 1 2015 2016 2018 35 v. Results for Third Grade Spanish and K’iche’ Reading Abilities Results in Spanish, third grade, area 2 All third grade students’ Spanish reading skills in area 2 increased from the midterm (2016) to the final evaluation (2018), as reflected in Graph 6 and Table 16. However, only the ability of “correct letters” was statistically significant, where the children of the final evaluation showed a 5.1% higher average in terms of reading abilities than the children of the MTE. The covariates “attended preschool”, “read to someone in the community”, and “someone helps you study” showed a positive association with several literacy variables. However, repeating first grade and repeating second grade were negatively associated with several literacy components, as was “works outside the home”. Graph 6. Third grade Spanish reading abilities in area 2 Source: Literacy Boost test results from midterm and final evaluation Table 16. Summary of third grade Spanish reading abilities by year, area 2 Year Area % Correct letters % Correct MUW % decoding % Fluency % Accuracy % Comprehension 2016 2 86 79 67 33.21 69 34 2018 2 92 82 71 39.83 75 48 Adjusted Mean difference 2016-2018 0.051** -0.002 -0.003 2.333 0.010 0.068 Source: Literacy Boost test results from midterm and final evaluation Regarding differences between girls and boys, boys showed a 12% improvement in comprehension from the baseline to the final evaluation and girls an improvement of 16%. Girls showed more improvement than boys but still across years, boys performed better than girls (as was seen in the results for second grade Spanish reading comprehension). Thus, girls did not surpass boys in comprehension results but showed a tendency of catching up with them in terms of average comprehension scores. Results in K’iche’, third grade, area 2 0 20 40 60 80 100 % Correct Letters % Correct MUW % Correct decoding % Fluency % Accuracy % Comprehension Third grade Spanish reading abilities, area 2 2016 2018 36 In area 2, all third grade students’ K’iche’ reading skills increased from the midterm (2016) to the final evaluation (2018), as reflected in Graph 2 and Table 17. The variables that demonstrated a statistically significant increase are “percentage of correct letters” (a 5.9% increase) and “comprehension” (a 9.5% increase). Positively associated covariates for third grade K’iche’ abilities were “having read to someone in the community”, “someone helps you study”, and “had breakfast in the morning, while negatively associated covariates were “repeating second grade” and “works outside the home”. Graph 7. Third grade K’iche’ reading abilities in area 2 Source: Literacy Boost test results from midterm and final evaluation Table 17. Summary of third grade K’iche’ reading abilities by year, area 2 Year Area % Correct letters % Correct MUW % Fluency % Accuracy % Comprehension 2016 2 63 52 16.6 41 11 2018 2 70 60 22.32 52 20 Adjusted Mean difference 2016-2018 0.059* 0.057 4.442 0.078 0.095** Source: Literacy Boost test results from midterm and final evaluation In sum, all third grade literacy abilities in Spanish and K’iche’ reflect improvement from the midterm to the final evaluation. Additionally, the positive change was statistically significant for third grade Spanish and K’iche’ “% of correct letters” as well as for reading comprehension for third grade K’iche’ literacy. The evidence also showed that some covariates are either positively or negatively or positively associated with third grade Spanish and K’iche’ literacy abilities. For example, positively associated covariates for both languages in third grade literacy abilities were “having read to someone in the community” and “someone helps you study”, while “repeating second grade”, and “works outside the home” were negatively associated with literacy abilities in both languages. 0 20 40 60 80 % Correct letters % Correct MUW % Fluency % Accuracy % Comprehension Third grade K'iche reading abilities, area 2 2016 2018 37 VI. Impact of Intermediate Results of Strategic Objective 1: Support to Improve Reading B. Quality of Teaching Improved Quality of Literacy Instruction; Increased skills and knowledge of teachers and school administrators During 2016 and 2017, a total of 2,359 teachers and 189 directors have been trained or certified. The following table shows the data, means of verification and the source. Table 18. Number of teachers/educators/assistants from schools trained or certified as a result of USDA assistance Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of Administrators of education centers and officials trained or certified as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, Ixil and K’iche’ areas SCI Database 72 117 IN FAIS 3B Number of teachers / educators / assistants from schools trained or certified as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI Database 842 1517 SEMESTER REPORTS SENT TO DONOR Source: Save the Children records Percentage of teachers trained, according to IDEA project records, surveys of 2nd and 3rd-grade teachers on trainings received from IDEA project and length of participation in the project The great majority of teachers and principals in both areas participated in the trainings, and, on average, this participation was for 30 months (area 2) or 31 months (area 1). Training in area 2 began 12 months after training started in area 1. According to teachers and parents, the training has helped to improve teaching methodologies and students’ engagement with learning. Many teachers commented specifically about the teaching practices shifting away from traditional approaches to more modern and innovative approaches: “We have been in constant training; this makes the difference between a model of traditional education to a more innovative one. We have been trained in letter knowledge, vocabulary, they taught us the technique of the phonemic jump [game]; this has facilitated us knowing how to teach. These also have encouraged the participation of children, they have become less shy, because before it was about just sitting down and writing, writing and more writing, everything was very traditional, while today they learn by playing and doing, thanks to all of the techniques that the IDEA program has taught us.” (Hemos estado en constante capacitación; esto hace la diferencia entre un modelo de enseñanza tradicional a uno más innovador. Hemos sido capacitados en conocimiento de letras, vocabulario, nos enseñaron la técnica de salto fonémico; esto nos ha venido a facilitar sobre cómo enseñar. Estas además fomentan la participación de los niños, ellos han perdido mucho la timidez, porque antes era de llegar a sentarse y a escribir, escribir y más escribir, todo era muy 38 tradicional, mientras que hoy ellos aprenden jugando y haciendo gracias a todas las técnicas que el programa IDEA nos ha enseñado.” (Second grade teacher). “The way of working has changed, from the traditional to the modern. Before the children memorized the contents for each course, now the teaching and learning is based on practical techniques and they learn through playing. Apart from this, the children have been timely, responsible, participative and collaborative.” (“Se ha cambiado la modalidad de trabajo, de lo tradicional a lo moderno. Anteriormente los niños memorizaban los contenidos por cada curso, ahora la enseñanza￾aprendizaje es basado a través de técnicas prácticas y jugando se aprende. A parte ello los niños han sido puntual, responsable, participan y colaboradores”.) (Teacher) “…the big difference is that before, it was a very passive education, boring to a certain point, the children were recipients and not actors, whereas today they are actors and not recipients only. That is the big difference, today education is more dynamic and together the teacher and child construct knowledge from their experiences and realities of the context. (“…la gran diferencia es que antes era una educación muy pasiva, aburrida hasta cierto punto, los niños eran receptores y no actores, mientras que hoy son actores y no receptores únicamente. Esa es la gran diferencia, hoy la educación es más dinámica y juntos el docente con el niño construyen conocimientos a partir de sus experiencias y realidades del contexto. (Teacher) Another teacher expressed that while she had seen that some children had improved reading and showed interest in improving literacy abilities, that was not the case with all of the students, but it is the teachers’ role to “support and orient them” (“apoyarlos y orientarlos”). One teacher manifested that the teachers are putting in practice the techniques they learned through the IDEA project, but that one aspect that could be improved is “the seating arrangement of the students in the classroom such as having it be a ‘U’ shape, rows, or others” (“la ubicación de los alumnos en el salón de clase así como en forma de ‘U’, filas u otros”). According to one teacher, the most impactful technique was the read-a-thon (“maratón de lectura”). Parents also commented that they have seen changes in their students regarding their engagement with learning: “The children participate in the readings, because they attend the reading camps and pay more attention in their classes, and are sick less and are more cheerful”. (mother, Nebaj). “Los niños participan en las lecturas, ya que asisten en los campamentos de lectura y ponen más atención en sus clases, y se enferman menos y están más alegres”. (mother, Nebaj). “Here in this school the changes we have seen…in the improvement of their writing and reading, mainly in understanding, because before the program the children had difficulty writing and reading also the children were not performing well in their learning.” “Aquí en esta escuela los cambios que hemos visto… en el mejoramiento de su escritura y lectura principalmente en la comprensión, porque antes del programa los niños tenían dificultades para escribir y leer también los niños no rendían en su aprendizaje.” (Mother, Trapichitos) Another teacher expressed that the teaching techniques were appropriate for the students’ needs and that she has seen their literacy skills improve: “In the year 2016, these trainings came to reinforce the work of the teachers at the CEF level, the methodologies and the techniques that we were taught are appropriate to the reality and to the needs of the children as they have shown improvement in each of the aspects and more in reading and writing. All I learned has been useful, it is up to each teacher to apply it or not, and we learn more techniques for strengthening reading.” (“En el año 2016, estas capacitaciones vinieron a reforzar el trabajo del docente en el nivel CEF, las metodologías y las técnicas que nos enseñaron 39 están adecuadas a la realidad y a la necesidad de los niños ya que ellos han demostrado mejora en cada uno de los aspectos y más en lecto-escritura. Todo lo aprendido nos ha sido útil, queda en cada docente si lo aplica o no, y que nos capaciten en más técnicas para el fortalecimiento de la lectura.”) However, not all of the teachers have changed their teaching practices. As expressed by one teacher: “(w)hat we need to improve first is our consciousness, because many of the teachers are trained and when they arrive in the classroom they are still doing the same thing, they can, but do not want to”. “(l)o que debemos de ir mejorando primeramente es nuestra conciencia, pues muchos docentes se capacitan y al llegar al aula siguen haciendo lo mismo, ellos pueden, pero no quieren.” This comment by a school principal may have been pointing to the same issue: that not all teachers put in practice what they are learning: “In my point of view, it has been the same. The only difference is that the teachers have been trained in new techniques and strategies to implement in their classroom”. ” En mi punto de vista ha sido igual. La única diferencia que los maestros han sido capacitado para nuevas técnicas o estratégicas para la implementación en sus clases.” While this comment by the principal may underscore that some teachers have not changed their practices, it is also possible that some principals are not routinely and systematically observing the teachers during instructional time. It is important that the project consider ways to encourage principals to document teaching practices on a regular basis both as a means to provide them support as well as for SC to have information on teachers and their change (or lack of) teaching practices. In sum, teachers expressed that they have learned new techniques and strategies that they are putting into practice, in some cases giving specific examples. Parents also commented on having observed their children learning more learning and being more engaged, particularly with reading and writing. It is important, however, to have a follow up mechanism in place for helping ensure that all teachers are taking advantage of the trainings and shifting their paradigms and practices about teaching and learning to be more consistent with innovative approaches offered through the IDEA project. C. Access to Learning Materials Percentage of practices and resources for improving reading in children evaluated Positive answers to the question “Is there a Book Bank in the school?” were higher in the final evaluation (FE) than in the midterm (MTE), as reflected in Graph 8. The difference between groups of the MTE and FE in terms of principals mentioning that they had books revealed that there is a higher percentage in the FE group compared with the MTE group. These differences are statistically significant (at a 95% confidence level). Thus, it is inferred that there was a positive change between the MTE and FE. However, the implementation of “reading buddies” and “reading camps” appears to be lower in the FE than in the MTE in both areas The difference between groups of the MTE and FE in terms of principals mentioning that they were implementing “reading buddies” and “reading camps” revealed that there is a higher percentage in the MTE group compared with the FE group for both. However, these differences are not significant (at a 95% confidence level). Thus, it is inferred that there was no change between assessments. 40 Graph 8. Existence of Book Banks and implementation of Reading Buddies and Reading Camps Source: Principal questionnaire In sum, the results in terms of changes in the existence of book banks, implementation of reading buddies, and implementation of reading camps are nuanced. It may point to the need to reinforce the implementation of reading buddies and reading camps in some areas and schools. Table 19. Number of textbooks and other educational and learning materials provided as a result of USDA assistance Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of textbooks and other educational and learning materials provided as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI Database 93,710 117,710 IN FAIS 10B Source: Save the Children database In total, 211,420 textbooks and other educational materials have been delivered. Greater compliance was observed in area 2 for the following criteria: tools to develop reading skills are in use and in sight in the classroom; the use of the lettered classroom method is visible (i.e. clothesline of letters, photographs, children’s works, posters, etc.) and it was observed that the teacher handles the tools in the mother tongue (L1) and Spanish (L2) languages. 41 Table 20. Observation of the use of methodology and tools in the classroom Type Area 1 Area 2 Recount % of column number Recount % of column number Tools to develop reading skills are in use and in sight in the classroom Does not meet criteria 1 5.6% 0 0.0% Meets criteria 17 94.4% 21 100.0% The use of the lettered classroom method is visible: clothesline of letters, photographs, children’s works, posters Does not meet criteria 2 11.1% 0 0.0% Meets criteria 16 88.9% 21 100.0% Se observa que el docente maneja las herramientas en L1 y L2 Does not meet criteria 0 0.0% 0 0.0% Meets criteria 18 100.0% 21 100.0% Source: Classroom observation guide The reason that teachers in area 1 show lower compliance than in area 2 is unknown, however the focus group data may point to potential reasons. As mentioned previously, not all teachers put into practice the techniques they have learned through the training provided by Save the Children. Additionally, several teachers mentioned that they ran out of materials: “(e)very one of the materials received from the project, from the teaching bag to the school supplies for the children, everything has been useful. In fact, sometimes it is not enough and we need to see how to buy more, so that we can continue with our activities. It has been constantly recommended that our classroom is full of teaching material to facilitate the learning of the child.” (“(c)ada uno de los materiales recibidos de parte del proyecto, desde la valija didáctica hasta los útiles escolares de los niños, todo ha sido útil. Es mas, a veces no nos alcanza y tenemos que ver cómo comprar para que podamos continuar con nuestras actividades. Nos han recomendado siempre que nuestra aula este lleno de material didáctico para facilitar el aprendizaje del niño.”) “When the program started it was lovely because we received a lot of materials but unfortunately this year they gave us very little, perhaps because it ends this year”. (“Cuando el programa empezó fue muy bonito porque recibimos materiales en abundancia pero lastimosamente que este año ya fue muy poco lo que se nos dio, quizás es porque este año termina”.) Another teacher mentioned that some of the techniques using the materials were difficult, such as the “serpiente emplumada”, because not all of the children know how to read. Even though not all the teachers in area 1 reflected ample use of the materials in their classrooms, the majority did and they spoke very positively of having these materials to help them teach more effectively. Teachers expressed that the materials were very helpful for a number of reasons: 1) they complement the teaching strategies, 2) they are easy to use and motivate the children, 3) they are context appropriate thus engaging for the students, and 4) they help facilitate learning in the L1 and the L2. Teachers expressed that the materials helped them to implement the techniques and strategies that they had learned in the trainings, and that they were easy to use: 42 “The materials are very important because through this it helps the teacher to teach and be able to see the level of the students” (“Muy importante los materiales porque a través de eso es auxilia el educando para impartir su labor docencia y poder ver el nivel de los alumnos”). “(t)he materials that Save the Children gave us are provided and it is a technique that helps us to form syllables, words, for example: the word table, it serves to teach through play and that way the students learn more quickly, the materials are easy because they are concrete materials and the children are able to handle them without any difficulty”. “los materiales que nos han dado en Save the Children, son dados y es una técnica que nos ayuda a formar sílabas, palabras, por ejemplo: la palabra mesa, nos sirve para enseñar jugando y así aprenden más rápido los niños… los materiales son fáciles porque son materiales concretos y los niños los manejan bien no tienen ninguna dificultad”. “All the didactic material that the IDEA program has provided us with is very useful, easy to use, and I do not believe that there is one that is difficult, that's what we were trained. The usefulness is reflected in the way in which they learn, before right when they heard the bell they would leave, their only interest was the snack; now they are very active, because they are no longer thinking about food only, but in the impact of their work”. (” Todo el material didáctico que el programa IDEA nos ha proporcionado es de mucha utilidad, fácil de utilizar y no considero que haya alguno que sea difícil, para eso fuimos capacitados. La utilidad se puede reflejar en el modo en que ellos aprenden, antes ellos solo al escuchar el timbre se salían, su único interés era la refacción; ahora son muy activos, porque ya no están pensando en los alimentos solamente, si no en el impacto de las tareas.”) “None of the material has been difficult, it hasn’t been difficult for us to use because Save the Children has trained us on how to use it with the children.” (“Ningún material nos ha sido difícil, no se nos ha dificultado su uso, porque Save the Children nos han capacitado sobre cómo usarlos con los niños.”) The fact that the materials were provided in the students in the L1 as well as Spanish was also highlighted in the comments of the teachers: “The materials are appropriate. Save the Children has given us the techniques to promote reading with children, they are easy to apply, there is material in L1 and L2, then this makes the children more motivated to read”. (“Los materiales son adecuados. Save the Children nos ha dado las técnicas para promover la lectura con los niños, son de fácil aplicación, hay material en L1 y L2, entonces esto hace que los niños se motiven más por la lectura.” “The material is suitable for children; all photos and drawings are suitable for children because they are contextualized and the children are familiar with the drawings, they are written in their own language and this is the important thing.” (“El material es adecuado para los niños; todas las fotos y dibujos son adecuados para los niños porque son contextualizados y los niños están familiarizados con los dibujos, están escritos en su propio idioma materno y esto es lo importante.”) The material provided through the IDEA project was also beneficial for developing literacy in Spanish. A school principal explained that it was very difficult for the children to learn Spanish, but with the L2 texts, it has improved. He pointed out that, of 305 students, 94% graduated from their grade and that the 6% that did not graduate includes children who dropped out due to family problems or learning problems/dissabilities. 43 D. School Attendance Improved school attendance: Increased student enrollment; number of students (male/female) attending classrooms regularly (80%); schools receiving USDA support While in the BL, school absenteeism was attributed solely to illnesses (diarrhea, respiratory infections, and conjunctivitis), in the MTE and FE. respondents added other causes such as agricultural tasks in times of sowing or harvest, and disability. In area 2, mothers and fathers attributed school drop out and non￾attendance to violence in the schools, specifically bullying. However, as reflected in the data, attendance appears to have improved in both areas when comparing the MTE with the FE. As expressed by the mayor in Uspantán: “In education, we see the results from the past year, there has been an increase as shown in the statistical data that many children go to classes, however, in some communities, we have had attrition due to migration that has drawn families to the capital, Mexico and the United States. But in general there has been an increase in the student population. This year, in comparison with the year 2017, we had an increase of 1.2% of children enrolled”. (“En educación nosotros vemos los resultados del año pasado, ha habido un incremento en cuanto a los datos estadísticos de que muchos niños van a recibir las clases, sin embargo, en algunas comunidades hemos tenido deserción por migración que han tenido las familias a la capital, México y Estados Unidos. Pero por lo general si ha habido un incremento de la población estudiantil. Este año en comparación con el año 2017 tuvimos un incremento del 1.2% de niños matriculados.”) Reasons given for this increased attendance are both due to the food provided as well as the new games and activities in the schools. As one father (in Nebal) expressed: “What I see is that now the malnutrition has diminished the in some children, the children attend school more and learn better, they say that today there are many games in the school; in addition, children no longer are hungry in their classrooms”. (“Lo que veo es que ahora se ha disminuido la desnutrición en algunos niños, los niños asisten más a la escuela y aprenden mejor, ellos dicen que hoy hay muchos juegos en la escuela; además los niños ya no tienen hambre en sus aulas”.) A teacher also echoed the importance of access to meals as a reason for improved school attendance: “Thanks to food that the program has given this food because it has helped the economy of many families and the nicest thing is that the kids like it and eat it. Also it helps that the menus vary from day to day so that they don’t get bored of eating the same thing; also we can confirm that the attendance of the children has been constant and their participation also in doing their homework and there has not been any problems or inconveniences”. (“Gracias a que el programa ha dado este alimento ya que ha venido a ayudar la economía de muchas familias y lo más bonito es que a los niños les gusta y lo comen. También ayuda los menús que se varían día tras día para que ellos no se aburren de comer lo mismo; también cabe confirmar que la asistencia de los niños ha sido constante y su participación también en el cumplimiento de hacer sus tareas y no ha habido algún problema inconveniente alguno.”) E. Student Attention in the Classroom Percentage of students paying attention in the classroom according to teacher perception As demonstrated in the graph, the means of both areas are greater in the FE than the means in the MTE in terms of children that pay attention in class. 44 Graph 9. Percentage of children that pay attention or not; teacher perception Source: Teacher questionnaire % of students paying attention according to their own perception However, with self-reported data from students, the results differ. 287 students from area 1 (Nebaj, Chajul and Cotzal) and 522 students from area 2 (Uspantán, Sacapulas and Cunén) participated in the FE. The means of both areas are lower than the results of the MTE in terms of the percentage of children who say that they pay attention all the time, and there was an increase in the average percentage of children who report paying attention most of the time but sometimes getting distracted. Graph 10. Percentage of children that pay attention or not; student perception Source: Student questionnaire Because the data is based on perception in both cases, it is difficult to determine if there is an overall increase or decrease in the percentage of children who pay attention and the degree to which this has changed over the time of the intervention. However, qualitative data complements the descriptive data, as presented in the following comments. 0 10 20 30 40 50 60 70 80 90 100 I pay atention all the time I pay attention most of the times, but sometimes I get distracted I am easily distracted I am always distracted MTE Area 1 MTE Area 2 FE Area 1 FE Area 2 0 10 20 30 40 50 60 70 80 MTE Area 1 MTE Area 2 FE Area 1 FE Area 2 Percentage of children that pay attention Percentage of children that try to pay attention Percentage of children that do not pay attention 45 Two different teachers stated that the food has not only helped improve attendance put also contributed to students paying more attention in class: No child has complained about or that they don't like the food, Thanks to God, all of them eat and pay more attention in their classes, since the way the food is prepared varies, they like it. For this year the school enrollment has improved, and I believe that is due to the IDEA program for all the help that they have given to the school”. (“Ningún niño se ha quejado o no le gusta la comida Gracias a Dios todos comen y ´ponen más atención en sus clases, como se varía la forma de preparación de los alimentos entonces ellos les gusta. En este año ha mejorado la matrícula escolar y considero que es por el programa IDEA por toda la ayuda que ha brindado a la escuela.”) “The food received has covered the food needs of the children; many do not have the opportunity in their home to eat a well-nourished meal like they are given here in the school. It has improved the health, growth and the size of the majority of the children; they are now more attentive and some attend the school because they know that here they are given food, that has motivated them not to miss classes.” (“Los alimentos recibidos ha venido a cubrir las necesidades alimentarias de los niños; muchos no tiene la oportunidad en sus casa de comer un platillo bien nutrido como se les da aquí en la escuela. Ha venido a mejorar la salud el crecimiento y la talla de la mayoría de los niños ahora se ven más atentos y algunos que asisten a la escuela porque saben que aquí se les da comida, eso ha motivado a que ellos no falten a clases.”) In sum, school attendance has improved over the duration of project implementation. Though there are likely various reasons for this, as indicated by the comments of the teachers and parents, the food provided (and type of food provided) has been very influential in improving school attendance as well as helping students to pay more attention during class. F. School Feeding Number of school-age children that receive daily school meals (breakfast, snack, lunch) as a result of USDA assistance Table 21. School feeding indicators Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of take-home rations provided as a result of USDA support Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI database School registry 66,522 66,522 FAIS 11T Number of people receiving take-home rations as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI database School registry 66,522 0* FAIS SUM OF 11H and 11I Number of school meals per day (breakfast, snack, lunch) provided as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas´ SCI database School registry 9,228,578 17,567,105 FAIS 11A Number of school-age children receiving daily school meals (breakfast, snack, lunch) as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI database School registry 63,809 107,472 FAIS SUM OF 11N and 11Ñ Source: Save the Children database (Note: In 2017, there were no take-home rations provided) 46 Food students receive during school hours 287 students from area 1 (Nebaj, Chajul and Cotzal) and 522 students from area 2 (Uspantán, Sacapulas and Cunén) participated in the FE. The means of both areas are higher than the results of the MTE for the question “Did you have breakfast this morning?” Graph 11. Percentage of Children that said they had breakfast in the morning Source: Student questionnaire Graph 12. Student responses about school providing food Source: Student questionnaire Schoolchildren’s access to food is an extremely important factor, since the chronic malnutrition of school￾age children leads to delayed growth, according to the 2015 Fourth National Census of Public-Sector Primary School Student Height of the Republic of Guatemala.10 10 http://www.sesan.gob.gt/index.php/noticias/censo-talla-2015/75-informe-ejecutivo-censonacional-2015/file 0 20 40 60 80 100 MTE Area 1 MTE Area 2 FE Area 1 FE Area 2 Student respone to "Did you have breakfast this morning?" Yes No 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% No Yes No Yes No Yes Does the school provide breakfast? Does the school provide a snack? Does the school provide lunch? MTE FE_Area 1 FE_Area 2 47 According to this census, "the prevalence of chronic malnutrition or stunting in the Republic of Guatemala has decreased by 1.3 percentage points from 1986 to 2001; by 4.2 percentage points from 2001 to 2008; and by 8.0 percentage points from 2008 to 2015, but it is still higher than expected since no region, department or municipality can be considered free of nutritional damage; all have a stunting prevalence well above the 2.5% expected for a normal population according to WHO recommendations.” The census revealed that prevalence of chronic malnutrition is on average 37.6% at the national level. Totonicapán and Quiché are two departments with a prevalence of chronic malnutrition that is higher than that of the national average; Totonicapán is as 59.5%, while Quiché is at 55.3%. They are both classified as having a very high vulnerability in regards to food and nutritional problems. The prevalence of stunting in the municipalities in which the IDEA project is present is: Chajul, 72.0% (which is among the ten municipalities with the highest prevalence of chronic malnutrition); San Juan Cotzal, 69.1%; Cunén 66.3%; Nebaj, 61.9%; Uspantán, 55.9%; and Sacapulas, 55.9%. All of these are classified as having a very high prevalence of stunting. The census provides an analysis by age, and reveals that the differences in prevalence of chronic malnutrition between the different ages is greater for boys and girls of eight and nine years, with a prevalence of 45.7% and 54.2% respectively. For boys and girls of six and seven years, the prevalence of chronic malnutrition was 30.9% and 32.7%, respectively. This shows that the older in age of boys and girls who attend first-grade, the greater the potential for nutritional damage. In addition, the census reports that children with Spanish as a mother tongue language reach a prevalence of 26.4%, while boys and girls whose mother tongue is not Spanish, reach a 51.0% prevalence of stunting or chronic malnutrition. Student perception about their hunger during school hours Students in the sample were asked how many times during thee day they were hungry. As Graph 13 illustrates, both areas show a reduction in students’ perception about their hunger. This reduction was more pronounced in area 2 than area 1. Graph 13. Number of times per day that children are hungry; student/self-perception Source: Student questionnaire 48 G. Reading Practices and Support (household and community) The literary environment in the household is closely related to the opportunities that children have to practice their literary skills. To determine the literary environment in the household, third-grade children were asked about access to reading material at home and interactions with the different members of their household. The first section describes the changes that occurred from the MTE and FE of the children in the Intervention area versus the control area for the different types of printed materials, including: textbooks, religious books, magazines, newspapers, storybooks and coloring books. The percentage for the MTE is the average between the two areas, due to the fact that they were very similar. Table 22. Types of printed materials in areas 1 and 2 in the midterm and final evaluations MTE Areas 1 & 2 FE Area 1 FE Area 2 Textbooks 73.7% 76.8% 63.7% Religious books 73.9% 83.9% 85.6% Magazines 3.5% 18.2% 26.9% Newspapers 14.5% 48.4% 43.7% Storybooks 27.9% 49.1% 42.4% Coloring books 26.1% 58.5% 48.8% Does not have any of these books 6.3% Source: Student questionnaire In terms of access to reading material in their homes, the data show a greater increase in magazines, newspapers, storybooks and coloring books in both areas. This can be interpreted as an increased interest of parents for their children to have access to reading material, and the books students borrow from the school. A greater percentage difference in area 2 was only identified in the case of access to magazines (a difference of 8.7%). A decrease of textbooks is observed in area 2, by 10 percentage points comparable to the midterm evaluation, which could indicate that the students did not receive their textbooks or have access to them to the same degree as in area 1. Table 23. Reading practices in the home MTE Areas 1 & 2 FE Area 1 FE Area 2 You saw someone in your family read 93.7% 90.2% 92.9% Someone in your family read you a storybook 87.4% 84.0% 80.7% Someone in your family told you a story 88.6% 86.8% 81.0% Someone in your family helped you study 99.8% 100.0% 98.3% Source: Student questionnaire 49 Table 24. Reading practices in the community MTE Area 1 & 2 FTE Area 1 FE Area 2 Someone from the community read to me last week 48.7% 25.8% 38.2% I read to someone in the community 62.4% 36.9% 48.5% I borrowed a book from someone in the community 40.4% 29.6% 45.2% Did you borrow books from the Book Bank last week? 44.5% 39.0% 49.0% Source: Student questionnaire The above data were collected from the student surveys. From this self-reported data, we can interpret that the reading activities identified by the boys and girls have decreased slightly in both areas in terms of a family member reading or telling a story to the child, but more so in area 2. This is similar at the community level in terms of fewer students having read to someone in the community or being read to, but even mores so in area 2. This is notable because “reading to someone in the community” was a factor associated with literacy gains for certain literacy abilities in grades, languages, and geographical areas. This suggests that it may be important to put more emphasis on increasing these practices in the home and community. In area 1, a decrease is reflected in the self-reported practice of borrowing a book from someone or the Book Bank, which may also indicate the need to reinforce this practice. In area 2 a slight increase is reflected in book borrowing; perhaps understanding why this varies between areas would be helpful for increasing community book borrowing in area 1. Based on the more substantial decreases in area 1 at the community level, it can be interpreted that in the family there may be more of an interest in learning to read than at the community level. VII. Impact of Strategic Objective 2: Increase in Health and Dietary Practices The data reflect that the project met the programed goal for 2017 regarding the number of PTAs. The role of the PTAs is particularly important for the making the IDEA project interventions successful. Members of the PTA prepare meals and keep the food properly stored, receive trainings on hygienic practices that they share with other parents, their children, and members of the community and they sensitize parents about the importance of their children attending school. Table 25. Number of Parent-Teacher Associations or similar "school" management structures supported as a result of USDA assistance Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of Parent￾Teacher Associations or similar "school" management structures supported as a result of USDA assistance Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI Database 391 668 FAIS 19A Source: Save the Children records 50 Two different teachers explain what the PTA members do: The food committee’s that the teachers are in charge of call upon the parents according to the list because here the mothers are the ones who take turns preparing food and in addition they have organized a group of mothers who are the guides and they are in charge of supervising in hygiene and the preparation of food that is given to the children.” (“La comisión de alimentación a cargo de los maestros son los que citan a las madres de familia según el listado porque aquí las madres son los que se turnan para la preparación de los alimentos y además se han organizado un grupo de madres que son las madres guías ellas se encargan de la supervisión en higiene y preparación de los alimentos que se les da a los niños.”) “…thanks to the program that has also provided training to the mothers, and even today you see the change and improvement in the hygiene and the provision of food.” (“…gracias al programa que también ha brindado capacitación a las madres de familia y hasta hoy se ve el cambio y la mejora en la higiene y la entrega de la refacción.”) The organization of the PTAs and their involvement in preparing the school meals has been crucial for the project’s success. A. Access to School Meals Increased Knowledge on Nutrition Practices; Number of school-age children that receive daily school meals (breakfast, snack, lunch) as a result of USDA assistance As the data reveal (Table 26), the project has made the necessary infrastructure changes for the improvement of school kitchen conditions. These changes have been coordinated with the MINEDUC and its directorate; these entities have expressed that there has been effective coordination with SC at national and local levels in order to achieve the necessary changes in educational infrastructure and that the resources of each have been used well. As one mayor expressed: There have been many projects but if we talk about this program the main thing is the food security and then the infrastructure services.”(“Ha habido muchos proyectos pero si hablamos del programa lo principal es la seguridad alimentaria y seguidamente pues ya los servicios de infraestructura.”) Table 26. Situation of school kitchens MTE FE_Area 1 FE_Area 2 Does the school have an improved stove? 59% 94.4% 81.0% Does the school have a wood-saving stove? 11% 77.8% 71.4% Does the school have a chimney? 66.7% 81.0% Does the school have a hearth? 30% 38.9% 81.0% Does the school have wood? 100.0% 100.0% Source: School observation guide Another mayor commented on the holistic approach, which has led to positive results: “If we speak in general of education, before there were few children coming to study in the schools, parents weren’t concerned about getting them enrolled, now all the parents are sending them. Save the Children has raised a lot of awareness about the importance of their children’s education, in addition the support provided such food, improvements to the school infrastructure, training for teachers and mothers, everything together has motivated parents to send their children, and the children are motivated to attend and not miss school.” (“Si hablamos en general de la educación, 51 antes eran pocos los niños que llegaban a estudiar a las escuelas, no había preocupación de parte de los padres para irlos a inscribir, ahora todos los padres los están llevando. Save the Children se ha preocupado por hacerles mucha conciencia sobre la importancia de la educación de sus hijos, además los apoyos que brindan como por ejemplo los alimentos, mejoras a la infraestructura de las escuelas, capacitación a maestros y a las madres, todo en su conjunto viene a motivar a los padres para enviar a sus hijos y los niños se motivan a asistir y no faltar a la escuela.”) In sum, various stakeholders and local actors have recognized that the school feeding program, training, awareness raising, infrastructure improvement, and the other interventions carried out as part of the IDEA project are perceived as having contributed to improving the children’s health and education. B. Feeding and Health Practices In the MTE, it was found that 21,147 children had received deworming and micronutrients as a result of SC actions with public health stakeholders. Other actions promoted by the project that aim at preventive health are food and nutrition training received by parents. The FE showed that there are 121,653 children that have received deworming and micronutrients as a result of SC actions with public health stakeholders. Table 27. Number of students that received deworming medication Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of students that receive deworming medication Monitoring Record, quantity disaggregated by sex, municipality, and by Ixil and K’iche’ areas SCI Database 41,245 80,408 SEMESTER REPORTS SENT TO DONOR. Source: Save the Children database As one mother explains, the deworming was very helpful due to the cost of this for all family members (i.e. the children in the program received deworming but she had to shoulder the cost for the children not benefitting from the program): “The deworming medication has helped a lot, particularly for the children and the family in general. I say this because I have other children and I had to buy the medication and it meant a cost to me”. (“Los desparasitantes han ayudado mucho en especial a los niños y la familia en general. Por qué digo esto, porque tengo otros hijos y tuve que comprar los desparasitantes por eso significó gasto para mí”.) The project contributes to the nutrition of schoolchildren by distributing food, such as rice, black beans, wheat flour, CSB, and oil, to 100% of the schools. From these foods, school-age children obtain a minimum acceptable diet. The food they give to schoolchildren offers 57.8 percent of their individual daily kcal needs, as established by the Institute of Nutrition of Central America and Panama (INCAP). The mayors from the different municipalities expressed their opinions regarding the value of the food interventions carried out through the IDEA project: “What the program has been efficient in has been, which is a tangible result, is the meals have been there in an opportune way in the different schools”. (“De lo que si ha sido eficiente el programa ha 52 sido en cuanto hay un resultado completo, tangible es en que los alimentos han estado de manera oportuna en los diferentes centros educativos.”) “Also in this case especially is the subject of the children’s nutrition because it is a fundamental part in terms of access and availability of food.” (“También en este caso especialmente es el tema de la nutrición de los niños porque es parte fundamental en cuanto al acceso y la disponibilidad de los alimentos.”) “One of the main results of the support from the IDEA program of Save the Children is that it has reduced the chronic malnutrition in children. Before many children came to school without breakfast and this will affect their learning, but now if they do, they know that in the schools they are going to eat enough food and this motivates them to attend school.” (“Uno de los principales resultados del apoyo del programa IDEA de Save the Children es que se ha disminuido la desnutrición crónica en los niños, antes muchos niños salían de sus casas a la escuela sin desayunar y esto les afectaba en su aprendizaje, pero ahora si lo hacen, saben que en la escuelas van a comer lo suficiente y esto los motiva a asistir a la escuela.”) One mayor commented that the project has helped tremendously through the food provided to the children, which allows them to concentrate and have energy in school, but that there are also factors that influence children’s attendance and learning in school such as family disintegration, child abuse, social issues, and other factors. To help mitigate these factors, these issues could be included in training (for teachers, principals, and families) if the project continues and/or is replicated in Guatemala or elsewhere. C. School Infrastructure and Access to Clean Water and Sanitation Services School infrastructure Save the Children and the Ministry of Education have been working together to improve the infrastructure of the schools in order to improve health conditions. Thus, the current state of the school infrastructure cannot be solely attributed to Save the Children. The data reflects the infrastructure at the time of the FE and not a comparison between the MTE and FE. Table 28. Storage conditions FE_Area 1 FE_Area 2 Has a door? 100.0% 90.5% Has cement floor (or of other material)? 94.4% 81.0% Has adequate ventilation? 55.6% 47.6% Is it organized? 77.8% 85.7% Is it clean? 77.8% 85.7% Is there a trash bin? 66.7% 61.9% Does it have a washbasin? 83.3% 52.4% Source: School observation guide As can be seen in the data, there is still much to accomplish, especially in terms of having adequate ventilation, trash bins, and washbasins. The organization of the committees in charge of food storage and food preparation activities, combined with the support and monitoring of the IDEA project staff was recognized by teachers as instrumental in achieving positive results with food and hygiene practices. As two different teachers commented: “The logistics committee is responsible for organizing and ensuring the storage of food, and is conformed of a teacher with the support of the director and education counsel. I am the one in the 53 committee who is referred to about difficulties, I have not found any since organizing well and coordinating the activities well, the food is received, and to weigh and store it has been easy.” (“La comisión de logística es la encargada de organizar y velar por el almacenamiento de los alimentos, está integrado por un docente con el apoyo del director y consejo educativo. Yo soy el que integra esa comisión referirse a dificultades no he hallado ninguna ya que organizándose bien y coordinando bien las actividades pues recibir los alimentos, pesarlo y embodegarlo ha sido fácil.”) “From the outset the working model of the IDEA program was that they should organize committees by component for participating teachers and parents, so here a logistics committee was formed to have better control of the food. Control is maintained by cardex, we have a warehouse that is clean, ventilated, safe, and free of entry of animals and rodents. Every three days an IDEA program staff member comes to do constant monitoring. (“Desde el principio el modelo de trabajo del programa IDEA era que debían de organizarse comisiones por componente durante participaran docentes y padres de familia, entonces aquí se formó una comisión de logística para tener un mejor control de los alimentos. Se lleva el control por medio de cardex, contamos con una bodega limpia, ventilada y segura, libre de ingreso de animales roedores. Cada tres días viene un técnico del programa IDEA a hacer monitoreo constante.”) Increased access to clean water and sanitation services; Number of schools that have improved sources of water supply Table 29. Access to water Does this school have water? Recount Percentage BL 30 90.9% FE Area 1 16 88.9% FE Area 2 17 81.0% Source: School observation guide According to what was observed, the schools have infrastructure (tubes, connections, waste deposits) for water to be accessed. However, some do not have water during school hours11, and the water that they have access to is not potable. To use it for food preparation, the mothers boil the water. However, Save the Children has provided training on water treatment, which has been important, as reflected in these mayors’ comments: “We know that Save the Children has given a lot of support through training on water treatment through various methods of purification, then the families can consume safe water, although I understand that most prefer to boil it, they do not have much confidence in the other methods such as the chlorination”. (“Sabemos que Save the Children apoya mucho con capacitaciones sobre tratamiento de agua a través de varios métodos de purificación, entonces las familias pueden consumir agua segura, aunque tengo entendido de que a la mayoría prefieren hervirla, no le tienen tanta confianza a los otros métodos como la cloraración.”) “As a municipality, we have supported a number of communities with water projects, for example in Xeputul, we did it in conjunction with Save the Children. Most communities have their water projects, they have been taught to purify it; although most of the people prefer to boil it, they have more confidence in this technique”. (“Hemos apoyado desde la municipalidad a varias comunidades con proyectos de agua, por ejemplo en Xeputul, lo hicimos en conjunto con Save the Children. La 11 The study did not gather data about the time that the schools have access to water. This was an observation of the field supervisors. 54 mayoría de comunidades cuentan con sus proyectos de agua, se les ha enseñado a purificarla; aunque la mayoría de la gente prefiere hervirla, le tienen más confianza a esta técnica.”) These passages demonstrate that there is support from the municipalities and from Save the Children to help ensure that the communities have safe drinking water. However, it is important to ensure that the techniques that are taught are consistent with practices that the people will put into practice so that resources are used as effectively as possible. Table 30. Situation of school warehouses Area 1 Area 2 There is a space exclusively for the storage of food Does not meet criteria 5.60% 14.30% Meets criteria 94.40% 85.70% It is clean and organized Does not meet criteria 0.00% 9.50% Meets criteria 100.00% 90.50% Walls are free from humidity Does not meet criteria 5.60% 9.50% Meets criteria 94.40% 90.50% Food is stored in pallets or inside furniture that is clean and safe Does not meet criteria 0.00% 4.80% Meets criteria 100.00% 95.20% There is evidence of rodents or cockroaches (other insect or animal) Does not meet criteria 72.20% 47.60% Meets criteria 27.80% 52.40% Kardex up to date Does not meet criteria 0.00% 14.30% Meets criteria 100.00% 85.70% Equipment for measuring rations is in good condition and clean Does not meet criteria 0.00% 0.00% Meets criteria 100.00% 100.00% Food is sealed and there is no evidence of spillage Does not meet criteria 0.00% 0.00% Meets criteria 100.00% 100.00% Transportation guides are available Does not meet criteria 0.00% 0.00% Meets criteria 100.00% 100.00% Food portion estimation table is in use Does not meet criteria 5.60% 0.00% Meets criteria 94.40% 100.00% There is a Save the Children identification sign Does not meet criteria 5.60% 14.30% Meets criteria 94.40% 85.70% The weekly menu is available Does not meet criteria 16.70% 9.50% Meets criteria 83.30% 90.50% The Save the Children technician visit sheet is available Does not meet criteria 5.60% 4.80% Meets criteria 94.40% 95.20% Source: School observation guide As the data shows, a high percentage of schools are implementing the majority of actions that demonstrate the proper handling and storage of food. The issue of pest management (e.g. cockroaches, rodents, etc.) is a situation that must be ameliorated due to health concerns, taking into account the rural context. 55 Increased access to clean water and sanitation services, Number of schools with sanitation services The following data was reported in the BL: Does the school have a latrine or toilet? 100.0% Does the school have at least one separate latrine for girls? 93.9% The sanitation conditions have been the same since the baseline; there are not statistically significant percentage differences. The data show that the conditions of infrastructure in regards to school hygiene are acceptable in many aspects, such as schools having separate toilets for girls and boys and having a wash basin or some sort of washing station (see Table 31). A substantial challenge remains in regards to schools having the materials for proper hygiene such as soap for washing hands. The issue of towels is not a problem, and in fact, since towels harbor bacteria, their presence and use can be counterproductive. It is critical that the issue of schools having soap be resolved due to the importance of washing hands with soap to prevent sickness and infection. Table 31. Situation of sanitation services Area 1 Area 2 Does the school have a latrine or toilet? No 16.70% 0.00% Yes 83.30% 100.00% Does the school have clean latrines or toilets? No 5.60% 14.30% Yes 94.40% 85.70% Does the school have toilets that have a key? No 5.60% 9.50% Si 94.40% 90.50% Does the school have a latrine or toilets that are separated for boys and girls? No 5.60% 14.30% Yes 94.40% 85.70% Does the school have a wash basin? No 5.60% 14.30% Yes 94.40% 85.70% Does the school have a sink? No 22.20% 38.10% Yes 77.80% 61.90% Does the school have any other type of washing station? No 55.60% 57.10% Yes 44.40% 42.90% Does the school have soap? No 22.20% 47.60% Yes 77.80% 52.40% Does the school have towels or other material for drying? No 50.00% 61.90% Yes 50.00% 38.10% Source: School observation guide D. Hygiene Practices The hygiene practices were documented through the observation guide of the children before and after they ate in order to confirm if the washed their hands with soap and brushed their teeth with toothpaste. The observations were carried out in all of the schools in the sample and one class in each school was randomly selected for observation. 56 Table 32. Children’s’ hand washing practices Area 1 Area 2 Do children wash their hands with soap before eating? No 0.00% 52.40% Yes 100.00% 47.60% Do children wash their hands with soap after eating? No 0.00% 52.40% Yes 100.00% 47.60% Do children have their own toothbrush and toothpaste No 11.10% 42.90% Yes 88.90% 57.10% Source: School observation guide The data reflect that, in area 1, children are reportedly washing their hands with soap before and after eating and the great majority reportedly has their own toothbrush and toothpaste. The lower rates of the desired practices that were reported at the school level in area 2 could be due to area 2 having less time receiving the interventions and/or cultural practices that are more difficult to change.12 Also, the lower rate of hand washing with soap is consistent with the information in the previous table that demonstrated that in area 2, only 52.4% of the schools have soap. Interestingly, in area 1, it was reported that in 100% of the schools, children wash their hands with soap, even though 77.8% reported having soap at the school. This can be explained by the fact that it is common for the teacher to keep the soap in the classroom and provide it to the students for hand washing. During the evaluation, it was observed that the children washed their hands with soap; where they stored the soap was not observed. E. Access to Food Preparation Equipment and Instruments Table 34. Situation of the school kitchens Type Area 1 Area 2 Recount % of column number Recount % of column number Kitchen free of garbage Does not meet criteria 1 5.60% 2 9.50% Meets criteria 17 94.40% 19 90.50% Kitchen free of animals Does not meet criteria 1 5.60% 2 9.50% Meets criteria 17 94.40% 19 90.50% Women wash their hands before preparing meals Does not meet criteria 0 0.00% 0 0.00% Meets criteria 18 100.00% 21 100.00% Wearing hairnets or tie their hair Does not meet criteria 4 22.20% 7 33.30% Meets criteria 14 77.80% 14 66.70% Use apron Does not meet criteria 2 11.10% 2 9.50% Meets criteria 16 88.90% 19 90.50% Outsiders are not permitted in the kitchen Does not meet criteria 3 16.70% 7 33.30% Meets criteria 15 83.30% 14 66.70% Source: School observation guide Overall, appropriate conditions for food preparation were identified. There were two aspects identified that reflect a need for improvement in order to prevent food contamination: the use of hairnets and other people (not in charge of food preparation) entering the area. Most of the kitchens have doors that could restrict the passage of people outside the area, however due to the amount of mothers cooking and the heat that is contained in these areas, they are kept open which makes it difficult to restrict entry. It is generally observed that students and other teachers who were not involved in the preparation of food would enter the area. 12 Another cause may be that in area 1, the teachers, students and community members were aware that that the data collection team was coming (and, as observed, were cleaning when they arrived). Thus, in area 1 the percentages could be lower normally. 57 Reinforcing the importance of wearing hairnets and avoiding teachers and students from entering the food preparation areas may help improve this situation. F. Partnerships with Local and Community Organizations Since the beginning of its intervention, the IDEA Project has involved several key stakeholders, including community leaders, representatives of the Community Development Councils (COCODEs, in Spanish), parents and school principals and teachers. SC has established public-private partnerships with 7 organizations, among them: Asociación Chajulense (who provided the facilities for the main warehouse located in Aldea Batzul, Chajul; agreements with two municipalities (Cunén and Uspantán), University of San Carlos de Guatemala, Bank of Central America, Colgate Palmolive and Asociación Equinos. Table 34. Number of public-private partnerships formed as a result of USDA assistance Data Source Means of verification Cumulative 2016 Cumulative 2017 SOURCE OF DATA Attendance Monitoring Record SCI Database Number of public￾private partnerships formed as a result of USDA assistance Monitoring Record, types of partnerships: agreements, letters of understanding, others. Provide information about the organization and its scope. SCI database 6 9 FAIS 11L Source: Save the Children records It was identified through interviews with different actors from the central and local governments, Ministry of Education departments/units, and the University of San Carlos of Guatemala, that there is a very good relationship between the IDEA project and the different institutions. This has resulted in agreements, understandings, and policies at the ministerial level. It has also facilitated the implementation of the project, its scalability, and replication in other geographical areas. These relationships, agreements, and policies, as well as the project’s expansion, have created some of the conditions for the sustainability of school feeding policies at the level of MINEDUC. Organizations that support the school Graph 14. Other organizations that provide support Source: Principal questionnaire 58 There is a greater support of NGOS in area 1. The organizations are devoting greater investment in the Ixil area. Previously there had been greater support for the K’iche’ area. In the final evaluation, changes in investment by NGOs by region were identified. As reflected in Table 35, the municipalities and private initiatives have invested in the area 1, not the large development organizations. This may be referring to the companies that exploit the natural resources and make social investment through the municipalities. In both areas, there are also other NGOS and faith￾based organizations that support schools. Table 35. Organizations that provide support in area 1 and area 2 Area 1 Area 2 World Vision No 100.00% 100.00% Yes 0.00% 0.00% UNICEF No 100.00% 100.00% Yes 0.00% 0.00% Cáritas No 100.00% 100.00% Yes 0.00% 0.00% Municipality No 83.30% 100.00% Yes 16.70% 0.00% Private business No 83.30% 100.00% Yes 16.70% 0.00% Other No 27.80% 52.40% Yes 72.20% 47.60% Source: Principal questionnaire Type of support that they receive from these organizations Graph 15. Type of report received by other sources Source: Principal questionnaire 59 The final evaluation results indicated that more support has been provided in the Ixil area than the K’iche’ area for school health and hygiene, education, school infrastructure, and school breakfast. Also, this has shifted from the time of the baseline data collection. According to a teacher (in Uspantán), though there has been support from other organizations, Save the Children’s support has been more comprehensive: “Before, programs from Hearts and Hands were also at the school, and Movimundo, but the help received, compared to what Save the Children has given, was very little. This program is focused on the well-being of the school and even the trainings that they gave us were excellent. I think that if their goal is to improve the quality of education, they did so because we have never had trainings like those, neither from NGOs and much less from the government or the municipality”. (“Antes aquí en la escuela también estuvieron los programas Corazones y Manos, y Movimundo, pero la ayuda recibida a comparación de lo que Save the Children ha dado fue muy poca. Este programa se enfocó en todo el bienestar de la escuela que hasta las capacitaciones que nos brindó estuvieron excelentes creo que si el objetivo de ellos es mejorar la calidad educativa lo lograron porque capacitaciones como esas nunca hemos recibido ni departe de ONG’s mucho menos de parte del gobierno o de la municipalidad)”. The municipalities explained how they have provided support to the communities, often complementing the interventions of the IDEA project: “We as a municipality have interventions in all, in the 49, because in one way or another we have supported, if the IDEA project makes a project, we get involved, if they buy the materials, we get them there, in all of the projects the municipality has an intervention.” (“Nosotros como municipalidad tenemos intervención en todas, en las 49 porque en alguna u otra hemos apoyado, si el proyecto IDEA hace un proyecto, nosotros nos involucramos, si ellos compran los materiales, nosotros los trasladamos, en todos los proyectos la municipalidad tiene intervención.”) (Mayor/councilmembers, Uspantan) “As a municipality we have created the office of water to give support to communities, because after performing some lab tests on the water that is consumed, in the majority of communities it was found that it is contaminated with animal manure and this is affecting the health, especially of children and of families in general”. (“Como municipalidad hemos creado la oficina de agua para darle acompañamiento a las comunidades, pues luego de realizar algunos exámenes de laboratorio al agua que se consume, en la mayoría de comunidades salió que está contaminada con estiércol de animales y esto afecta en la salud, especialmente de los niños y de las familias en general.”) (Mayor/councelmembers, Uspantan) In the municipality of Nebaj, council members expressed that with the support of different organizations they are doing some of the activities in health, nutrition and education in the municipality. For example, HH is the Foundation against Hunger (FH) that provides training for women in the municipality. Additionally, there are Peace Corps volunteers that have been supporting in the training of young people on the topic of violence prevention. Also, the organization Pencils of Promise and APEDIVIMI have supported the construction of classrooms in schools. In the year 2016, FH and the municipality built a treatment plant. Finally, they expressed that they have not received support like they have now with Save the Children which provides a full support to the schools. In the municipality of Sacapulas, they explained that they have coordinated with different NGOs such as HELVETAS which is implementing training in health and nutrition to families in the municipality; these same actions they have been doing also with the organization NEXOS Locales. They provided other examples as well, including coordination with the Universidad del Valle who will be offering a certification for cultivating amaranth to then incorporate it into the school feeding in the municipality. In San Gaspar Chajul, it was shared that the municipality has supported the recruitment of 34 primary-school teachers and that they coordinate with the CTA to identify where they lack teachers. They also work with 60 different organizations such as APEDIBIMI and Horizons without Boundaries. Support is for reducing malnutrition as well as providing support for elementary school (though the support wasn’t specified). In San Juan Cotzal, the municipality shared that they are supporting a number of communities in the issue of malnutrition. They coordinate efforts between city hall and the Pan American Health Organization (PAHO), based in Nebaj. Another nutrition program is from USAID, and the UNDP supports three groups of women with chicken farms. They have a municipal committee that works hand in hand with the external/international cooperation. In sum, there are various types of support that come directly from the municipality and there are also different organizations offering support in health and education (amongst other things). The comments suggest that from the perspective of the municipalities and the communities, the support offered by Save the Children is comprehensive and having a positive impact in health and education. It is less clear what the specific types of support are that the other organizations are offering in health and education. It is important to ensure that these actions are complementary and not duplicating efforts, as well as assuring that the approaches are coherent with each other rather than offering contradictory approaches (as could be the case with literacy development support that is L1 only versus L1 and L2 and pedagogical approaches and techniques that may be very different from each other). Furthermore, it appears that the municipalities are organized to coordinate these efforts; they can provide detailed information about the interventions. It is recommended that Save the Children have a database/spreadsheet for each municipality and community that details the support provided by the municipality and other organizations so that efforts can be maximized. VIII. External and Internal Factors that Affected the Results A. External Factors Save The Children has several strengths that have facilitated the success of the IDEA Project. First, it is recognized for its experience and credibility by the Government of Guatemala, especially the Ministry of Education (MOE) at the central, departmental, municipal and local levels, Municipal Governments, international cooperation entities and it is also recognized by peer organizations for its successful implementation of programs and projects focused on promoting and protecting the rights of children and adolescents. SC’s specific structure, process, policies and procedures facilitate effective implementation of their various programs and projects. Their internal control and monitoring / evaluation systems reflect constant and transparent accountability. Their human resources facilitate project progress because they possess the ideal characteristics: they have a profound understanding of, and sensitivity to, the realities of Guatemalan families, especially those who live in poverty and extreme poverty and who encounter marginalization and limited access to their rights. The work team has experience in the creation of educational models and models that seek food security with the added value of being contextualized, modern and innovative. The work team also is well versed in inter-institutional coordination with peer organizations and especially with interested parties from the various projects and programs being implemented. They value democratic, consensual and inclusive participation while functioning as facilitators of the process. Another advantage of the field team is that they are able to function in the local languages. External factors that facilitate project success include the willingness and strong participation of the interested parties in the implementation of projects focused on educational quality. They also include the 61 MOE’s acceptance of SC’s innovative contributions and the political will to validate them, put them in practice and scale them up. Additionally, a facilitating element has been the community leaders’ interest in improving educational quality and thus their involvement and support. The parents are a key factor for social auditing which in the country’s current socio-political environment is a factor that motivates teachers to increase teaching quality. The big challenges facing the educational system are circumstances that limit the achievement of expected results. The educational indices in Guatemala are the lowest in the Central American region. The main indicators for this situation are: i. Primary education coverage continues to decline: The net schooling rate in 2013 was 85.4%, in 2014, 82.0% and in 2015, 81.0%. In 2009, primary school attendance was nearly universal at 98.3%.13 ii. Increased dropout rates. Of the children who begin school in first grade, only 60% finish sixth grade and only 39% are the right age when finishing. An average of 7.71% of students abandon primary school every year and 14.86% fail their grade. iii. The extremely limited budget for the educational system in general. Of the budget assigned to Education, 70% is used for teachers’ salaries and 29% for administrative staff salaries. Only 1% is invested in infrastructure and projects to improve educational quality. iv. Minimal and weak school infrastructure. In 2015, no new schools were built and no damaged infrastructure was repaired. v. The lack of teachers and their poor distribution. There is a prevalence of schools where only one teacher covers the entire school population. vi. Reading and Mathematics evaluations in 2015 continue to show that students are behind in their management of the expected knowledge and abilities. Reading achievement for 2015 was 25.97%, while in 2014 it was 26.02 %.14 The SERCE study published by UNESCO in 2008, placed Guatemala close to last place in Latin America and the Caribbean in Math and Reading tests.15 vii. Inadequate and limited teacher training. viii. The lack of linguistic and cultural relevance in the educational system.16 ix. In Guatemala, illiteracy is a reflection of the exclusionary economic, social and cultural environment that has relegated the educational system to adults over 15 years of age, especially in relation to women and youth living in poverty and extreme poverty in the rural areas. Between 2002 and 2011, illiteracy in people over 15 was tending to decrease as it went down from 29.32 to 17.46%. That is, it dropped 11.86 percentage points. Nevertheless, literacy for women is still a challenge. In 2011, three of every 10 women over 14, and one of every 10 between 15 and 24, do not know how to read and write. One of every two indigenous women over 14 does not know how to read and write.17 13 SEGEPLAN. 2010. Third report on the progress in accomplishing the Millennium Development Goals. 14 Results published in the MOE’s General Directorate for Educational Evaluation and Investigation (Digeduca). 15 UNESCO. 2008. SERCE Study. 16 National Development Plan K’atun: Our Guatemala 2032 17 National Development Plan K’atun: Our Guatemala 2032. p.168 62 In addition to the structural problems, the effects of poverty and extreme poverty in which children are living is reflected in high malnutrition and child labor rates. Studies have shown that malnutrition negatively affects enrollment, permanence in school and academic performance, especially in primary school students.18 Another recent factor that puts the project’s results at risk are the high crime rates that force children to be absent from school. This was found in the schools included in the sample for the evaluation. The situation of violence and insecurity that characterizes the country is a serious obstacle for progress and democracy and other development processes. At the local levels some of the limiting factors are:  Conflicts between leaders and teachers in some communities  A reduced number of class days o Strikes and other unofficial activities reduce the number of class days by approximately 30%  Poor school infrastructure: in some cases, there are not enough classrooms, they are in poor condition and the land does not belong to the Ministry so they cannot make investments there  Some principals, teachers and Educational Supervisors are also resistant to change  Parents often have to seek work in a variety of ways and thus their accessibility for participating and learning is limited Some internal SC factors that limit project success include the time it takes the technical team for delivery of materials and food, which limits the time they can spend: 1) following up and supporting teachers in the classroom to monitor and advise them on the implementation of the educational methodology, 2) support for reading camps, and 3) collecting information to compile lessons learned and good practices. It also results in limited participation in opportunities for coordination, organization and negotiations at the municipal level (COMUDE). 1. Facilitators a) Support and/or will of the central/local governments, support from community members Community members have participated by supporting the various aspects of the project. For example, parents have donated wood, they participate in the trainings, and they have helped build the school stoves. The mothers’ preparation of the food for the children at school has also been valuable: “To achieve 100% of Project execution, parents have contributed with firewood, and sending tortilas their children, since in at school only a meal is provided. They have also contributed with their time receiving training, and through workforce for the construction of improved stoves for the school kitchen. Mothers have participated in the preparation of meals, more at the beginning of the project; today, there are three cooks who prepare the meals, but as mothers we always show up when we are requested” (Mother, from Nebaj). (“Para que se logre el 100% sobre la ejecución del proyecto, los padres de familia han contribuido con leñas y les envían tortillas a sus hijos ya que en el establecimiento solo dan comida. Se ha aportado también tiempo para las capacitaciones y mano de obra para la construcción de las estufas mejoradas en la cocina de la escuela. Las madres han participado en la preparación de los alimentos más al principio del proyecto hoy pues hay tres cocineras que preparan los alimentos, pero nosotros las madres siempre vamos cuando nos citan)”. Teachers commented on how responsible the logistics committee at the community level has been, and that the training from the IDEA project has helped make this happen, as well as the support of FOCE. The good communication and coordination was also highlighted, as well as the integrated way that the MINEDUC and IDEA work together. 18 Educational challenges for Guatemala http://programassocialessandratorres.com.gt/2015/06/desafios-educativos￾de-guatemala/ 63 “As far as food management, it has been great. The members of the logistics committee are very responsible, we have not had any repairs as of now; I think we owe most of this to trainings that the IDEA program, along with FOCE, has provided. Additionally, I think that the good communication and coordination has been a determining factor for the success of food management, more so today because we do it in a complimentary way. Now we don´t say `these are IDEA’s, let’s take better care of them´ or `these are from MINEDUC´; now, all of it is the school’s, now it’s all for the children and should be taken care of; the commission sees it also in an integrated--MINEDUC/IDEA--way” (Teacher). (“En cuanto al manejo de los alimentos, ha sido muy buena, los integrantes de la comisión de logística son muy responsables, no hemos tenido ningún reparo hasta ahora; creo que esto en gran parte se lo debemos a las capacitaciones que el programa IDEA, juntamente con los de FOCE nos han brindado; aparte creo Yo la buena comunicación y coordinación ha sido un factor determinante para el éxito del manejo de los alimentos, y hoy más porque lo hacemos de forma complementaria, ahora ya no se dice estos son de IDEA cuidémoslos más y estos son del MINEDUC, ahora todo es de la escuela, ahora todo es para los niños y deben ser bien resguardados, la comisión hoy lo mira también de manera integrada MINEDUC/IDEA.”) The director of DEFOCE also talked about the good relationship between the Ministry of Education and Save the Children, and that SC’s initiatives have been complementary and aligned with those of the Ministry: “Particularly in terms of the management from the Ministry of Education and Vice Ministry´s Office, there has been good communication. Her (Dr. Mendez), representing Save the Children participates in any of our initiatives and always is careful to make sure its aligned with the Ministry´s needs; her ways of advocating, complementing and supporting are very good, so we have a good relationship” (DIGEFOCE Director). (“En lo particular con esta gestión del ministerio de educación y el vice￾despacho, ha habido muchísima relación, muy buena, ella (Dra. Méndez) representando a Save the Children participa en cualquiera de las iniciativas que nosotros tenemos y siempre lo hace con el cuidado de que sea alineado en las necesidades del ministerio, pero su forma de asesoría, complemento y acompañamiento es muy buena, entonces tenemos muy buena relación”. b) Other projects/interventions that are complementary As discussed in the previous section, schools receive help from the municipal authorities, private enterprises, and others in school health and hygiene, education, school infrastructure and school breakfasts. This has been an additional facilitator in the implementation of the IDEA project. The Vice Minister of MINEDUC commented that the IDEA project is “aligned and complements the efforts” and that it is well adapted to the environment of the communities. He also commented that the needs were well identified and that it was clear to him that there was a good relationship between the project staff and community members and that logically leads to successful implementation. He also commented on the central office of SC (in Guatemala City) having “leadership that has allowed for good coordination at a central and local level”, that they have good knowledge of the work and issues, and that they have a “special dynamic” for working. A professor from the University of San Carlos in the Quiché region commented that the project activities that he participated in were interesting and that he had heard from his colleagues that they have a very positive impression of the IDEA project. 2. Obstacles and Challenges One challenge pointed out by the Vice Minister of Education is that many donors tend to work in the Western region of Guatemala, perhaps because of the history of the war and the difficulties that region has faced. However, he believes it is important to move to other regions of the country, like the eastern region where similar problems exist. He thinks that it is very important to focus on making sure that there is installed 64 capacity in where the projects have been operating, and that these best practices should then be replicated in these other areas of the country. He sees this not particularly as a challenge only for Save the Children, but as a systemic challenge involving the government and donors in general. B. Internal factors 1. Facilitators The Vice Minister of Education commented that there is excellent coordination with Save the Children. According to him: “Save the Children has the advantage that prior to doing an intervention, they always take into account the Ministry to hear their opinion, their needs, the general guidelines, and they have the ability to adapt all of this and incorporate it within their project, and make it much more dynamic.” (“Bueno la coordinación es excelente, digamos Save the Children tiene la ventaja que antes de hacer intervención, siempre busca al ministerio para oir su opinión, sus necesidades, las orientaciones generales y ellos tienen la capacidad de adaptar todo esto para ponerlo dentro de la idea de su proyecto y hacerlo mucho más dinámico”.) He believes that there are best practices from the project that should be replicated in other areas of the country and adapted so to the cultural richness that exists and varies across Guatemala. The director of DEFOCE emphasized the importance of the government knowing what they want in terms of policies and interventions, and that he has a clear vision of for the guidelines for food and health interventions. He talked about how it has not been at all difficult to unify forces with SC, PCI, and CRS to work together and that when there is a law in place, they can all continue to coordinate and advance. 2. Obstacles/Challenges Some of the difficulties mentioned by the teachers were: ● In the beginning, parents had a hard time culturally accepting the food offered by the project, and that mothers got tired of making the food at the school. Both of these issues were resolved through dialogue. ● When mothers cooked the beans at the school, they weren’t sufficiently cooked by school-feeding time. So they resolved this by having the mothers bring the beans already cooked and to just heat up. ● It was difficult for the mothers to know how much was flour was needed for the required number of servings ● A school principal mentioned that it was hard to get some teachers to change their teaching practices and they resisted the change. They had to become more conscious in order to implement the new practices. ● The infrastructure of the kitchen is not always big enough to prepare food for all of the children. 65 ● When they didn’t receive food for the school from SC at one point, children were hungry and some didn’t attend school. When the food was provided again, everything went back to normal. In sum, though there were some difficulties at the community level (mostly at the beginning of the project), these difficulties were resolved through dialogue and the support of SC. IX. Relevance, Efficiency and Efficacy A. Relevance of the Project to the Context IDEA project actions are aligned with the MOH 2016-2020 Strategic Plan, and its fundamental principles: Transparency, Inclusion, Dialogue and Social Participation, Linguistic and Cultural Ownership, Multiculturalism and Inter-culturalism, Education as a Right, and Equity and Equality. The project’s alignment with the strategic guidelines of the current MOE Strategic Plan can be seen in actions such as: promoting inclusion of the groups least covered by the system, which aligns with the strategy of: Equal opportunities for all. Modernization of the educational system is another MOE strategic direction. The schools present a delay in modernization. In the rural area they are far from having access to technology for education. The limited professionalization of the teachers means that the methodology, educational material and pedagogical approaches being used are obsolete. The innovations promoted by the project are strongly supporting modernization. The educational tools for strengthening reading ability, the formal training of teachers, the methods and educational materials, the distribution of school supplies, learning materials and backpacks support the educational quality that is an important part of the MOE Strategic Plan. Achieve adequate and well equipped schools is another component of the MOE Strategic Plan and the project contributes to it through the rehabilitation/construction of school infrastructure, and equipping kitchens and storage areas. This is a contribution that adds value to the beneficiary schools which are in the areas least attended to by the state. The educational material for children and for the teachers also supports this strategy. Education as an inalienable right is another component where the project and Save the Children as an institution that promotes children’s rights and advocates with national and local authorities so that children can enjoy this right to education; they also promote this right with local leaders and especially parents. Commitment and participation of everyone to support improved education is another strategic component of the MOE and is strengthened by the participatory actions promoted by the project. In addition to the support that the project provides for the MOE, the project strengthens the achievement of the Peace Accords which, although they were signed in 1996, still represent a national agenda towards reducing the structural causes that lead to the exclusion and inequality of large population groups who live in poverty and extreme poverty. The IDEA Project is also aligned with the “National Development Plan, el K’atun: Our Guatemala 2032” in terms of educational priorities. This Plan defines the country’s development over the next 20 years. The IDEA Project is also aligned with the programmatic priorities of the United Nations Development Assistance Framework -UNDAF- especially with regard to Social Development. It shares the commitment 66 to development of the same target population: boys, girls, women and indigenous populations and strives for inclusive and sustainable development, opening spaces for a multicultural and diverse society. The IDEA Project coincides with the USAID NATIONAL STRATEGY FOR DEVELOPMENT COOPERATION - GUATEMALA 2012-2016, specifically in Development Objective Two that seeks greater economic growth and greater social development in the western highlands. USAID seeks results in education focused on improved quality and access to education (RI 2.3). Its goals are: Better reading ability for children in primary school. USAID’s support for reading and writing is focused on improving the education models and on strengthening the quality of basic education with the long-term goal of increasing the number of students in the early grades who read at the level established in the standards. 19 The different parties to the project are: the MOE authorities in the Central offices, in the project area (CTAs) and in schools (directors and teachers). Additionally, the municipal governments represented by the mayors and finally the community leaders who represent the communities and the households from which the beneficiary students come. The central office of the MOE authenticates the diploma program that the teachers receive in the Literacy Boost methodology. The MOE authorities trust that with the project’s intervention there will be important improvements in the key educational indicators that will orient education policies and programs in the country. The Technical Administrative Coordinators (CTAs), principals, and teachers are aware of the agreements and contracts that have been signed; they are organized, participate actively in the planned actions and are willing to help accomplish the objectives. The teachers and directors recognize that the project’s methodology directly contributes to the achievement of the objectives aimed at integrating the country’s cultural and linguistic diversity and improving the reading abilities and nutrition indicators of school children. The parents as well as the leaders understand that the project’s contribution goes beyond mere distribution of food, school supplies and improvements to the school infrastructure. They recognize the value of the project with regard to improving their children’s reading abilities and their nutrition as they relate these two benefits to having an impact on their children’s lives. The municipal mayors see the project as an opportunity for improvement in their municipality and thus have signed cooperation agreements (political, logistical and resources) in order to also contribute to achievement of the objectives. The project is based on the Norms and Guidelines of the Ministry of Education and complements the policy of free school feeding and the policy of improving the infrastructure and sanitation of MOE schools. The project complements the MOE’s educational quality actions. Their principal actions complement the National Reading Program - “Let’s Read Together”. Although the organized community leaders do not always have formal plans, they know which improvements the communities need and among them are child protection, educational quality, improved school infrastructure and decreased hunger. For this reason they see the project as a complement to their efforts. At the beginning of 2016, there was a change in municipal governments and the project continued implementation during this transition. During the political campaign, the project stayed on the margins of promoting the activities and once the new authorities took possession of their posts, close coordination was begun in order to seek complementarity interventions for the benefit of the communities. 19 USAID. ESTRATEGIA DE COOPERATION PARA EL DESARROLLO DEL PAÍS - GUATEMALA 2012-2016. https://www.usaid.gov/sites/default/files/documents/1862/Guatemala_CDCS_Spanish.pdf 67 The project has not lost relevance. Its objectives, plans, strategies and activities continue being relevant and appropriate in the current context, which has actually changed very little. Some indicators have improved such as community organization and community participation in the educational process. Other indicators continue to be the same, such as poverty, low educational quality, low investment in development, and the lack of plans and actions that promote sustainable development. Other indicators have gotten worse, such as primary school coverage, violence, decreased public investment, and the political crisis. In this context, the project fosters optimism for the children, their parents, the leaders, and local governments. i. Extent to which the project complements health and education policies and educational objectives and approaches Mothers and fathers were asked how they help or contribute to the implementation of the IDEA project. They expressed that they mothers have participated in preparing food and helping improve personal hygiene such as hand washing and tooth brushing. They also expressed appreciation for the support that the municipality provides through paying for a cook for to prepare school meals. They explained that there are committees to help with the organization and oversight for each of the project’s interventions, which helps to accomplish the goals. Additionally, the municipalities are aware of the improvements and help work towards the goals. They keep track of the levels of malnutrition and make contributions to reduce it. In education, they (in Nebaj) worked with SC to restore the municipal libraries, which helps the children with their studies. See citations from the focus groups and interviews below: “Most families have participated in the preparation and manipulation of food, personal hygiene, hand washing, teeth-brushing” (father, from Nebaj). (“La mayoría de las madres de familia han participado en la preparación y manipulación de los alimentos, higiene personal, lavado de manos, cepillado de dientes)”. “Before, we had to go prepare the food for our children, but now the mayor pays a small amount towards having someone to cook; although, we always help her because on her own is not enough, it´s a lot of food that is prepared for the children” (mother, from Chajul). (“Antes a nosotros nos tocaba ir a preparar la comida de nuestros hijos, pero ahora que el alcalde paga cocinera es muy poco, aunque siempre vamos ayudar porque ella sola no se alcanza, es mucha la comida que se hace para los niños)”. “As a member of the education council, we always come to check and organize the warehouse and place the products according to what we have been instructed in the trainings we have received; also, the technician always shows up to supervise that the products are well placed and conserved— if it´s not like this, we get in trouble” (mother, from Uspantán). (“Yo que forma parte del consejo educativo siempre venimos a revisar y a ordenar la bodega y ubicamos los alimentos según lo que nos han dicho en las capacitaciones que hemos recibido y además el técnico siempre llega a supervisar si los alimentos están bien ubicados y conservados, si es todo lo contrario nos regañan)”. “In the case of men, we have become organized through the logistics committee which distributed different roles to fathers for reception and storage of the products in the warehouse. Which is, by the way, a very organized and clean place; in the case of other fathers, they help with workforce of the school´s needs; for example, with school movement, among other activities. Mothers are specifically in charge of meal preparation and providing their time for trainings programmed through the project” (father, from Cunen). (“En caso de los hombres nos fuimos organizando con la comisión de logística quien distribuyo los cargos a los padres para la recepción de los alimentos y 68 almacenarlos en la bodega que por cierto un lugar muy limpio y ordenada en caso de otros padres ayudan con la mano de obra con las necesidades de las escuelas por ejemplo la circulación de la escuela entre otras actividades. Las madres se encargan específicamente en la preparación de los alimentos y dan su tiempo en las capacitaciones programadas por el proyecto)”. “We have had meetings in COMUSAN and according to the reports at the municipal level, there has been an achievement of reduction in the percentage of chronic malnutrition; and if it hasn´t gone down a lot at least we´re making an effort for it to not increase. With Save the Children´s support, we re-opened the school library in the municipality; children as well as youth, can now visit the library and partake in their school research. It´s been very useful for them” (Council member, from Nebaj). (“Hemos tenido reuniones en la COMUSAN y según los informes a nivel municipal se ha logrado reducir en un porcentaje la desnutrición crónica, y si no baja mucho por lo menos nos estamos esforzando para que no se aumente. Con el apoyo de Save the Children reactivamos una biblioteca escolar en el municipio, tanto niños y jóvenes asisten a la biblioteca a realizar las investigaciones que les dejan en sus escuelas, está siendo de mucha utilidad para ellos)”. Also, to ensure that the interventions are aligned, SC participates in coordination with other donors and government entities. SC participates in two of the big structures of the Ministry of Education; one is the inter￾institutional coordination (SC, PCI, and CRS) and the other is with the general directorates of the Ministry of Education that are involved with these projects: ● DIGEBI: Bilingual and Intercultural Education Directorate ● DIGECABI: Education Quality Directorate ● DIGECUR: Curriculum Directorate ● DIGEPSA: Community Participation and Support Services Directorate This helps to ensure good coordination and communication and that all of the goals are aligned towards the same overarching government goals. As the director of DIGEFOCE explained: “And we as DIGEFOCE and the three of them, PCI, CRS and Save the Children, who come from the part of the budget provided by USDA, are involved with each other. Then comes the other moment, the part where they participate, that is the inter-institutional coordinating group for the implementation of the school feeding law. The actors involved are FAO, CARE, ICAP, WFP, CRS, PCI, Save the Children, Ministry of Agriculture, Ministry of Health, the Secretary of Food Security, Ministry of Education, and it is now integrating the office of the Procurator of Human Rights. So those are two areas where we have a high-level coordination; one is to work together in what we do and the other is for policy advocacy related to the school feeding law”. (“Y nosotros como DIGEFOCE y ellos tres, PCI, CRS y Save The Children, que vienen de un tronco presupuestario que se llama USDA, tenemos este tipo de acercamiento, luego viene el otro momento, la otra parte donde participan, que es la mesa de coordinación interinstitucional para la implementación de la ley de alimentación escolar, ahí participa FAO, CARE, ICAP, PMA, CRS, PCI, Save the Children, ministerio de agricultura, ministerio de salud, la secretaria de seguridad alimentaria, ministerio de educación y ahora se está integrando la Procuraduría de Derechos Humanos. Entonces esos son dos ámbitos en donde nos encontramos y tenemos una coordinación de alto nivel, una es de trabajo conjunto en lo que hacemos y la otra es para la incidencia en la política relacionado a la ley de alimentación escolar)”. This joint work is important in terms of getting the law into place and assuring that the main implementers of USDA funding are aware of and aligned with the Guatemalan governments laws and goals. 69 Key and governmental actors were asked about which health, nutrition and school education actions they are supporting in the municipality. The mayor of Uspantán mentioned that they had experienced many difficulties in terms of children having access to food, and that the IDEA project perfectly complements their government plan. The project’s interventions are opportune and respond to the needs of the communities. Even though the government provides some money for school feeding, it is not enough to provide a meal that is sufficient and balanced, which they can now do with the help of SC. In Nebaj, the municipality considers that the actions of SC complement those that the municipality provides as well as support from other donors. They expressed that they are continuing the priorities of the previous municipal government, which is to improve literacy and health, and SC is helping to support these goals. ii. Importance of the interventions given the actual situation of health and education Parents were asked what has changed in schools since the IDEA project began. They highlighted the improved nutrition, infrastructure, and hygiene of the children. They also mentioned the importance of the school stoves and the development of health and education in the community. “Hygiene is very helpful for children since they are being taught to wash their hands and brush their teeth. Before, there was a lot of suffering seen in mothers in charge of food preparation since the school did not have a kitchen, but thank God that Save the Children has supported us with the construction of a kitchen and there has been an improvement in the development of school feeding, health, and education.” (“La higiene es una de mucha ayuda para los niños ya que se les ha enseñado como lavarse las manitas y el cepillado de los dientes, antes se sufrían mucho en caso de las madres para la preparación de los alimentos porque en la escuela no contábamos con la cocina, pero gracias a Dios que Save the Children que nos apoyó en la construcción de la cocina y si, tenido hemos tenido un buen desarrollo en cuanto a la alimentación, salud y educación)”. “Save the Children´s actions through the IDEA program definitely respond to real and heartfelt needs of the population, especially the school population. The scourge of malnutrition has been affecting the Ixil area in general; and it´s in its right to recognize Save the Children´s contribution in this. Now, children in school are well fed; I understand that the food they receive is nutritious, mothers have been trained in a variety of recipes that can be made with the products and, the best part is that today the meals that are prepared in kitchens are dignified, since in a tripartite way (Save the Children, the municipality, and the community) we have been improving the kitchen environment, and Save the Children has provided the equipment” (teacher, from San Juan Cotzal). (“Las acciones realizadas por Save the Children por medio del programa IDEA si responden a las necesidades reales y sentidas de la población, especialmente la población escolar. El tema de desnutrición es un flagelo que ha venido afectando al área Ixil en general y es digno de reconocerle a Save the Children la contribución que realiza en este tema, pues hoy los niños en las escuelas son bien alimentados, tengo entendido que los alimentos que reciben tienen muchos nutrientes, las madres por su parte han sido capacitadas sobre variedad de recetas que se pueden elaborar con el producto y lo mejor es que hoy los alimentos se preparan en cocinas dignas, pues de manera tripartita (Save the Children, municipalidad y comunidad) hemos venido mejorando los ambientes de cocina, además Save the Children las ha equipado”.) B. Efficiency in the Use of Resources The team that implements the project is technically very efficient and is characterized by many years of experience in development. They have well-recognized experience in management of programs and 70 projects focused on health, nutrition and education in the Guatemalan context. They recognize and are deeply sensitive to the conditions of children, especially in the rural areas and they have significant experience in managing US funded multi-year programs. The Vice Minister of Education recognized the leadership and commitment of the Save the Children project staff: “The leadership and the empathy of the project are triggers, and they identified the needs very well, they had very good approach, and always do so with the best disposition; then the people that receive the assistance, logically, someone who is there by obligation if you do not see someone who is there for passion” (“El liderazgo y la empatía del proyecto son detonantes e identificaron muy bien las necesidades, tuvieron muy buen acercamiento y lo hacen siempre, con la mejor disposición, entonces la gente que recibe la ayuda lógicamente alguien que está ahí por obligación si no ve a alguien que está ahí por passion”). The management team has negotiation skills which when added to Save the Children’s positive reputation allows them to achieve high levels of coordination and the establishment of networks and associations with government, the private sector, and cooperation agencies and enables them to involve key local stakeholders. The entire project team is notably committed to the achievement of project objectives. The field team is fluent in the local language, and thus maintains good communication with the population. Another strong point is the ability the team has for developing methods, tools, and contextualized educational materials adapted to various Mayan languages. The project’s coordination with the Guatemalan government through the Ministry of Education is successful. Coordination is maintained with the Bilingual and Intercultural Education Directorate (DIGEBI), the Educational Community Strengthening Directorate (DIGEFOCE), the Education Quality Management Directorate (DIGECADE), the Education Planning Directorate (DIPLAN)j, and the Departmental Education Directorate (DIDEDUC) in El Quiché. The project coordinates with the Ministry of Health by means of the Ixil and Santa Cruz del Quiché Area Health Offices, Health Centers in the six municipalities and with the Departmental Commission of Healthy Schools. Close coordination is maintained with the Municipal Governments enabling the mayors to provide valuable contributions. For example the municipality of Cunén provides office space for the project, the Municipalities of Cotzal, Cunén, Uspantán and Sacapulas provide transportation for the foods, material for improving the roads that lead to the schools and materials for construction of storage areas. The teachers have been trained in the Literacy Boost methodology through a diploma program validated by the MOE and the University of San Carlos. The effectiveness of the training can be seen in the children’s improved reading abilities and the ability of teachers to apply modern, creative, entertaining, innovative, and contextualized techniques thus breaking traditional teaching paradigms. Another relevant factor is the ability of the teachers to teach children to read in their mother tongue languages. This is important because the reading methodology and material in the local languages is limited; in the case of the Ixil language, its variations make the situation even more complex. With the training received, teachers’ enthusiasm has grown, they have created learning communities, they are more active in extracurricular activities that promote reading (camps, festivals, marathons, etc.), and they have improved relationships with parents. Furthermore, the innovative activities that the teachers use with their students permit more involvement with the PTAs. y have continuously used education methods with the teachers such as to communicate with them. 71 i. Perspective of local and government stakeholders Fathers and mothers were asked how efficient the project was to them. One father (Nebaj) explained that they came to an agreement in the community about how to implement the project; at first the mothers were enthusiastic but then they got tired of preparing the meals. The solution was that every family would contribute Q.40.00 yearly to pay three mothers to do the cooking. Another father (Nebaj) mentioned that committees were organized for each component of the project to be in charge of the activities, but now they are not following through with what they are supposed to do because they want to receive incentives. One teacher commented that there are parents who do not want to participate in the project due to the commitment it requires. These parents send their children to urban schools, which (in the opinion of this teacher) seems very unfair to the children. The issue of incentives is something that should be looked into in more depth in each of the communities; it could be contributing to certain difficulties in implementation of the interventions through IDEA; parents and community members sometimes expect incentives instead of collaborating without a material incentive. Teachers were asked how efficient the project was to them. They mentioned various actions that contribute to the success of the project and make it more efficient, or may affect the efficiency in some way: ● When extra resources are needed, at a community level they find ways to generate these resources. ● After the training, sometimes the teachers forget what they learned. The pedagogical specialists from the IDEA project accompany the teachers in the classroom and help reinforce what they have learned. ● In the school there is a committee for pedagogical orientation that is in charge of creating the assessments for each grade and to support teachers. Now it is the pedagogical specialists of the IDEA project that provide this support to teachers in the classroom and they help them overcome difficulties and reinforce good pedagogical practices. It is important to know to what degree the committees that were already created for this are being strengthened and not replaced. The director of DIGEFOCE recognized that it is the government’s responsibility to provide food to the children, but that the IDEA Project is helping to complement that in the short-term. His vision is that SC should help the government (DIFOCE) strengthen the government school feeding program in the long-term. ii. Quantity, use, and benefits of government and private-sector resources: Perspectives of donor, implementers, and government actors Municipal representatives were asked how efficient the project was to them. The mayor of Sacapulas expressed that the training for the teachers and the infrastructure improvement is important, but also mentioned that he had some reserves about the school feeding program because it was unclear if families were saving any money due to it: “The result is acceptable; if I could rate it from one to ten, I would give it an 8, from what I have seen in my community. Today, through school feeding and infrastructure improvement, children are stimulated to attend school and I´m sure this helps their learning process. Now, it´s difficult to measure if families save up money from this or not when children do not eat breakfast at home. 72 Because in my case, the same amount of food is prepared despite the fact that they go eat breakfast at school, but I`m sure there are families that have been able to save up” (Mayor, Sacapulas). (“El resultado es aceptable; si se le pudiera poner una puntuación de 1 a 10 le pondría 8, hablo por lo que he visto en mi comunidad. Hoy a través de los alimentos y mejoramiento de la infraestructura los niños son estimulados para asistir a la escuela y seguro que esto les ayuda en su aprendizaje. Ahora es difícil medir si las familias ahorran o no, cuando los niños no desayunan en sus casas, porque en mi caso siempre se prepara la misma cantidad de alimentos como que si los niños fueran a desayunar, pero seguramente habrá familias que si ahorran un poco)”. In the Quiché Department, it was recognized that the project was presented by Dr. Lucrecia Méndez and that there was a series of meetings to determine the where the support would be the most important and how to best implement the actions. There was no imposition, rather agreements on all levels: “Dr. Lucrecia Mendez introduced herself, and presented a technical proposal of the project in English. Then, there were a series of meetings at the management level, an analysis of the proposal; a series of weaknesses were identified that could be presented in the implementation process, and depending on this we began working. From IDEA management there were also contracts made with the Minister of Education, and additional work on those levels as well. It has been hard work, but it has been done in coordination with the different government entities, contrary to other organizations that work only at the central level to seek approval and begin to work. This is how it was at the beginning of the IDEA program: a lot of communication, many reunions, strong consensus, socialization, and travelling to the field to sustain meetings with CTA personnel to empower them; it was not an imposition on anyone” (Departmental Office, Quiché). (“Nosotros estamos muy agradecidos con el programa IDEA no por nosotros mismos si no por los niños de las escuelas, y por eso hacemos esfuerzos. La Doctora Lucrecia Méndez se presentó y lo hizo una presentación técnica en inglés la propuesta del proyecto, luego siguieron una serie de reuniones a nivel de dirección, se hizo un análisis de la propuesta y se identificaron una serie de debilidades que se nos podían presentar en el proceso de implementación, en función de esto comenzamos a trabajar; de parte de la dirección de IDEA también hubo contactos con la señora Ministra de Educación, a esos niveles se trabajó, ha sido un trabajo arduo pero con coordinaciones en las diferentes instancias del gobierno, contrario a otras organizaciones que parten únicamente de la suscripción del aval central y comienzan a trabajar. Así fue el inicio del programa IDEA mucha comunicación, muchas reuniones, muchos consensos, mucha socialización, mucho de ir al campo a sostener reuniones con los señores CTA para empoderarlos, no fue una imposición”). C. Efficacy in Achieving the Objectives Teachers were asked how effective the project was to them. They highlighted changes they believed to be very significant, such as students being happier at school, more motivated, and regularly not missing attending school. They also pointed to the effectiveness of SC’s efforts of raising awareness with parents about the importance of sending their children to school. As they stated: “Children attend class constantly; today, they do not want to miss one single day. They miss the food, and since today they learn a lot better and are not afraid of being at school; before they got in trouble often because they did not pay attention, but today even in that aspect the nutritious and healthy food provided has helped them” (teacher, from Nebaj). (“Los niños vienen constantemente en las clases, hoy no quieren faltar ni un solo día, les hace falta la comida y como hoy aprenden mejor ya no les da miedo estar en la escuela, antes los regañaban mucho porque no ponían atención, pero hoy hasta en eso les ha ayudado la comida que comen, es muy nutritiva y saludable)”. 73 “Children miss class sometimes because they are sick, they do not assist class. Contrary to this, they do not want to miss, today they enjoy going to school, they say if they don´t go to school, they won´t eat that day. Additionally, they say if they miss school they will get delayed in their school work” (teacher, from Nebaj). (“A veces faltan porque los niños están enfermos, ya no vienen a la escuela; de lo contrario no quieren faltar, hoy les gusta ir mucho a la escuela, ellos dicen que si no van a la escuela no van a comer igual ese día, además dicen que si no van a la escuela se van a quedar atrás en sus clases)”. “In our days, boys and girls go to school equally, we do not see this, of only boys attending. Today, it´s all the same, Save the Children has raised our consciousness about the importance of all attending equally and we have followed their advice because it’s for our children´s good. Also, the meal they receive in school is complete and there are many games. So, our children do not want to be absent, they used to miss a lot because they were lacking the motivation that they have today” (teacher, from Chajul). (“En nuestros días todos los niños y las niñas van a la escuela por igual, ya no se da aquello de que solo niños, hoy todos por igual, Save the Children nos ha hecho mucha conciencia de que debemos mandar a todos por igual y nosotros hemos seguido los concejos porque es por el bien de nuestros hijos; además la comida que dan en la escuela es completa y hay muchos juegos, entonces hoy nuestros niños no quieren faltar, antes faltaban mucho porque no había mucha motivación como hoy)”. “Well thank God children always attend school; we always advise them and make them see that studying can be very benefiting for their life, to which Miss Petrona adds: my children only miss class when they are sick, but this is rarely and in this case sometimes the teachers even go visit them at home (teacher, form Uspantán). (“Pues gracias a Dios los niños siempre asisten a la escuela, nosotros siempre los aconsejamos y les hacemos ver que estudiar les puede ser de mucho beneficio en su vida, a lo que doña Petrona agrega: mis hijos solo faltan a clases cuando están enfermos, pero eso es de vez en cuando y a veces los maestros llegan a visitarlo)”. “In general, children attend every day with exception of those who only participate when there is thanksgiving and when someone has passed away. Contrary, they attend thank God they all attend. With respect to the differences, parents are conscious that boys and girls have the same rights and equalities” (teacher, from Cunen). (“Por lo regular los niños asisten todos los días a excepción de otros solo cuando hay acción de gracias participan y cuando hoy muertos de lo contrario todos asisten gracias a dios que aquí todos con relación a las diferencias, los padres de familia ya están conscientes que los niños y niñas tienen los mismos derechos e igualdades)”. “To date, we have not faced this problem because children attend class always, all equally. Today, thank God they all attend; we are pending that our boys and our girls go to school, they both have the same rights” (teacher, from Cotzal). (“Hasta la fecha no se ha encontrado con ese problema porque los niños asisten siempre a clases todos por igual, hoy gracias a Dios todos van, nosotros estamos pendientes de que tanto nuestros hijos y nuestras hijas vayan, los dos tienen los mismos derechos)”. Other stakeholders also gave their opinions, emphasizing that students attend class consistently, pay more attention in class, are more motivated, learn more, and have adopted good hygiene practices. They also pointed out that parents participate more. Additionally, the formation of the school governments have been an important change. As stated: “Students staying in school is one of the main results; in real terms, when there is feeding, children attend school not because they enjoy it but because of the need to be fed. Today there is more attendance, there are more children in school, enrollment rate went up, and we have noticed that parents save up some money and can use it towards other needs. Teachers on the other hand, have less difficulty teaching, because well-fed children pay more attention and their learning is better. Similarly, we can see that children seem more motivated, with a new face, because they have adopted good hygiene practices. There has been more participation from parents in their 74 children´s reading activities; there are reading opportunities in the community and they are currently supporting the construction of dining areas for them to have an adequate place to consume their food. Another result, is the creation of school governments; it’s incredible how children express themselves, this is thanks to the IDEA program, empowering boys and girls in their school´s administration” (MINEDUC Departmental Director, Quiché). (“La permanencia es uno de los principales resultados, en términos reales cuando hay alimentos los niños asisten a la escuela no porque les guste si no porque tienen necesidad de ser alimentados. Hoy hay más asistencia, hay más niños en las escuelas, subió la matrícula, también hemos notado que ya los padres de familia ahorran un poco y lo destinan para otras necesidades, los docentes por su parte ya no tienen mayores dificultades en la enseñanza, porque un niño bien alimentado es un niño que pone atención y su aprendizaje es mejor; así mismo, podemos ver como los niños ya se ven más motivados, con un nuevo semblante, porque han adoptado las buenas prácticas de higiene, ha habido más participación de los padres de familia en las actividades de lectura de sus hijos hay espacios de lectura en la comunidad, actualmente están apoyando también la construcción de comedores para que tengan un lugar adecuado para comer sus alimentos. Otro resultado es la formación a los gobiernos escolares, es increíble como los niños se expresan, y eso se le debe reconocer al programa IDEA el empoderamiento de los niños y niñas en la administración de las escuelas)”. “In this case, teachers´ motivation, reading comprehension improvement, and learning to put to practice what they learn; this last one, I find very significant. Behavior change is a little difficult, but with practice, this program has achieved it” (USAC, Quiché) (“En este caso, la motivación de los docentes, la mejora de su comprensión lectora y aprender a poner en práctica lo que aprenden, esto último me parece muy significativo. Cambiar la conducta que es un poco difícil pero que con la practica en este programa se ha podido)”. “En algunos casos si se marcó la diferencia por el hecho de que un niño de sexto ayudara a un niño de primero, entonces ellos hacían lo que les correspondía y luego el apoyo. Dicen que cuando uno ayuda a una persona uno mismo está multiplicando lo que uno ya sabe, en el caso de los voluntarios también, porque vuelven a retomar el aprendizaje y hay niños que si tal vez no al 100 pero se notaban mucho más motivados para aprender. Al menos yo vi algunos casos cuando llegaba a las comunidades y miraba a los niños leyendo otros libros, entonces si se ve la diferencia. Los voluntarios son jóvenes y padres de familia de la comunidad que llegaban en la tarde a ayudar, los técnicos los reunían y capacitaban para que pudieran apoyar, se les daba una guía de como trabajar, actividades previas durante y después (Técnicos DIFOCE, Quiché)”. In sum, concrete results were recognized such as the improved school attendance, improved teaching strategies and engagement of the students, improved student learning, formation of school governments, volunteers such as young people and mothers who help out in class/school, children reading books, and the motivation of parents to collaborate. i. Monitoring and evaluation system (M&E) The project has a monitoring and evaluation system based on the results framework, in order to record, measure and analyze the process and results indicators. Through it, all of the activities that are carried out and the total number of beneficiaries per activity are recorded. As stated by at the University of San Carlos (Quiché): “We as FOCE we do not provide assistance, but the IDEA program checks the attendance by means of the portions of food that are handed out on each day”. (“Nosotros como FOCE no nos encargamos de la asistencia, pero el programa IDEA si verifica el control de asistencia por medio de las raciones de comida que se van repartiendo en el día.)” 75 Using this data various reports are obtained in a timely manner and these are disaggregated by gender, by component, by school, and by geographic area. Based on the results, the project staff, specialists, and coordinators make decisions focused on achieving the goals and objectives. The monitoring is done by means of electronic controls backed up by physical evidence. The team is very competent at data management and thus the information that is processed is of high quality, reliable, confidential, punctual and timely. At this time the project is ready to use a virtual platform for monitoring and generating reports. The use of this technology will be helpful for the monitoring system given that the database will provide reports in real time, more rapidly, and with better connection between the different variables. The director of DIFOCE highlighted the importance of evaluating interventions, and that this should be done from the beginning. He mentioned that Save the Children has always taken the Ministry of Education into account at each stage of evaluation, even in the planning phase. X. Sustainability and Scalability The IDEA Project has a sustainability strategy that is systematized in the “Manual for preparing a sustainability strategy”. This was created with participation of all the main stakeholders who were consulted in January and February 2016. The manual includes the concept of sustainability as well as the steps for the community to take to plan the sustainability of their projects. This manual has clear and straightforward language and can be used by communities without technical support. The project promotes sustainability as a cross cutting theme and to date has carried out various actions in the communities and schools as well as with municipal governments and authorities from the Ministries of Education and Health in the department of El Quiché. Assemblies have been held in the communities to provide information about the project’s purpose, and letters of commitment were signed with the communities. Leaders have been trained for coordination with other non-governmental and private entities. They were given the “Manual for preparing a sustainability strategy.” Community self-management is promoted. Families are already involved in the construction and rehabilitation of school infrastructure. Volunteerism is promoted in the communities to support reading and writing. Parents’, leaders’ and teachers’ knowledge and abilities with regard to maintenance and preventive and corrective use of school sanitation infrastructure were strengthened. A guide was created to support this. Greater awareness in parents of the benefits of education can help make the actions promoted by the project sustainable for families and communities. The project involves various officials from the municipal governments in the implementation of education, health and nutrition actions as well as in the construction of the school sanitation infrastructure. The project prepared a “Guide for use and maintenance of infrastructure,” which is to teach water and sanitation infrastructure maintenance, preventive maintenance of school infrastructure, and correct use and maintenance of the infrastructure for food preparation and storage. This guide has been given to the Ministry of Education, to principals and teachers of the target schools, and to leaders and parents. This guide, if used, will help to ensure the sustainability of the improvements to school infrastructure. The Municipal governments hire teachers and personnel for food preparation in some schools and also contribute with transportation of food and materials to the schools. For the sustainability of the IDEA project’s interventions, it will be important that communities are proactive in seeking out this support over time. 76 A key action in seeking sustainability is the increase in teachers’ abilities for teaching reading and writing using engaging and innovative methodologies. The teachers who have completed the training have been certified in the Literacy Boost methodology, with the endorsement of the University of San Carlos of Guatemala through the University Center in El Quiché. This certification increases their credits for the teaching pay scale. The project has influenced and advised the Departmental Directorate of Education for the legalization of land in the name of the State to thus be able to build or improve schools, especially in regards to sanitation infrastructure (toilets and hand washing stations), kitchens, firewood saving stoves, and storage areas. The project coordinates with the Healthy Schools Commission for the implementation of actions that ensure healthy and fitting environments for students. Achieving sustainability is not easy. Some factors that limit this are the limited vision of sustainability held by the key stakeholders in the community, authorities, teachers, leaders, and parents. The communities lack plans for local development. They cannot envision their future and thus are always waiting for outside help. Their organizational structures are weak with limited management capacity and a hard-to-overcome paternalistic focus. The parents and community leaders have not identified other stakeholders with whom they could coordinate so that the actions of the IDEA project can continue and they also lack management and negotiation abilities. For the mothers’ group, sustainability is less visible given that the women have limited local decision making participation and their schooling level is very low. The Municipal governments lack development plans and have limited economic and human resources so investment in education is limited. Additionally, the municipal authorities have limited abilities for seeking resources. MOE investment is limited and is dependent on the policies of whichever government is in office. A. Execution of the Sustainability Plan i. Actions of local actors in different aspects of the project Stakeholders were asked about the project’s sustainability and the actions to provide continuity. At the school level, there were not responses that reflected a sustainability plan for the food provided by Save the Children. Most of the responses were about finding other institutions that could continue to support the school feeding program: “If the program is discontinued, what we would need is for another institution to help us as Save the Children has done, because otherwise there is no way for us to get food and here in the Uspantán municipality, we do not know of another institution that helps with school feeding. What the government provides is too little and without a doubt the children will feel the difference, it would not be the same and the consequences that would follow for children not having what to eat we hope will not be grave” (school principal, from Uspantán). (“Si el programa deja de ayudar lo que necesitaríamos es que haya otra institución que nos ayude como lo ha hecho Save the Children porque de otra forma no hay como conseguir los alimentos y aquí en el municipio de Uspantán no conoceos de otra institución que ayude con alimentos escolares, lo que da el gobierno es muy poco y sin duda alguna los niños lo van a sentir, ya no sería lo mismo y las consecuencias que traiga a que los niños ya no tengan de que comer esperemos que no sean graves)”. “With support from the principal´s office, teachers and parents would find an institution or organization to support us with school feeding for children. Now, if there was not another, then we 77 would just stay with what the government gives us, but we hope another program shows up to help us because the school has benefitted from the APEDIBIMI program, and when it ended the follow year IDEA came, so we have not felt lacking anything thanks to God, and we hope that if IDEA leaves this year, there is another program in the following years (school principal, Trapichitos). (“Pues con el apoyo de la dirección, maestros y padres de familia buscaríamos una institución u organización que nos apoye con la alimentación de los niños ahora si no lo hubiera pues solo nos quedaríamos con lo que el gobierno nos da, es más esperamos que venga otro programa que nos ayude porque la escuela estaba beneficiada con el programa de APEDIBIMI cuando terminó al año siguiente vine IDEA entonces no nos hemos quedado sin nada Gracias a Dios y esperamos que si IDEA se va este año haya otro programa en los siguientes años)”. “We do not want to imagine that Save the Children will leave the school one day. The children would be the first to miss it all; I´m not saying it would all go back to how it used to be, but this requires a joint effort from the education community. Now, we have not identified another similar organization with Save the Children`s focus, I see it very difficult. The interventions of Save the Children are very complete, I think we have to begin a dialogue with the school community to analyze this situation” (school principal, from Cunén). (“No queremos imaginarnos que Save the Children un día se va a ir de la escuela, los niños serían los primeros en extrañarlo todo, no quiero decir que todo va a volver a ser como antes, pero esto va a necesitar de un esfuerzo conjunto de la comunidad educativa. Ahora mismo no tenemos identificada a otra organización similar al enfoque de Save the Children, lo veo muy difícil, las intervenciones de Save son completas, creo que debemos de iniciar un diálogo con toda la comunidad educativa para analizar esta situación).” “At school, children receive breakfast which is prepared with the food provided by the IDEA program, with the complementary products from MINEDUC. After this, during recess they also receive a snack that is prepared with the food purchased with MINEDUC funds, which means children are very well fed” (school principal, from Cunén). (“En la escuela los niños reciben un desayuno que se prepara con los alimentos que nos da el programa IDEA, con el complemento que da el MINEDUC, posterior a eso en el recreo reciben una refacción que se prepara con alimentos que se compran con fondos del MINEDUC, quiere decir que los niños están bien alimentados)”. Also, in regards to the education interventions, it was recognized by the University of San Carlos (Quiché) that there is a lack of resources on their part to continue the intervention: “We do not get involved in that directly because as I said, what they ask from us is the academic endorsement from the university, to give it the proper backup in that sense, but we also worry because the programs do comply in that sense. As a university, we do not have our own resources to promote these type of degrees, which is why we hope for other institutions to approach us with new proposals” (USAC, Quiché). (“Nosotros no nos involucramos en eso directamente porque como le digo a nosotros lo que nos piden son los avales académicos de la universidad, para darle respaldo en ese sentido, pero también nos preocupamos por que los programas también cumplan en ese sentido. Nosotros como universidad no tenemos recursos propios para poder promover este tipo de diplomados, por lo que esperamos a que las otras instituciones se acerquen con nuevas propuestas”.) However, at the government level, there was more optimism regarding continuing to sustain innovative teaching practices by uniting efforts between the different stakeholders/actors: “We are becoming informed on how we can continue with what we have learned from the program, when it leaves. Not only in the schools where they participated but also in the schools where they didn´t; in this, I mean the teachers´ practices inside the classroom. What we did was develop a meeting with FOCE, CTAs, Health personnel and the Municipality, for us to all get together and commit, for one way or another continue to work, and other organizations are also committing with the ministry of health and the Municipality. Maybe in terms of materials or feeding we cannot do 78 much because we do not have funding, but we can achieve something with the trainings, by teaching other teachers at schools where the project did not participate” (DIGEOCE technician, Quiché). (“Nosotros estamos informándonos de como poder seguir con lo aprendido por el programa cuando se vaya, no solo en las escuelas que participaron si no también en las escuelas que no participaron, esto estamos hablando de las practicas del docente dentro del aula. Lo que hicimos fue hacer una reunión, estaba FOCE CTA Salud y Municipalidad, para que nos unamos todos y adquiramos compromisos para que de alguna manera se pueda seguir trabajando, y ahí se están comprometiendo otras organizaciones como el ministerio de salud y la municipalidad. Tal vez en el tema de materiales o alimentación no podamos hacer mucho porque no tenemos los fondos, pero si podemos lograr hacer algo con lo que ya se fue capacitado para enseñarlo a los docentes de las escuelas en donde no participó el proyecto)”. With regard to the participation of the municipality and a strategy for sustainability, the mayor of Nebaj stated that they had not yet elaborated a sustainability strategy with the staff of the IDEA project. However, he expressed that the parents and teachers have learned from the workshops and can maintain their practices. In terms of food, he recommended the idea of school gardens, which had been done in the past: “With the implementation that has occurred in more than 50 establishments, the purpose is to integrate all development programs. In terms of sustainability, we have not reached the level of developing a strategy together for follow-up of the IDEA program. However, I believe that the program has empowered families and teachers with trainings, and we hope that what they have learned will be continued to put into practice. As far as feeding, school gardens can be made; we used to do this before as teachers, we grew radishes, beets… I´m talking about the year 1968; today teachers must bring back these ideas. It´s an investment that is not immediately seen, but long-term the product is visible. Shared learning will be cultivated when children are older” (Mayor, Nebaj). (“Con la implementación que se ha dado a más de 50 establecimientos, tiene como finalidad integrar los programas para el desarrollo. En cuanto a la sostenibilidad no hemos llegado a ese nivel de que elaborar juntos una estrategia de seguimiento del programa IDEA; sin embargo, yo creo que el programa ha empoderado a las familias y maestros con capacitaciones, esperamos que lo aprendido lo sigan poniendo en práctica, ahora en cuanto a los alimentos, se pueden hacer huertos escolares, antes lo hacíamos como maestros, sembrábamos rábanos, remolachas, hablo del año 1968, hoy los maestros deben retomar esas ideas; es una inversión que no se ve pero a la larga se verá el producto. El aprendizaje compartido se cosechará los niños cuando sean grandes).” The mayor of San Juan Cotzal expressed that a weakness of the IDEA program was that it did not have a sustainability plan from it’s initiation, which he believes all programs should have: “I think that all the programs must come with a sustainability plan; I think this has been a weakness in the IDEA program, which was not contemplated from the beginning. One example could be that we installed a water office in the municipality for promotion, because water consumed by inhabitants is not safe. We have done lab tests and have detected that water that has been consumed contains animal feces. Because of this also, we have opted to circulate water sources to assure animals do not enter or people with bad intentions” (Mayor, San Juan Cotzal). (“Creo que todos los programas deben venir acompañados de un plan de sostenibilidad, esta creo Yo ha sido una debilidad del programa IDEA que no lo contempló desde su inicio. Un ejemplo podría ser de que nosotros instalamos una oficina de agua en la municipalidad para velar porque el agua que consumen los habitantes sea segura, hemos hecho exámenes de laboratorio y se ha detectado de que el agua que han venido consumiendo contiene estiércol de animales; por eso mismo también hemos optado por circular las fuentes de agua para asegurar que no entren animales o personas mal intencionadas”. The mayor of Uspantán stated that communities that did not receive the support of SC asked for the support of the municipal government. While the municipal government cannot provide the type of support that SC 79 provides in terms of food and education support, they have been building sinks, improving the kitchens, and repairing and building bathrooms: “One initiative that we have had to make in terms of making the IDEA program sustainable in the future is creating a municipal public policy for education quality. This is our own personal initiative, not produced from dialogue with Save the Children; this policy will be useful to make political incidence at the governmental level and most importantly if we achieve it, we will institutionalize it for all municipal governments to follow through with it, as it was done with the PDM or municipal development plan which has been institutionalized. All future negotiations must comply with it, and same would be for this public policy quality education” (Mayor, Sacapulas). (“Una iniciativa que hemos tenido para hacer sostenible las acciones del programa IDEA en el futuro es la creación de la política pública municipal de calidad educativa; esta es iniciativa propia, en ningún momento ha sido producto de un diálogo con Save the Children; esta política nos servirá para hacer incidencia política a nivel de gobierno, y lo mejor es que si la logramos la vamos a institucionalizar para que todo gobierno municipal pueda darle seguimiento, al igual como se hizo con el PDM o plan de desarrollo municipal que se ha institucionalizado todas las gestiones futuras deberán cumplirlo, igual pasaría con la política pública de calidad educativa”.) Another mayor (Sacapulas) pointed out the importance of creating a municipal public policy for education quality that would influence the government to support education quality improvement: “In the beginning, Save the Children worked on its own with communities. It wasn´t until some problems came up with communities not accepting support, that they came to ask for support from the municipality to solvent their inconveniences. The municipality was available to provide them with the necessary support because we knew it would benefit the children, but at the beginning the communities did not see it like this. As time passed, communities and community authorities have become satisfied with Save the Children´s work and more interestingly, the product of this team work has improved education in the municipality” (Mayor, Sacapulas). (“Al principio Save the Children trabajó por su cuenta en las comunidades, fue hasta que surgieron algunos problemas con comunidades que no aceptaban el apoyo, cuando ellos vinieron a pedir apoyo a la municipalidad para solventar esos inconvenientes; la municipalidad estuvo presta para brindarles todo el apoyo necesario porque sabíamos que iba a ser para el beneficio de los niños, pero al principio no lo miraban así las comunidades; conforme el tiempo ha pasado las comunidades y autoridades comunitarias están muy satisfechas con el trabajo de Save the Children y lo más interesante es que producto de ese trabajo en equipo se ha mejorado la educación en el municipio”.) To continue supporting the quality of education, it was recommended by a member of DIFOCE (Quiché) that the MINEDUC’s program SINAE (Sistema Nacional de Acompañamiento Educativo) could continue providing this support: “What we have been doing is that since now there are schools that have the program and schools that don´t, so we agreed with Save the Children technicians that at some point we will achieve all these ideas and trainings in schools that are not in the program. Since the ministry of Education is in a program called SINAE (National System of Educational Support) which is pedagogical support, the plan is to follow up on what is already being worked on” (DIGEFOCE technician, Quiché). (“Lo que hemos estado trabajando es que desde ahora como hay escuelas que tienen el programa y escuelas que no entonces nosotros nos pusimos de acuerdo con los técnicos Save que en algún momento se lograra dar todo ese tipo de ideas y capacitaciones a las escuelas que no están dentro del programa, entonces el ministerio de educación está en un programa que se llama SINAE (sistema Nacional de acompañamiento educativo) que es el acompañamiento pedagógico, darle seguimiento a lo que ya se está trabajando.”) In sum, the sustainability of the IDEA project’s interventions depends much on the ability of the local actors to seek support of the municipality and work with and through them to form the necessary alliances required 80 to maintain the components that are feasible to maintain. These include areas that they have received training in (e.g. hygiene, teaching methodologies, etc.) and the continued use of the guides and materials that SC has provided. B. Scalability i. Interventions that are replicable on a larger scale To date, the most successful actions that could be taken to the national scale are: a) The guide for parents used for sensitizing and including them in the educational community to seek the wellbeing of their children. This guide is a compilation of the methods that have been used and was developed in coordination with the MOE. This guide is certified by the Ministry of Education in Resolution DACPI/DIGEACE (Accreditation and Certification Directorate) No. 0068-2016 26 July 2016. b) The Literacy Boost methodology can be implemented at the municipal, departmental, and national levels by first to third grade teachers. It is a methodology that addresses the strengthening of the five reading abilities that children should acquire. It can also be used by 4th-6 th grade teachers. c) The Literacy Boost diploma program to professionalize teachers in the area of reading, both in Spanish and in local languages. Though there are challenges with sustainability at the moment, it is a model that can be replicated (with special attention given to future sustainability). When asked about replicating the program in other parts of the country, a member of DIFOCE (El Quiché) stated: “Definitely all, because we are talking about the same contexts, where we are working if you realize when they see the municipalities that have been involved, there are other municipalities that would like to be a part of the program also.“ (“Definitivamente todas, porque estamos hablando de los mismos contextos, donde estamos trabajando si usted se da cuenta cuando ven los municipios que se han involucrado hay otros municipios que quisieran ser parte también del programa.”) The Vice Minister of Education stated that all of the directorates of the MINEDUC were given the responsibility to put into practice all of interventions that have had a positive impact. They are supposed to identify best practices, document them, and adapt them to the different contexts. According to him, SC has contributed a lot with a program called “quality circles” that they carried out in various regions of the country. For this they put together different groups of teachers, parents, and students and they identified challenges and defined goals. He gave other examples of collaborative efforts with SC that were also successful. It is important to provide follow up and help ensure that these best practices are documented in a way that the Ministry of Education can implement them in various parts of the country. In sum, there are specific components of the program that can be replicated and scaled up to other parts of the country. There appears to be enthusiasm on the part of the local and central government to be part of this effort to scale up the successful components of the country. ii. Challenges for replicating and/or scaling up interventions Some of the challenges for scaling up the interventions were identified as teachers resisting changing their teacher teaching practices, making sure materials do not get lost and/or misplaced, and other minor challenges at the beginning of the program that were resolved through dialogue and agreements. According to a professor at the University of San Carlos (Quiché), the strategies can be replicated. However, he expressed that he didn’t know what organizations, entities, or mechanisms could assure continuity of the interventions. While as a university, they have the willingness to keep providing support such as academic 81 endorsement; it would be hard for them to continue to be involved in the financial and technical aspects. As he explained: “Of course it can be replicated. In the strategies that can be defined I think, according to experience that what has been done until now is what should be continued to be reinforced or look for alternatives or instances. In our case, I will be honest, I don´t know the processes or outside research of what could give the program permanence and with what we have seen in this program so far, it has worked and we, as a university, are willing to continue supporting and giving academic endorsement, but it would be difficult for us to get involved in seeing other aspects such as academic and technical ones, our involvement in this would be weak. But we could support these processes and these alliances that are of benefit to society and mostly to next generations in this department”. (“Por supuesto que se puede replicar. En las estrategias que se pueden definir creo según la experiencia de lo que se ha hecho hasta ahora es que se debiera seguir reforzando o buscar otras alternativas con otras instancias, en el caso nuestro pues yo si un poco le soy sincero, desconozco los procesos de búsqueda hacia afuera de lo que le podría dar permanencia al programa y con lo que hemos visto en este programa hasta ahora ha funcionado y nosotros como universidad estamos dispuestos a seguir apoyando y dando el aval académico, pero si nos costaría meternos a ver otros aspectos como lo financiero y lo técnico, quedaríamos débiles en involucrarnos. Pero si apoyar estos procesos y estas alianzas que son de beneficio para la sociedad y más para las nuevas generaciones de este departamento”.) In sum, there have not been challenges that would prevent the scaling up of the program, or components of the program. However, for scaling up certain components (such as the Literacy Boost diploma) there will be a need to form alliances in the new regions/municipalities where these interventions will be implemented. XI. Conclusions Overall, the final evaluation of the IDEA project has revealed many positive results, including changes from the time of the midterm evaluation. There are a few areas in which there was little or no improvement, but the great majority of interventions have shown a positive impact. The main conclusions are detailed by area/component. Improvement of Children’s Literacy Skills Overall, the final evaluation revealed improvement in both grades and all languages from the baseline to the final evaluation, and some improvements from the midterm to final evaluation. The trend for the percentage of second graders that can read with comprehension in all three languages was an increase in “beginners” and a decrease in both “readers” and “non-readers”, especially for the local languages. Third grade also showed an increase of “beginners”, yet also an increase in “readers”. Thus, even though percentages in the “readers” category don’t show an increase, the decrease in “non-readers” for both grades, both geographical areas, and all three languages reflects a positive trend towards students developing reading comprehension. For specific literacy skills, the statistically significant improvement from the baseline to the final evaluation for reading comprehension in second grade in all three languages and both geographical areas and for Spanish and K’iche’ third grade is notable, as is the overall improvement from the baseline to the final evaluation in second grade K’iche’ and Ixil literacy abilities and third grade K’iche’ abilities. Thus, it is plausible to assume that the teaching strategies learned and implemented by the teachers may have led to these improvements, which should be highlighted as a major achievement of the project. At the same time, the decrease in performance in some second grade literacy skills from the midterm to the final evaluation, especially in area 1, is something that requires further inquiry. For the local languages, this may be due to difficulties in the local language skills of the teachers, teaching practices that emphasize the development of certain literacy skills over others, or other factors. The evaluation results provide the detailed information 82 for each literacy skill, which can be used to analyze teaching practices, materials, and other aspects that may influence learning outcomes. Also notable is the finding that “having read to someone in the community” was a covariate associated with improved literacy skills for both grades and geographical areas in all languages. Documenting this, as well as reinforcing its practice as well as the implementation of reading buddies would be beneficial. Support to Improve Reading As revealed in the qualitative data, (e.g. opinions of teachers and parents), the training has helped to improve teaching methodologies and students’ engagement with learning. Teachers commented specifically about the teaching practices shifting away from traditional approaches to more modern and innovative approaches, however, not all of the teachers have changed their teaching practices. It is unclear what is needed to get these practices to shift. SC technical staff could work with school principals to document specific teaching practices as well as the strengths and weaknesses of teaching abilities, including language abilities. The materials provided in both the L1 and L2 were considered to be instrumental in the students acquiring one or both of the languages, which is consistent with research findings on L1 and L2 acquisition. While the increase in Book Banks is positive, it is important to assure that reading practices continue to be encouraged. The decrease in area 1 of the practice of borrowing a book from someone or the Book Bank may indicate the need to reinforce this practice. Since a slight increase is reflected in book borrowing in area 2, understanding why book borrowing varies between areas could be helpful for increasing community book borrowing in area 1. Overall, there was a relatively high percentage of methodologies and tools for teaching L1 and L2 being put into practice by teachers, however some teachers pointed to not having enough materials for the activities they wished/planned to do in order to improve literacy. It is possible that this limited some teachers’ abilities to put into practice certain activities. Future interventions could contemplate a way to assure adequate supplies and make this is sustainable through alliances with the private sector and/or municipalities. One very concrete result reflected in the quantitative and qualitative data is the improvement in school enrollment and school attendance. The reason for this appears to be largely due to the school feeding program, however some local actors also mentioned it being attributed to students being more interested in school due to the more engaging activities. Increase Health and Dietary Practices Various interventions have contributed to the improved health and dietary practices. These include the school feeding program and the training on hygiene and sanitation. The PTA organization and work they carry out has been critical in accomplishing these goals, especially the food provision in schools, as have been the other organized committees and support of the municipal government. As for school infrastructure, SC and the MINEDUC have been working on improving this. However, there is still much to accomplish, especially in terms of keeping pests out of the storage spaces and kitchens. The majority of actions that demonstrate the proper handling and storage of food are being implemented by schools at a high percentage in both areas, Though causality from project interventions cannot be determined, it is likely that the training and support through the IDEA project has contributed to this. The issue of “evidence of rodents or other insect or animal” being reported at a high rate may require additional infrastructure support. 83 The separation of toilets for girls and boys is another positive change achieved through the efforts of SC and the municipalities. However, a challenge remains in regards to schools having the materials for proper hygiene such as soap. Additionally, the practice of hand washing before and after meals should be reinforced in area 2. Another positive finding is that, overall, appropriate conditions for food preparation were identified. The two aspects to improve (wearing of hairnets and other people entering the kitchen) are resolvable with some reinforcement from the SC project staff. From the beginning, the IDEA project has involved key stakeholders, including community leaders, representatives of the Community Development Councils, parents and school principals and teachers. This has been a strength of the project and stakeholders attributed the projects’ success in part to this approach. Additionally the close collaboration with the municipal governments, University of San Carlos, and the different directorates of the Ministry of Education has resulted in agreements, understandings, and policies at the ministerial level. It has also facilitated the implementation of the project, its scalability, and replication in other geographical areas. Above all, it has created some of the conditions of sustainability at the level of MINEDUC regarding school feeding policies. The increase in other health and education interventions, especially in area 1, is important to highlight. However, it is unclear what the specific interventions are and if, in all instances, they are complementary to the interventions of the IDEA project. Relevance, Efficiency, and Efficacy Overall, the FE revealed that there is excellent communication and coordination between SC and the different directorates of the MINEDUC, at the central and departmental levels. The project is aligned with the government’s goals and priorities, and it is perceived that SC will continue to contribute to the implementation of the school feeding law. Coordination with other donors working towards similar goals has also been important. Furthermore, the MOE authorities trust that with the project’s interventions in education, such as the diploma program for teachers, there will be important improvements in the key educational indicators that will orient education policies and programs in the country. Thus, the IDEA project is considered highly relevant for the current context and future goals for the Guatemalan government. Sustainability and Replicability There are multiple challenges to achieving sustainability of the IDEA projects interventions including the communities’ lack of plans for local development and the tendency to seek external support as a first solution. However, several tools prepared by SC, such as the “Guide for use and maintenance of infrastructure,” if used, can contribute to improved water and sanitation infrastructure maintenance, preventive maintenance of school infrastructure, and the correct use and maintenance of the infrastructure for food preparation and storage. One step towards sustainability in improved teachers’ abilities for teaching reading and writing using entertaining and innovative methodologies is the certification process endorsed by the University of San Carlos of Guatemala through the University Center in El Quiché. It is unclear if this can be replicated or sustained, but the coordination with SINAE may be one way to do this. In sum, the sustainability of the IDEA project’s interventions depends much on the ability of the local actors to seek support of the municipality and work with and through them to form the necessary alliances required to maintain the components that are feasible to maintain. These include areas that they have received training in (e.g. hygiene, teaching methodologies, etc.) and the continued use of the guides and materials that SC has provided. 84 As discussed in the evaluation report, there are several components of the IDEA project that can be replicated and taken to scale: 1) the guide for parents used for sensitizing and including them in the educational community, 2) the Literacy Boost methodology that can be implemented at the municipal, departmental and national levels by first to third grade teachers (and can also be used for 4th-6 th grade teachers), and 3) the Literacy Boost diploma program to professionalize teachers in the area of reading, both in Spanish and in local languages. The enthusiasm of the local and central government to scale up and/or replicate such efforts in different parts of the country increase the feasibility of doing so. XII. Recommendations Improvement of Children’s Literacy Skills ● Documenting and highlighting the improvement in the reading comprehension literacy skill across grades, languages, and geographical areas will be helpful for diseminating best practices as well as for replicting and/or scaling up. ● It is important to understand why reading comprehension showed a statistically significant change while other literacy abilities did not; analyzing the teacher training methodology and materials as well as their actual classroom practices could be a means of gaining such insight. It may be that more emphasis should be placed on the other components of literacy development, in addition to encouraging teachers to continue their efforts with reading comprehension (recognizing that this has been the skill that has shown the most growth). ● It is important to share the results of the evaluation with the principals and teachers and discuss with them the specific findings on changes in the different literacy skills. This can inform their teaching practices as well as help gain insight into the reasons for the results (both in the case of improvement or lack of improvement). ● The literacy skill results point to the need for further inquiry to understand why more literacy abilities showed a positive trend across time in area 2 than in area 1 in order to implement effective interventions from area 2 in area 1 and have targeted interventions to improve specific literacy skills. ● Covariates that showed a positive correlation across readings skills measured, as in the case of “having read to someone in the community”, should be further examined, documented as a best practice, and reinforced. ● The covariates where no statistical significance was found for any model (e.g. "Books at home", "has a reading buddy”, and "percentage of people that help you study" are not consistent with what would be expected. Thus, exploring why these actions do not contribute to improved literacy results merits further inquiry. ● Documenting and further exploring the relationship between the positive changes for second grade K’iche’ and Ixil second grade literacy and the teaching approaches provided through the IDEA program’s trainings (as well as the degree they have been put into practice) would be beneficial for replicating and disseminating best practices. ● For cases in which literacy skills reflected a decrease from the midterm to final evaluation, the causes for this decrease should be further explored. ● Further inquiry should be done on the causes for the difference in comprehension between boys and girls to understand why boys are performing better in this literacy skill. 85 Support to Improve Reading ● It is recommended that continual inquiry and documentation be done to better understand teachers who resist putting into practice the innovative and engaging teaching strategies provided through the IDEA project training. Meanwhile, more consciousness raising and pedagogical support in the classroom may help get teachers to shift these practices. Additionally, as one teacher recommended, paying attention to the classroom configuration so that it is more likely to engage students (such as having them sit in a “U” shape) may help reinforce more innovative teaching strategies. ● Organizing an exchange between teachers could be beneficial, especially given the difference in literacy skill improvement between area 1 and 2 (area 2 showing more improvement). For example, groups of teachers from area 1 could visit area 2 and vice-versa. If teachers observe other ways of teaching and discuss teaching approaches, it may help lead to shifts in paradigms and teaching practices. ● It is common for teachers to lack literacy skills in the local language they are supposed to teach, which may be why improvement wasn’t shown for more of the third grade K’iche’ literacy skills. SC technical staff could document which teachers are able to speak, read, and write in the local language (L1) and to what degree. Teachers who lack locoal langauge liteeracy abilities could be provided support such as access to a diploma/courses and learning materials in the local language. ● It important to have a follow up mechanism in place for helping ensure that all teachers are taking advantage of the trainings and putting into practice what they learn so that the teaching and learning is consistent with innovative approaches offered through the IDEA project. A tool (such as an observation protocol) can be developed and a system put in place for SC staff to routinely document this during their field visits. ● Reading buddies should be reinforced in the schools. There are often students who are much more advanced in their literacy skills than other students. These more advanced students can be paired with the less advanced ones. Teachers should receive guidance on how to do this and be encouraged to put it into practice. ● It is important to assure that reading practices continue to be encouraged. Since a slight increase is reflected in book borrowing in area 2, understanding why book borrowing varies between areas could be helpful for increasing community book borrowing in area 1. ● The lack of sufficient materials/supplies for teachers to put into practice some activities for literacy skill improvement may have limited their ability to do so in some cases (or with some activities). Future interventions could contemplate a way to assure adequate supplies and make this is sustainable through alliances with the private sector and/or municipalities. Increase Health and Dietary practices ● Though SC and the MINEDUC have been working on improving school infrastructure and some improvements are still needed. The most crucial problem to resolve is that of rodents and other pests (e.g. cockroaches, rodents, etc.) entering the storage spaces and kitchen. It is a situation that must be ameliorated due to health concerns, taking into account the rural context. ● A substantial percentage of schools in both areas (roughly half in area two and one third in area 1) do not have some materials and practices that are crucial for proper hygiene such as soap. It is critical that this be resolved due to the importance of washing hands with soap to prevent sickness and infection. 86 ● As a mayor commented, the IDEA project has helped tremendously through the food provided to the children, which allows them to concentrate and have energy in school, but that there are also factors that influence children’s attendance and learning in school such as family disintegration, child abuse, and social issues/problems (e.g. bullying, gender-based violence). If there is a continuation of a project, or a replication of the interventions, it is recommended that the trainings (for teachers, principals, and families) also work these themes in to address such issues. SC should seek an alliance with an organization that specializes in such issues so that teachers and principals learn how to address such issues and help ensure a healthy learning environment. ● The FE revealed that there is support from the municipalities and from Save the Children to help ensure that the communities have safe drinking water. However, it is important to ensure that the techniques that are taught are consistent with practices that the people will put into practice (such as boiling water) so that resources are used as effectively as possible. ● While overall, appropriate conditions for food preparation were identified, there are two aspects to improve (wearing of hairnets and other people entering the kitchen). These practices should be reinforced from the SC project staff. Relevance, Efficiency, and Efficacy ● While the project is highly relevant for the needs and priorities of the Guatemalan government, working in other regions (like the Eastern region) that receive less donor attention is worth considering, in addition to assuring that there is installed capacity where the projects have been operating. ● As recommended by the Vice Minister of Education, there are specific best practices from the project that should replicated in other areas of the country and adapted so to the cultural richness that exists and varies across Guatemala. ● The issue of incentives for parents and community members to participate in the project is something that should be looked into in more depth in each of the communities; it could be contributing to certain difficulties in implementation of SC’s interventions because parents and community members begin to expect a material incentive for their contribution to the project’s activities. ● In the schools there are committees for pedagogical orientation that are in charge of creating the assessments for each grade and supporting teachers. Now it is the pedagogical specialists of the IDEA project that provide this support to teachers in the classroom; they help them overcome difficulties and reinforce good pedagogical practices. It is important to know to what degree the committees that were already created for this are being strengthened and not replaced. ● It should be verified that the health and education interventions of other organizations in the areas where the IDEA project operates (and/or will expand to) are complementary and not duplicating efforts, as well as assuring that the approaches are coherent with each other rather than offering contradictory approaches (and/or requiring too much/contradictory community support). Furthermore, it appears that the municipalities are organized to coordinate these efforts; they can provide detailed information about the interventions. It is recommended that Save the Children have a database/spreadsheet for each municipality and community that details the support provided by the municipality and other organizations so that efforts can be maximized. ● Save the Children should improve the monitoring of student attendance by working with the teachers and principals, such as providing them with an effective and feasible way of routinely registering attendance. Another way would be to disaggregate the data on daily portions of food served to the students. 87 Sustainability and Replicability ● The tools created by SC, such as the “Guide for use and maintenance of infrastructure”, can contribute to the sustainability of interventions to improve water and sanitation infrastructure maintenance, preventive maintenance of school infrastructure, and the correct use and maintenance of the infrastructure for food preparation and storage. It is recommended that SC emphasize the importance of these guides being widely distributed and put to use. ● In line with the suggestion of the director of DIFOCE, who recognizes that the school feeding support from SC is short-term, SC should help the government strengthen their school feeding program for the long-term. ● SC should continue to work with the government on the implementation of the school feeding law as well as helping to widely disseminate information about it. ● The Municipal governments hire teachers and personnel for food preparation in some schools and also contribute with transportation of food and materials to the schools. For the sustainability of the IDEA project’s interventions, it will be important that communities are proactive in seeking out this support over time. ● To continue supporting the quality of education, efforts should be made to seek the support of SINAE (Sistema Nacional de Acompañamiento Educativo). ● It is important to ensure that best practices are documented in a way that the Ministry of Education can implement them in various parts of the country. Three specific components of the IDEA project were identified in the FE as interventions that can be replicated and/or scaled up (the guide for sensitizing parents, the Literacy Boost methodology, and the Literacy Boost diploma). In close collaboration with the MINEDUC and municipalities, a process should be initiated to determine where it is the most necessary and feasible to implement these interventions. ● In all future interventions, it is highly recommended that a clear and realistic plan for sustainability of all project components be developed in tandem with all stakeholders and that this plan be monitored and adjusted as necessary throughout the life of the project. 88 XIII. Best Practices A. Success stories 1) COMPLEMENTING FOOD AND THE IMPACT ON THE WELL-BEING OF CHILDREN The EORM school of the Rio Azul village, in the municipality of Nebaj, El Quiché, was founded in 1982, it is located in the north of the department of El Quiché, 7 kilometers from the municipal seat of Nebaj and 175 kilometers from Guatemala City. The school population consists of 95% belonging to the Ixil ethnic group and 5% from a Ladino background belonging to families that have migrated from other places. For Professor Miguel Brito, who since 1997 has been serving as director of the establishment, the accompaniment that the IDEA program has given them since 2013 has helped them to promote teamwork with the entire educational community. El Professor Miguel as we affectionately call him at school shares: "The IDEA program has taught us that the school success of children, can only be achieved if each of those who make up the educational community join forces and bring each one the best we have, it can not be otherwise. " Professor Reyna Mendoza, responsible for logistics says: At the beginning there was a separation, the inputs provided by the MINEDUC were handled very apart from those provided by the IDEA program, but that instead of strengthening and uniting us, it was affecting us and causing many problems; but also the positive thing about this was that we began a process of reflection as teachers, a process that led us to make the decision that the resources would be managed in a complementary manner so that there would be no duplication and waste. After we could have agreed as teachers, we launched the proposal to parents; I do not deny that for a moment we came to think that parents were not going to seem the idea, because at the beginning they did not accept a lot of food that the IDEA program gave us, but already agreeing with the teachers was easy to convince the parents and they saw it as positive, they realized that it was the best option and that this would allow us to optimize the available resources. I remember we told you the following: "Well, gentlemen, family members, the IDEA program is providing us with some foods such as rice, beans, flour, and oil, but we do not have the other inputs such as onions, tomatoes, salt, sugar, which are the supplement to be able to prepare a nutritious diet for their children. " On that occasion, each family was able to contribute those inputs, because apart from all the funds of the MINEDUC, they had not been disbursed to us; then parents seemed the idea, then was agreeing with them that MINEDUC s funds were to be used for the purchase of these inputs that they were giving us, and until then so far used. For the logistics commission led by Mrs. Reina Mendoza, this achievement was only the first step, but the other was how the mothers in charge of the preparation would see it, since it required a change of attitude on their part; however, I must remind the IDEA program of all its contribution in the subject of training for 89 mothers; then we began to dialogue with the eight mother guides, was the next group to convince, and each of them had a group of mothers in charge of cooking, was the strategy of replicating the training provided by the IDEA program. They had already been aware of the training received, where they had learned how to make a variety of recipes with food; And what was our surprise, the mothers gave us a very good response, that personally as the person in charge of the logistics commission, it filled me with joy. Miss Reina expresses with great emotion: " The administration and monitoring of food was difficult, even from those who came to supervise both the MINDEDUC and the IDEA program itself, who focused on seeing what each one contributed, and the new management practice required a change of all, and also required a paradigm shift in those responsible for monitoring as well. Miss Reina shares that another equally important step has been the decision taken jointly with the parents of formally hiring three mothers as responsible for preparing the snack, they receive a monthly incentive of Q. 500.00 each one, funds that are contributions from the parents, since each of them contributes Q. 80.00 per year. Senora Reina Mendoza expresses: I think it was the best time to present this new initiative to parents, we explained the advantages that this would bring. Of course this was achieved after three years, before it was the mothers who organized and coordinated. Currently children receive a nutritious breakfast, certainly has been a great impact since it has increased the participation of children, come to school with the motivation to eat well and that has helped to improve their concentration and participation in the classroom. The prof. Miguel Brito, director of the establishment shares that statistically an increase in coverage, attendance, permanence, has decreased school dropout and the successful promotion of boys and girls has increased. The EORM of Río Azul, currently receives in sufficient quantities the food provided by the IDEA program, while the MINEDUC receives a total of Q. 42,800.00, which are used for the acquisition of complementary inputs to those given by the IDEA program, and that by managing them in this way is impacting the lives of boys and girls, actions that translate into integral well-being and school success of boys and girls; That is what the professor Miguel Brito: "After 21 years of having my office address of the establishment, I can say that it is so sad what we have lived in the past in Nebaj, as one of the municipalities most affected by the armed conflict. Today I can see nuances of hope for the boys and girls, nuances that have been formed with the support of Save the Children and their intervention with the IDEA program." 2) EDUCATIONAL MANAGEMENT AS A TOOL FOR THE EDUCATIONAL IMPROVEMENT AND SCHOOL SUCCESS OF CHILDREN The Official Mixed Rural School Jornada Matutina, Aldea Chiul, is located 6 kilometers from the municipal seat of Cunén, in the department of El Quiché, Guatemala; the predominant language in its inhabitants is the K'iche '. The IDEA program of Save the Children, started in school at the end of 2015. The start of the program was through a community assembly, where technicians from Save the Children, who socialized the work plan of the program, shared the mission, vision and objectives, as well as the commitments that the community should acquire. The community authorities, teachers and council of parents very motivated after being informed and as a sign of acceptance signed a cooperation agreement, assuming the commitment to participate and promote community participation in the activities is driven by the program. Among the main benefits that has been obtained may be mentioned, graduate teachers in techniques and methods to promote reading in the classroom, for example, reading festivals, reading marathon and reading buddies, while at level community the reading camps have been implemented. Other components that have been relevant have been the provision of educational supplies and food that has n components contributed to increased school coverage and attendance and decreased dropout as a whole. Miss Yusely, director of the express establishment: "Before the program came to the community, the school seemed abandoned the traditional teaching model was, we did not have all the resources we have today thanks to the program; education is participatory, now the focus is the construction 90 of knowledge together with children, they are no longer receptors but actors are active in their own education and from the perspective of their own worldview ". The atmosphere in the school is now different, it used to be an abandoned house, it was used for multiple uses, since apart from everything it was not circulated, there the youths and adults met to play games of chance, to consume alcoholic beverages and many other things; now the school has a different face we have been concerned to dignify it as a teaching center where the new citizens of the country are trained with principles and values for life. Miss Yusely director of establishment shares "since becoming the first country or direction was built in conjunction with the faculty council teachers and parents work plan, we set dignify the teaching community; We knew that it would take a lot of effort, but we were willing to take on the challenge. "We then began to manage resources in the municipality, private initiative and individuals, to make raffles, and to transfer the vision to the parents; we knew that we were not alone because we had the support of the IDEA program, with which we have an excellent relationship based on respect and good communication. Mr. Yusely with great joy expresses: All the effort has been worth it, now we are a model school, we have decent services such as kitchen, toilets, circulation with beautiful gates, water storage tanks, and equipment was enabled with pump a deposit artisan that was abandoned at school, all thanks to the decided support of the IDEA program. I remember that my husband told me: "I am ashamed that you only walk in the street, asking for debts, you are even neglecting the family", but the truth is that we had set out to dignify education in the community, we had the support of the IDEA program and with the support of the community, all this impelled me to overcome any adversity. I knew that all the effort was for the boys and the girls. The vision that we have built with the teaching staff and council of parents and validated by the parents' assembly, has allowed us to take advantage of every resource that we have been able to receive from the IDEA program especially; and that all these resources as a whole have helped us to improve the educational indicators from the increase in school enrollment to the completion of the primary school in a successful way, that is our vision and that is where we focus our efforts and every resource we can receive. But another great benefit of the IDEA program has been the change of paradigm and attitude of the teachers, today the classrooms are covered with print materials in L1 and L2, the teaching is participatory; we see how our priorities have changed, because we no longer focus on giving contents, after content, but on building knowledge together with children, using even the least of the resources offered by the environment. For Yusely, the principal of the establishment, the presence of the IDEA program in the community has driven in a responsible and committed way a paradigm shift in how to tea XIV. Annexes Annex 1. Reliability of literacy skills subtests Literacy skills subtest Inter-rater reliability Qualification Knowledge of writing in Spanish 0.80 Excellent Knowledge of writing in K’iche’ 0.89 Excellent Knowledge of writing in Ixil 0.88 Excellent Most used words in Spanish 0.85 Excellent Most used words in K’iche’ 0.89 Excellent Most used words in Ixil 0.86 Excellent Decoding of words in Spanish 0.74 Good Fluency in Spanish 0.95 Excellent Fluency in K’iche’ 0.88 Excellent Fluency in Ixil 0.86 Excellent Precision in Spanish 0.90 Excellent Precision in K’iche’ 0.93 Excellent Precision in Ixil 0.89 Excellent Reading comprehension in Spanish 0.85 Excellent Reading comprehension in K‘iche’ 0.74 Good Reading comprehension in Ixil 0.89 Excellent 92 Annex 2: Literacy Boost Test (Spanish and K'iche’) USDA Guatemala IDEA. Instrumento de Evaluación Estudiantes Línea base-Español y K’iche I. INTRODUCIENDO AL NIÑO A LA PRUEBA E INFORMACIÓN GENERAL Evaluador/a: Fecha (DDMMAA): Municipio: Uspantán _____ Cunen ______ Sacapulas ______ Chaju _______ Cotzal _______ Nebaj _______ Nombre de la Escuela (línea de base): Nombre de la Escuela (línea intermedia): Nombre del niño (primer nombre y apellido): Grado: Segundo___ Tercero_____ Sección: (cuando haya dos o más secciones) Sección 1_____ Sección2____ Sección 3______ Identificación única del niño: Hora actual: ____________ AM/PM (Encierre uno) Rol del evaluador: ____ Habla con el niño/a y evalúa (1) ____ Observa y evalúa (2) Hola, buenos días, mi nombre es ____________________ y trabajo en ADOC. Hoy he venido a tu escuela porque quiero conocer cómo las niñas y niños aprenden a leer. También estoy interesado/a en saber sobre tu escuela y tu familia, por eso me gustaría que me ayudes. Te voy a hacer algunas preguntas sobre ti y tu familia y luego te voy a pedir que leas algunas letras, unas palabras, y un texto. Lo que hagamos hoy no tiene nada que ver con tus notas, ni cómo te llevas con tu maestro. Así que no te preocupes. Voy a anotar tu nombre y apellido, pero nadie sabrá lo que hemos conversado. Sí no quieres participar no tienes que hacerlo, puedes regresar a tu clase sin ningún problema. Pero me gustaría que me ayudaras. ¿Comenzamos? ¿El estudiante da su consentimiento? ____ No (0) _____ Sí (1) II. INFORMACIÓN SOCIOECONÓMICA Instrucciones: Lea cada pregunta exactamente como está escrita. Sí el niño no entiende, puede refrasear la pregunta un poco para el entendimiento del niño. Seleccione NR Sí el niño no sabe o no responde a la pregunta. Cuando una pregunta abierta pide dar un número, escriba sólo el número. Por ejemplo, para la pregunta: ¿Cuántos años tienes? Escriba “8”, no “8 años”. Para las preguntas cerradas marque con un X la respuesta que corresponde. 93 1) Sexo: _____ M (0) _____F (1) 2) Nombre del profesor/a: ____________________________________ 3) Nombre del padre/responsable:: ____________________________________ 4) ¿Cuántos años tienes? _____ (años) 5) ¿Estudiaste la preprimaria? _____No (0) _____ Sí (1) ______NR (999) 6) ¿Repetiste primer grado? _____No (0) _____ Sí (1) ______NR (999) 7) ¿Estás repitiendo segundo grado? _____No (0) _____ Sí (1) ______NR (999) 8) ¿Qué idioma hablas en casa? _____ Español (1) _____ K’iche (2) _____ Español y K’iche (3) _____ Otro(888) _____ NR (999) _____ Ixil (1) _____ Español e Ixil (2) 9) ¿De qué está hecho el techo de tu casa? _____ Paja (1) _____ Teja (2) _____ Lámina (3) _____ Terraza (4) _____ Ladrillo (5) _____ Otro (888) _____ No sabe/No responde (999) 10)¿Cuál es el material de las paredes de tu casa? _____ Adobe (1) _____ Madera (2) _____ Ladrillos de barro (3) _____ Block (4) _____ Otro (888) _____ No sabe/No responde (999) 11)¿Tienes en tu casa…? (Lea las opciones y seleccione todas las que apliquen) a. Televisíón _____No (0) _____ Sí (1) b. Radio _____No (0) _____ Sí (1) c. Celular _____No (0) _____ Sí (1) d. Electricidad _____No (0) _____ Sí (1) e. Refri _____No (0) _____ Sí (1) f. Bicicleta _____No (0) _____ Sí (1) g. Moto _____No (0) _____ Sí (1) h. Carro _____No (0) _____ Sí (1) 11.a)Si tiene electricidad, qué tipo de electricidad? i. Solar _____No (0) _____ Sí (1) ii. Planta _____No (0) _____ Sí (1) iii. Empresa eléctrica_____No (0) _____ Sí (1) iv. Otro _____No (0) _____ Sí (1) 12)Tu familia tiene…? (Lea las opciones y seleccione todas las que apliquen.) a. Vacas _____No (0) _____ Sí (1) b. Cerdos _____No (0) _____ Sí (1) c. Gallinas _____No (0) _____ Sí (1) d. Conejos _____No (0) _____ Sí (1) e. Burros _____No (0) _____ Sí (1) f. Caballos _____No (0) _____ Sí (1) 13)¿Tiene tu casa una cocina separada de dónde duermes? _____No (0) _____ Sí (1) 14)¿Cuántos cuartos tiene tu casa? ____1_____2_____3_____4 o más 94 15)¿Hay en tu casa…? (Lea las opciones y seleccione todas las que apliquen) a. Libros escolares _____No (0) _____ Sí (1) _____NR (999) b. La Biblia/Libros religiosos _____No (0) _____ Sí (1) _____NR (999) c. Revistas _____No (0) _____ Sí (1) _____NR (999) d. Periódicos _____No (0) _____ Sí (1) _____NR (999) e. Cuentos para niños _____No (0) _____ Sí (1) _____NR (999) f. Cuentos para colorear _____No (0) _____ Sí (1) _____NR (999) 16) ¿Ayudas en los oficios de tu casa? _____ No (0, salte a la pregunta 17) _____ Sí (1) _____ NR (999) a) Si sí, ¿cuáles oficios? (Lea las opciones y seleccione todas las que apliquen) a. Cocinar _____No (0) _____ Sí (1) b. Lavar platos _____No (0) _____ Sí (1) c. Lavar ropa _____No (0) _____ Sí (1) d. Tejer _____No (0) _____ Sí (1) e. Ir por agua _____No (0) _____ Sí (1) f. Ir por la leña _____No (0) _____ Sí (1) g. Cuidado de niños menores _____No (0) _____ Sí (1) h. Cuidado de animales _____No (0) _____ Sí (1) i. Cultivo _____No (0) _____ Sí (1) j. Otros _____No (0) _____ Sí (1) 16b) ¿Cuánto tiempo dedicas a los oficios de la casa cada día? ______Poco tiempo (1 a 3 horas) (1) _____ Ni poco ni mucho tiempo (4 a 5 horas)(2) _______Mucho tiempo (1 a 3 horas) (3) 16c) ¿Ayudas en oficios de la casa antes de ir a la escuela? _____ No (0) _____ Sí (1) 16d) ¿Ayudas en oficios de la casa después de llegar de la escuela? _____ No (0) _____ Sí (1) 16e) ¿Dejas de ir a la escuela a veces por hacer oficios de la casa? _____ No (0) _____ Sí (1) 17) a) ¿Trabajas por fuera de tu casa? _____ No (0, salte a la preg. 18) _____ Sí (1) _____ NR (999) 17b) Si sí, ¿qué tipo de trabajo realizas? a. Trabajo para otra familia _____No (0) _____ Sí (1) c. Trabajo en una tienda _____No (0) _____ Sí (1) b. Trabajo en el mercado _____No (0) _____ Sí (1) d. Otro _____No (0) _____ Sí (1) 17c) ¿Cuántos días a la semana trabajas por fuera de tu casa? _____ 1-2 días (1) _____ 3-4 días (2) _____ 5-7 días(3) 17d) ¿Dejas de ir a la escuela a veces por ir a trabajar? _____ No (0) _____ Sí (1) 18) ¿Fuera de la escuela, cuánto tiempo estudias cada día? _____ Nada (0) _____ Poco tiempo (1 a 2 horas) (1) _____Ni mucho ni poco tiempo(3 a 4 horas) (2) _____ Mucho tiempo (5 a 6 horas) (3) 95 19) a) ¿Cuándo estoy en clase yo presto atención?: (Lea todas las opciones y seleccione la que el niño escoja) (Si el niño contesta 3,2,1 o NR siga la pregunta 20) _____Atención todo el tiempo (3) _____la mayoría del tiempo(2) _____me distraigo con facilidad(1) _____siempre estoy distraído (0) _____ NR (999) 19b) ¿Cuál es la principal causa por la que te distraes en clase?: Seleccione una de las opciones si esta en la lista de lo contario escríbala Están cansadosTienen sueñoDistraídosSe despertaron temprano Otros_ _____Tengo hambre (1) _____Me siento enfermo (3) _____Me siento triste (2) _____Que el profesor habla mucho y no entiendo (0) _____Quiero jugar con mis amiguitos (5) _____Estoy cansado (6) _____Tengo sueño(7) _____Me desperté temprano (8) _____Otro (888) Especifique ___________________________________ _____ NR (999) 20) a) ¿Desayunas antes de venir a la escuela? _____ No (0, salte a la preg. 21) _____ Sí (1) _____ NR(999) 21) Si, si ¿Con que frecuencia? Siempre (5 veces)_____ A veces (3 o 4) _____Muy pocas veces (1 vez) 22) Que comida te dan en la escuela? Desayuno(1)____ Refacción(2)____ Almuerzo(3)______ 22a) ¿Te gusta la comida de la escuela? ____ No (0) _____ Sí (1) 23)¿En la escuela yo? No siento hambre(0)_____ Pocas veces tengo hambre(1)____ Varias veces tengo hambre(2) ______ Siempre tengo hambre(3)______ NR(999) 24)Yo voy al baño en: a. Un arbusto o en el monte (1) c. ______Baño o inodoro (3) b. ______ En Letrina (2) d. ______ Otro (888) e. ______ NR (999) 96 III. AMBIENTE LITERARIO EN EL HOGAR Pregúntele al niño, ¿Con quienes vives? A medida que el/la niño/a responda, llene las casillas de abajo. Para cada persona quien el/la niño/niña nombre, pregúntele: ¿has visto a esta persona leer en la última semana? (anote 1 para SI, 0 para NO, NR para no responde), etc. Sólo considere los miembros del hogar que tienen por lo menos 5 años de edad. Si hay más de 8 miembros, márquelo al final de la página. ¿Cuántas personas viven en tu casa? ________ personas Nombre Parentesco Mamá = 1, Papá = 2, Hermana = 3, Hermano = 4, Abuela = 5, Abuelo = 6, Otro (mujer/niña) = 7, Otro (hombre/niño) = 8 ¿Viste a esta persona leyendo la semana pasada? No = 0, Sí = 1 ¿Esta persona te leyó un cuento de un libro la semana pasada? (Se comprende que es leído) No = 0, Sí = 1 ¿Esta persona te contó un cuento la semana pasado (Se comprende que es oral) No = 0, Sí = 1 ¿Esta persona te animó a estudiar la semana pasada? No = 0, Sí = 1 1) 2) 3) 4) 5) 6) 7) 8) ¿El estudiante vive con más de 8 personas mayores de 5 años? _____ No (0) _____ Sí (1) 97 IV. AMBIENTE LITERARIO EN LA COMUNIDAD 1) ¿Alguien que no vive en tu casa te leyó la semana pasada? _____ No (0) _____ Sí (1) _____ NR (999) 2) ¿Le leíste un libro/cuento a alguien que no vive en tu casa en la semana pasada? _____ No (0) _____ Sí (1) _____ NR (999) 3) ¿La semana pasada le prestaste un libro/cuento a alguien que no vive en tu casa? _____ No (0) _____ Sí (1) _____ NR (999) V. PREGUNTAS DE PARTICIPACIÓN EN ACTIVIDADES DE LECTURA 1) ¿La semana pasada pediste prestado libros del banco de libros? _____ No (0) )(salte a la pregunta 2) _____ Sí (1) _____ NR (999) a. Si sí, ¿cuántas veces? ______ veces b. Si sí, ¿cuál es el nombre/título de tu cuento favorito del Banco de Libros? _____ No nombra el título (0) _____ Nombra el título (1) 2) ¿Tienes un amigo/compañero de lectura? _____ No (0) (salte a la pregunta 3) _____ Sí (1) _____ NR (999) a. Si sí, ¿cómo se llama? _____ No nombra al amigo (0) _____ Nombra al amigo (1) b. Si sí, ¿este fue asignado por tu maestro del campamento de lectura? _____ No (0) _____ Sí (1) _____ NR (999) c. Si sí, ¿te reuniste a leer con tu amigo de lectura la semana pasada? _____ No (0) _____ Sí (1) _____ NR (999) d. Si sí, ¿cuántas veces te reuniste a leer?? ______ veces 3) ¿Fuiste al campamento de lectura la semana pasada? _____ No (0) (salte a la pregunta 4) _____ Sí (1) _____ NR (999) a. Si sí, ¿cuántas veces? _____ veces 5) ¿Creaste materiales para llevar a casa en el campamento de lectura la semana pasada? _____ No (0) _____ Sí (1) _____ NR (999) a. Si sí, ¿con qué frecuencia? _____ (ponga el número de veces que el niño menciona) 98 VI. CONOCIMIENTO DEL ALFABETO. ESPAÑOL Dar al niño(a) la hoja plastificada con las letras en español. Dígale al niño(a): Vamos a mirar algunas letras en Español y K’iche´. Vamos a empezar con las letras en Español. Me gustaría que las leyeras. Puedes empezar acá (señale la primera letra) y continuar en esta dirección? (mueva su dedo de izquierda a derecha). Dime y señálame con tu dedo a medida que me digas el nombre de cada letra. Es claro? Bueno, puedes comenzar. Ponga un 1 en el espacio al lado de la letra si el niño puede decir cualquier de estas:  El nombre de la letra en el idioma materno o el idioma de instrucción  El sonido de la letra (válido en la lengua materna o en el lenguaje de instrucción)  Una respuesta en la que el niño diga “Comienza como..” dando una palabra para la que la letra sea la letra inicial Si el niño duda cualquiera de las letras por más de 5 segundos pregúntele: o ¿Sabes su nombre? O el sonido que hace? o ¿Conoces cualquier palabra que empiece por esta letra?  Si el niño duda por 5 segundos más, dígale: Bueno, no tienes que ir en orden, ¿pero me puedes señalar y decir las letras que sepas?  Si el estudiante duda por 10 segundos más, pare. Agradézcale al estudiante por su esfuerzo. Marque todas las letras que el estudiante no identificó o no intentó como incorrectas con un 0. Continúe a la siguiente sección. e ____ L____ v____ G____ P____ 5 S ____ b ____ Q ____ r ____ D ____ 10 h ____ w ____ Y ____ K ____ u ____ 15 N ____ t ____ A ____ x ____ J ____ 20 c ____ Z ____ m ____ f ____ O ____ 25 i ____ Ñ ____ 30 Total de letras correctas: ________/27 99 VII. CONOCIMIENTO DEL ALFABETO. K’ICHE’ Dar al niño(a) la hoja plastificada con las letras en k’iche’. Dígale al niño(a): Ahora vamos a leer las letras en K’iche’ Me gustaría que las leyeras. Puedes empezar acá (señale la primera letra) y continuar en esta dirección? (mueva su dedo de izquierda a derecha). Dime y señálame con tu dedo a medida que me digas el nombre de cada letra. Es claro? Bueno, puedes comenzar. Ponga un 1 en el espacio al lado de la letra si el niño puede decir cualquier de estas:  El nombre de la letra en el idioma materno o el idioma de instrucción  El sonido de la letra (válido en la lengua materna o el lenguaje de instrucción)  Una respuesta en la que el niño diga “Comienza como..” dando una palabra para la que la letra sea la letra inicial Si el niño duda cualquiera de las letras por más de 5 segundos pregúntele: o ¿Sabes su nombre? O el sonido que hace? o ¿Conoces cualquier palabra que empiece por esta letra?  Si el niño duda por 5 segundos más, dígale: Bueno, no tienes que ir en orden, ¿pero me puedes señalar y decir las letras que sepas?  Si el estudiante duda por 10 segundos más, pare. Agradézcale al estudiante por su esfuerzo. Marque todas las letras que el estudiante no identificó o no intentó como incorrectas con un 0. Continúe a la siguiente sección. E ____ L____ k’____ q’____ P____ 5 s ____ B’ ____ Q ____ r ____ t’____ 10 Ch ____ w ____ Y ____ K ____ u ____ 15 n ____ T ____ a ____ X ____ j ____ 20 Tz ____ ch’____ M ____ tz’ ____ O ____ 25 i ____ ( ‘ ) ____ 30 Total de letras correctas: ________/27 100 VIII. PALABRAS SIMPLES (VOCABULARIO). ESPAÑOL Dele al niño la copia plastificada de la lista de palabras en Español. Diga: Me gustaría que leas algunas palabras en Español y K’iche´ para mí. Vamos a empezar con las palabras en Español. Por favor, señala y di cada una de estas palabras empezando acá (señale la primera palabra) y siguiendo cada línea así (indique la dirección de izquierda a derecha con su dedo). Esta claro? Ok, puedes empezar. Ponga un 1 debajo de cada una de las palabras que el niño(a) lee correctamente y un 0 al lado de las palabras leídas incorrectamente o que el niño(a) no conoce. Se califican como correctas las palabras que el niño lea sin importar el orden en que las lea. Adicionalmente, una palabra es contada como correcta si el niño la lee mal primero y después se autocorrige. Cuente el número total de palabras que el niño(a) lee correctamente y ponga este número en la columna final. Si el niño(a) lee las palabras en desorden entonces muéstrele las palabras que se pueden haber saltado.  Si el niño(a) duda cualquier palabra por 5 segundos, pregúntele, ¿Hay alguna palabra en la lista que sepas? Dime y señálame las palabras de la lista que sabes. Repita y anime al niño a continuar.  Si el niño duda por más de 10 segundos, pare. Agradézcale al niño por haber hecho su mayor esfuerzo. Marque todas las palabras que el niño no identificó o no intentó como incorrectas. Continúe a la siguiente sección. mamá ____ palabra____ sol ____ hoja____ sobre ____ lectura ____ luna ____ comunidad ____ libro ____ nombre ____ casa ____ Güisquil ____ compañero ____ papel ____ lugar ____ día ____ grupo ____ dedo ____ diccionario ____ personas ____ Total palabras correctas: ______/20 101 IX. PALABRAS SIMPLES (VOCABULARIO). K’ICHE´ Dele al niño la copia plastificada de la lista de palabras en K’iche’. Diga: Ahora vas a leer las palabras en K’iche’. Por favor, señala y di cada una de estas palabras empezando acá (señale la primera palabra) y siguiendo cada línea así (indique la dirección de izquierda a derecha con su dedo). ¿Está claro? Ok, puedes empezar. Ponga un 1 debajo de cada una de las palabras que el niño(a) lee correctamente y un 0 al lado de las palabras leídas incorrectamente o que el niño(a) no conoce. Se califican como correctas las palabras que el niño lea sin importar el orden en que las lea. Adicionalmente, una palabra es contada como correcta si el niño la lee mal primero y después se autocorrige. Cuente el número total de palabras que el niño(a) lee correctamente y ponga este número en la columna final. Si el niño(a) lee las palabras en desorden entonces muéstrele las palabras que se pueden haber saltado.  Si el niño(a) duda cualquier palabra por 5 segundos, pregúntele, ¿Hay alguna palabra en la lista que sepas? Dime y señálame las palabras de la lista que sabes. Repita y anime al niño a continuar.  Si el niño duda por más de 10 segundos, pare. Agradézcale al niño por haber hecho su mayor esfuerzo. Marque todas las palabras que el niño no identificó o no intentó como incorrectas. Continúe a la siguiente sección. nan ____ pix____ ali ____ meseb’al____ wuj ____ tz’il ____ winaq ____ juyub’ ____ tijob’al ____ chak ____ ja ____ chakach ____ me’s ____ tz’i’ ____ che’ ____ q’ij ____ B’alam ____ ch’ek ____ kej____ wachalal ____ Total palabras correctas: ______/20 102 X. PALABRAS INVENTADAS (ESPAÑOL) Dele al niño la copia plastificada de la lista de palabras inventadas: Diga: Me gustaría que leyeras otras palabras para mí. Esta vez, son palabras inventadas en español que no tienen ningún significado. Por favor señala con tu dedo y di cada una de estas palabras. Si no sabes cómo leer algunas, no te preocupes. Da lo mejor de ti y eso será suficiente. Antes de comenzar, vamos a intentar leer dos de ellas como práctica. Puedes leer la primera palabra? Señale la primera palabra (lapi). Si el niño(a) la pronuncia correctamente diga “muy bien, la primera palabra es lapi”. Si el niño (a) no lee la palabra o la pronuncia incorrectamente diga “Esta palabra dice lapi”. Repita este proceso para la segunda palabra de prácitca. Sin importar si el niño pudo leer estas palabras continúe a la prueba. NOTA: Sólo corrija al niño(a) durante la práctica. Para la prueba, simplemente marque la palabra como correcta o incorrecta pero no corrija al niño(a) verbalmente o de ninguna indicación de si el niño(a) la leyó correcta o incorrectamente. Marque un 1 al lado de las palabras que el niño lee correctamente; marque un 0 al lado de las palabras pronunciadas incorrectamente o que el niño no intenta leer. Se califican como correctas las palabras que el niño lea sin importar el orden en que las lea. Adicionalmente, una palabra es contada como correcta si el niño la lee mal primero y después se autocorrige. Cuente el número total de palabras que el niño(a) lee correctamente y ponga este número en la columna final. Si el niño(a) lee las palabras en desorden entonces muéstrele las palabras que se pueda haber saltado.  Si el niño(a) duda cualquier palabra por 5 segundos, pregúntele, ¿Hay alguna palabra en la lista que sepas? Dime y señálame las palabras de la lista que sabes. Repita y anime al niño a continuar.  Si después de 10 segundos el niño no ha podido leer más de dos palabras pare la prueba. Agradézcale al niño por haber hecho su mayor esfuerzo. Marque todas las palabras que el niño no identificó o no intentó como incorrectas. Continúe a la siguiente sección. Practice words: lapi ____ tara ____ Assessment words: rexta____ fuande ____ lombro ____ bersones ____ beta ____ greto____ tuso ____ rompeñero____ infas ____ saciones ____ tobre ____ vecto ____ raca ____ serta ____ vugar ____ camaludad ____ pada ____ ascrapa____ tatá ____ salabra ____ Total words correct: ______/20 103 XI. PASAJE DE LECTURA. ESPAÑOL 1. Entregue el pasaje de lectura en Español al niño. Diga: Me gustaría que leyeras dos historias, una es en Español y la otra en K’iche´. Después de que termines cada historia te haré algunas preguntas sobre lo que leíste. 2. Diga: Cuando te diga ‘comienza’, empieza a leer en voz alta. Trata de leer cada palabra sin saltarte ninguna. Si llegas a alguna palabra que no sepas leer yo te la puedo decir. Lee lo mejor que puedas. Tienes alguna pregunta sobre lo que quiero que hagas? 3. Diga: ‘Empieza’ y comience a contar el tiempo con el cronómetro cuando el niño(a) empiece a leer la primera palabra del título. Si el niño/a no puede leer la primera palabra después de unos cuantos segundos márquela como incorrecta y dígale la palabra al niño/a. Deje que continúe leyendo y tache (/) las palabras que el niño/a no puede leer correctamente. A. Caso 1) Si el niño/a no lee por lo menos 5 palabras correctas en 30 segundos pare la prueba y agradézcale. No le haga las preguntas de comprensión. B. Caso 2) Si el niño/a lee al menos 5 palabras correctas en 30 segundos, déjelo/a continuar leyendo y marque con un círculo la palabra a la que el niño/a llega a los 60 segundos y deje que termine de leer el texto. Si el niño/a termina el texto antes de un minuto marque el tiempo exacto al final del párrafo. 4. Agradézcale al niño(a) cuando termine la lectura. EL Burro Mentiroso 1 2 3 Un burro que andaba por la montaña, encontró 4 5 6 7 8 9 10 11 Un libro de lectura que un niño había 12 13 14 15 16 17 18 19 olvidado. El animal lleno de alegría pensó– si 20 21 22 23 24 25 26 27 Yo me comiera este libro aprendería a leer 28 29 30 31 32 33 34 35 Sin el menor esfuerzo– y dejando la hierba 36 37 38 39 40 41 42 44 Se puso a masticar el libro. 45 46 47 48 49 50 El burro pensó tontamente que ya sabía leer 51 52 53 54 55 56 57 58 Y por eso andaba todo alegre gritando a 59 60 61 62 63 64 65 66 Los demás– ¡ya sé leer!, 67 68 69 70 71 ¡ya sé leer!– sintiéndose más inteligente que todos 71 72 73 74 75 76 77 78 Los burros. 79 80 Cierto día unos burros que caminaban por la 81 82 83 84 85 86 87 88 Montaña encontraron un periódico; pero como nadie sabía 89 90 91 92 93 94 95 96 leer, fueron corriendo a buscar al burro que 97 98 99 100 101 102 103 104 leía, uno de ellos le dijo: —léenos el 105 106 107 108 109 110 111 112 Periódico pues queremos saber lo que dice. 104 113 114 115 116 117 118 119 El burro con mucha tranquilidad tomó el periódico, 120 121 122 123 124 125 126 127 Lo miró por un rato, pero no dijo 128 129 130 131 132 133 134 135 Palabra alguna. El burro sintió vergüenza y fue 136 137 138 139 140 141 142 143 Objeto de burlas y carcajadas por parte de 144 145 146 147 148 149 150 151 Los otros burros. 152 153 154 Número de palabras correctas en 60 segundos: ________ XII. COMPRENSIÓN DE LECTURA. ESPAÑOL Diga: Ahora te voy a hacer algunas preguntas sobre el pasaje que acabas de leer. Puedes mirar la historia de nuevo si necesitas. Realice cada una de las preguntas de comprensión. Escriba un 1 frente a cada respuesta correcta, y un 0 frente a cada respuesta incorrecta. Al final sume y escribe el número total de respuestas correctas. 1) ¿De qué trata la historia? (El niño tiene que mencionar por lo menos tres de estos puntos: A) Es de un burro que quería aprender a leer, B) El burro se comió un libro, C) El burro creyó que comiéndose el libro iba a aprender a leer, D) Los otros burros le pidieron leer el periódico pero no pudo E) El burro fue objeto de burlas por parte de los otros burros al no poder leer el periódico.) _____Correcta (1) ______Incorrecta (0) 2) ¿Qué encontró el burro? (Respuesta: un libro de lectura, un libro de lectura que un niño había olvidado) _____Correcta (1) ______Incorrecta (0) 3) ¿Qué pensó el burro que pasaría si se comiera el libro? (Respuesta: que ya sabía leer, que podría leer) _____Correcta (1) ______Incorrecta (0) 4) ¿A quién fueron a buscar los burros que encontraron un periódico? (Respuesta: al burro lector) _____Correcta (1) ______Incorrecta (0) 5) ¿Qué hizo el burro después de tomar el periódico? (Respuesta: Lo miró por un rato, no pronunció palabra alguna) _____Correcta (1) ______Incorrecta (0) 6) ¿Cómo quedó el burro frente a los otros burros después de no haber leído el periódico? (Respuesta: Quedó en ridículo, le dio verguenza) _____Correcta (1) ______Incorrecta (0) 7) ¿Por qué crees que el niño olvidó su libro? (Respuesta: porque se distrajo con algo más, porque estaba apurado de volver a casa, porque un amiguito lo escondió mientras jugaban). _____Correcta (1) ______Incorrecta (0) 105 8) ¿Por qué crees que los burros corrieron hacia el burro lector cuando encontraron el periódico? (Respuesta: porque tenían apuro de saber lo que decía, porque estaban emocionados de pensar que alguien les iba a leer el periódico, etc.) _____Correcta (1) ______Incorrecta (0)) 9) ¿Crees que los burros son malos por burlarse del burro lector? ¿Por qué? (Respuesta: válida desde que la opinión contenga detalles de la historia) _____Correcta (1) ______Incorrecta (0) 10) ¿Crees que el burro lector intentaría comerse otro libro para aprender a leer? (Respuesta: válida desde que la opinión contenga detalles de la historia) _____Correcta (1) ______Incorrecta (0) Total preguntas de comprensión correctas: ______/10 XIII. PASAJE DE LECTURA. K’ICHE´ 1. Entregue el pasaje de lectura en K’iche´ al niño. Diga: Me gustaría que ahora leyeras esta historia en Ixil. Después de que termines cada historia te haré algunas preguntas sobre lo que leíste. 2. Diga: Cuando te diga ‘comienza’, empieza a leer en voz alta. Trata de leer cada palabra sin saltarte ninguna. Si llegas a alguna palabra que no sepas leer yo te la puedo decir. Lee lo mejor que puedas. Tienes alguna pregunta sobre lo que quiero que hagas? 3. Diga: ‘Empieza’ y comience a contar el tiempo con el cronómetro cuando el niño(a) empiece a leer la primera palabra del título. Si el niño/a no puede leer la primera palabra después de unos cuantos segundos márquela como incorrecta y dígale la palabra al niño/a. Deje que continúe leyendo y tache (/) las palabras que el niño/a no puede leer correctamente. A. Caso 1) Si el niño/a no lee por lo menos 5 palabras correctas en 30 segundos pare la prueba y agradézcale. No le haga las preguntas de comprensión. B. Caso 2) Si el niño/a lee al menos 5 palabras correctas en 30 segundos, déjelo/a continuar leyendo y marque con un círculo la palabra a la que el niño/a llega a los 60 segundos y deje que termine de leer el texto. Si el niño/a termina el texto antes de un minuto marque el tiempo exacto al final del párrafo.  4. Agradézcale al niño(a) cuando termine la lectura. 1 Ri ukojik ri nab’e taq uxaq ab’ 1 2 3 4 5 6 7 K’o jumul pa Ri komon Kequlew, chi rochoch 8 9 10 11 12 13 14 15 Tlin, chi ri ab’ix kutzaq le nab’e taq 16 17 18 19 20 21 22 23 uxaq, konojel ri Alaxik kib’an ri jun wa’im, 24 25 26 27 28 29 30 31 nab’e ri Tlin rachi’l ri unan, sib’alaj atam 32 33 34 35 36 37 38 39 keb’e pa ke’m, k’are k’uri’ kich’up le nab’e 40 41 42 43 44 45 46 47 taq uxaq ab’ix, Xa rumal chuwach le jun 48 49 50 51 52 53 54 55 uxaq ab’ix kipis wi le kwa’ch taq sub’, 56 57 58 59 60 61 62 63 ri kib’im ri qanan qatat rech wa’ keb’ 64 65 66 67 68 69 70 71 ri reqa’n ri ab’ix kuya’o. Je k’uri’ kikamisaj 72 73 74 75 76 77 78 79 jun ama’ ak’, rech wa’ ktij ruk’ ri 80 81 82 83 84 85 86 87 sub’, ktijtaj ri wa’im konojel ri alaxik kimulij 88 89 90 91 92 93 94 95 ri uxaq ab’ix, k’ate k’uri, kixim cho le 96 97 98 99 100 101 102 103 ixk’ub’, k’ate kesaj chila’ chi’ kib’an ri jach’, 104 105 106 107 108 109 110 111 are’ wa’ ri nojib’al xretamaj ri Tlin kuk’ 112 113 114 115 116 117 118 119 ri unan utat, rajawaxik wa’ ma qasach taj 120 121 122 123 124 125 126 127 ri nojib’al. 128 129 Número de palabras correctas en 60 segundos: ________ XIV. COMPRENSIÓN DE LECTURA. K’ICHE’ Diga: Ahora te voy a hacer algunas preguntas sobre el pasaje que acabas de leer. Puedes mirar la historia de nuevo si necesitas. Realice cada una de las preguntas de comprensión. Escriba un 1 frente a cada respuesta correcta, y un 0 frente a cada respuesta incorrecta. Al final sume y escribe el número total de respuestas correctas. I. ¿ B’ij chwe su xata chi upam le tzijob’elil? (Respuesta: ri ukojik ri k’ak’ taq uxaq ab’ix, ri, ri wa’im kb’an chi’ kkoj ri uxaq ab’ix) -----------------je’ (1) -----------------je’ taj (0) 2 2. ¿Su ub’i’ ri komon? (Respuesta: Kequlew) -----------------je’ (1) -----------------je’ taj (0) 3. ¿Su kub’an ri Tlin ruk’ ri unan atam? (Respuesta: keb’e pa ke’m) -----------------je’ (1) -----------------je’ taj (0) 4. ¿Su che kb’an kwa’ch che le sub’? (Respuesta: rech keb’ reqa’n ri ab’ix kuya’o) -----------------je’ (1) -----------------je’ taj (0) 5. ¿ Su kikamisaj rech kitij ruk’ ri sub’? (Respuesta: Jun ama’ ak’) -----------------je’ (1) -----------------je’ taj (0) 6. ¿Chi kixim wi ri uxaq ab’ix chi xkojtajik? (Respuesta: (cho le tem, pa uwi’ le ixk’ub, pa uwi’ le sib’) -----------------je’ (1) -----------------je’ taj (0) 7. ¿Su kraqtajik we ma kb’an chik we nojib’al ri? (Respuesta: ksach uwach le nojib’al, kuya’ ta reqa’n le ab’ix) -----------------je’ (1) -----------------je’ taj (0) 8. ¿Su che’ kb’an wa’ we nojib’al pa taq qakomon? (Respuesta: rumal are’ le nojib’al kiya’om kanoq le qamam qati’t, are wa’ le kinojib’al qamam qati’t ) -----------------je’ (1) -----------------je’ taj (0) 9. ¿ Su kachomaj at la utz kb’an wa’ we nojib’al ri? (Respuesta: utz rech kik’is ta le qanojib’al, rech qaya’ ujiq’ij le qa b’antajik par i qachoch) ----------------je’ (1) -----------------je’ taj (0) 10. ¿Su xawetamaj chi upam we tzijob’elil? (Respuesta: Le nojib’al rech kkoj le nab’e taq uxaq ab’ix, le ronojel le nojib’al kb’anik rech ukojik le uxaq ab’ix) ----------------je’ (1) -----------------je’ taj (0) Total preguntas de comprensión correctas: ______/10 3 Annex 3. Explanation of the statistical models Multilevel Models The hierarchical model that was used is the following: Level 1: Yij= β0j+ ΣβqjXij+ rij (1) Level 2: β0j= γ00+ γ01Tij+Σγqj Wj+ u0j (2), where Yij represents the reading ability of student i nested within school j. Yij also represents each one of the dependent variables mentioned previously. β0j represents the school average j; βqj represents the first￾level regression coefficient associated with each level 1 covariate in the model; Xij is a set of covariates that represent the characteristics of the students; and rij is the random error associated with each student. γ00 is the general average of all the schools; γ01 is the effect of the treatment’s intensity; Tij is the variable that measures the treatment’s intensity; γqj represents the coefficients of each variable Wj; and Wj is a set of covariates that represents the characteristics of the highest level of nesting which, in this case, are aggregations at the classroom level, of each variable of the student's level. Finally, u0j is the random error associated with level 2 of the model. The variables Xij and Wj are observed directly in tables 1 to 5, where the results of the models are also presented. To start the analysis, null or empty models were run to determine the intraclass correlation for each of the dependent variables. This intraclass correlation helps to determine the magnitude of the variance between classrooms and allows determining the need to use hierarchical models due to the intercorrelation of the data. Thus, to the extent that the intraclass correlation increases, the use of hierarchical models is more than justified. Difference in differences model (exposure to treatment) This model was adjusted for second grade using the baseline and final evaluations. In addition, the model included a series of explanatory variables. The model was adjusted with R software. The mathematical form of this model is as follows: Yit= β0 + β1Ait+ β2Ti+ β3AitTi+ βqXit +eit (3), where Yit is the variable that measures the students’ reading abilities. In this sense, the reading ability was measured with the following seven variables: (a) Percentage of correct letters in Spanish, (b) Percentage of correct MUW in Spanish, (c) Percentage of correct decoding in Spanish, (d) Fluency in Spanish, (e) Accuracy in Spanish, (f) Percentage of comprehension in Spanish, and (g) Percentage of literal comprehension in Spanish. Ait is a dichotomous variable (1 = Area 1, 0 = Area 2) that represents the intervention in the defined areas. Ti is also a dichotomous time variable (0 = measurement in the baseline evaluation, 1 = measurement in the final evaluation); β3 is a coefficient that measures the effect of the treatment. βq is a vector that includes the coefficients of the covariates included in the model and Xit is a matrix of student characteristic covariates (for example, socioeconomic status, sex, age, etc.), which can be seen in the tables that present the results of the models. This model was adjusted by estimating robust standard errors for clustering (schools). Regression models, Pre (baseline, midterm evaluation) and Post (final evaluation) The response variables in the adjusted models were the following: (1) Percentage of correct letters in Spanish, (2) Percentage of correct MUW in Spanish, (3) Percentage of correct decoding in Spanish, (4) Fluency in Spanish, (5) Accuracy in Spanish, (6) Percentage of comprehension in Spanish, and (7) 4 Percentage of literal comprehension in Spanish. With the exception of variable (3), the rest of the variables were also used in the K'iche’ and Ixil languages. Dependent variables used in the pre-post models and their acronyms VARIABLE VARIABLE ACRONYMS SPANISH K’ICHE’ IXIL Percentage of correct letters Eltrspct Kltrspct iltrspct Percentage of correct MUW Emuwpct Kmuwpct imuwpct Percentage of correct decoding Dcwpct ------ ------ Fluency Ewcpm Kwcpm iwcpm Accuracy Ereadpct Kreadpct ireadpct Percentage of comprehension Ereadcomppct kreadcomppct ireadcomppct Percentage of literal comprehension Ereadlitcomppct kreadlitcomppct ireadlitcomppct The mathematical model used was the following multiple linear regression: Yi= β0+ β1Ti+ ΣβiXi+ ei (4), where Yi represents the reading ability of student i in each of the variables presented in Table 1. β0 is the model’s intercept; β1 captures the difference in measurement between the baseline (or intermediate as the case may be) and final evaluations. βi captures the regression coefficient corresponding to the covariate Xi. Xi represents the covariate i, which corresponds to the characteristics of the student. Finally, ei is the random error associated with each student. In total, 65 regression models were adjusted. The models adjusted by language and by area are presented in the following table. Number of models adjusted for the second and third-grade pre-post analysis Spanish K'iche' Ixil Baseline and final evaluation 2nd grade Area 1 7 6 ----- Area 2 7 ----- 6 Midterm and final evaluation 2nd grade Area 1 7 6 ----- Area 2 7 ----- 6 Midterm and final evaluation 3rd grade Area 2 7 6 ----- 5 Annex 4. Distribution of Non-Readers, Beginners, and Readers In calculating the Z score, the data reflects a movement to the left in terms of the quantity of reading comprehension questions that students could respond to, reducing the average of 37% of the Reading comprehension questions that students managed to adequately respond to 20% in area 2 and 14.66% in the area 1. Graph A. Spanish correct literal comprehension, Area 1 Mean= 0.14664; Standard Deviation= 0.099300; N= 144 In calculating the Z score for the categories (“Non-readers”, “Beginners”, “Readers”, you see a movement from the mean to the right. This can be explained by the increase of the “Beginners and reduction of the "Non-readers", which decreases to reach the level of "readers". This can be influenced by the type of assessment is not a captive that is not assessed to the students themselves in the assessment of baseline and midterm, combined with the time in which the children were exposed to the methodology due to the strike of the magisterium, a month prior to the final evaluation. Graph B. Spanish correct literal comprehension, Area 2 Mean= .20005, Standard deviation= 1.002808, N=384 Punctuation Z: % Literal spanish correct comprehension Area 1 Frecuency Media= .14664 Standard deviation=.99300 N= 144 Punctuation Z: % Literal spanish correct comprehension Area 2 Frecuency Media= .20005 Standard deviation=1.00208 N= 384 6 Graph C. Spanish correct literal comprehension, midterm evaluation Mean = .37523; Standard deviation= 0.89267, N= 261 Graph D. Reading comprehension by levels, Area 1 Mean= 0.25018, Standard deviation = 0.95674, N = 287 Punctuation Z: % Literal spanish correct comprehension Midterm Frecuency Media= .37523 Standard deviation=.89267 N= 261 Punctuation Z: Reading comprehension by levels Area 1 Frecuency Media= .25018 Standard deviation=.95674 N= 287 Non-readers Beginners Readers 7 Graph E. Reading comprehension by levels, Area 2 Mean = 0.13110, Standard deviation= 0.89875, N= 522 Graph F. Reading comprehension by levels, midterm evaluation Mean=0.0078, Standard deviation=1.10235, N=463 Punctuation Z: Reading comprehension by levels Area 2 Frecuency Media= .13110 Standard deviation=.89875 N= 522 Non-readers Beginners Readers Punctuation Z: Reading comprehension by levels Midterm Frecuency Media= .00728 Standard deviation=1.10235 N= 463 Non-readers Beginners Readers 8 Annex 5. Statistical results of literacy components and associated covariates Difference in differences model for the variable Percentage of letters read correctly in Spanish Coefficient Standard error t-value p-value Intercept 0.595 0.096 6.180 0.000 *** Treatment (Area 1 =1, Area 2 = 0) β1 -0.136 0.038 -3.550 0.000 *** Time (2018 = 1, 2015 =0) β2 0.148 0.034 4.336 0.000 *** Treatment x Time β3 0.050 0.042 1.194 0.232 Sex (Male = 0, Female = 1) β4 -0.009 0.011 -0.843 0.399 Age β5 0.003 0.007 0.393 0.694 Attended preschool (Yes = 1, No = 0) β6 0.014 0.017 0.794 0.428 Repeating second grade (Yes = 1, No = 0) β7 -0.033 0.014 -2.315 0.021 * Speaks Spanish at home (Yes = 1, No = 0) β8 0.060 0.029 2.039 0.042 * Books at home (scale) β9 0.003 0.007 0.414 0.679 Helps with house chores (Yes = 1, No = 0) β10 0.027 0.042 0.651 0.515 Works outside of the home (Yes = 1, No = 0) β11 -0.009 0.013 -0.694 0.488 Had breakfast this morning (Yes = 1, No = 0) β12 -0.013 0.027 -0.499 0.618 Read to someone in the community (Yes = 1, No = 0) β13 0.051 0.014 3.530 0.000 *** Went to Reading Camp (Yes = 1, No = 0) β14 0.046 0.026 1.796 0.073 . SES β15 0.004 0.002 1.830 0.067 . Percentage of people that help you study β16 0.016 0.029 0.543 0.588 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.2184, R2 -adjusted = 0.2109. Residual Standard Error 0.2419 on 1655 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. Difference in differences model for the variable Percentage of correct MUW in Spanish Coefficient Standard error t-value p-value Intercept 0.402 0.132 3.043 0.002 ** Treatment (Area 1 =1, Area 2 = 0) β1 -0.156 0.054 -2.917 0.004 ** Time (2018 = 1, 2015 =0) β2 0.264 0.061 4.318 0.000 *** Treatment x Time β3 -0.024 0.073 -0.330 0.742 Sex (Male = 0, Female = 1) β4 -0.019 0.016 -1.159 0.247 Age β5 -0.005 0.010 -0.474 0.636 Attended preschool (Yes = 1, No = 0) β6 0.052 0.026 2.000 0.046 * 9 Repeating second grade (Yes = 1, No = 0) β7 -0.120 0.021 -5.605 0.000 *** Speaks Spanish at home (Yes = 1, No = 0) β8 0.114 0.044 2.576 0.010 * Books at home (scale) β9 0.005 0.010 0.507 0.612 Helps with house chores (Yes = 1, No = 0) β10 0.015 0.050 0.304 0.761 Works outside of the home (Yes = 1, No = 0) β11 -0.034 0.020 -1.691 0.091 . Had breakfast this morning (Yes = 1, No = 0) β12 -0.033 0.037 -0.908 0.364 Read to someone in the community (Yes = 1, No = 0) β13 0.101 0.021 4.826 0.000 *** Went to Reading Camp (Yes = 1, No = 0) β14 0.043 0.047 0.917 0.359 SES β15 0.007 0.003 2.366 0.018 * Percentage of people that help you study β16 0.007 0.043 0.159 0.874 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.2423, R2 -adjusted = 0.235. Residual Standard Error 0.3512 on 1655 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. Difference in differences model for the variable Percentage of correct decoding in Spanish Coefficient Standard error t￾statistic p-value Intercept 0.399 0.132 3.012 0.003 ** Treatment (Area 1 =1, Area 2 = 0) β1 -0.118 0.052 -2.258 0.024 * Time (2018 = 1, 2015 =0) β2 0.267 0.065 4.117 0.000 *** Treatment x Time β3 -0.021 0.077 -0.276 0.783 Sex (Male = 0, Female = 1) β4 -0.022 0.016 -1.338 0.181 Age β5 -0.015 0.010 -1.390 0.165 Attended preschool (Yes = 1, No = 0) β6 0.044 0.025 1.751 0.080 . Repeating second grade (Yes = 1, No = 0) β7 -0.116 0.022 -5.181 0.000 *** Speaks Spanish at home (Yes = 1, No = 0) β8 0.119 0.044 2.718 0.007 ** Books at home (scale) β9 0.003 0.011 0.250 0.803 Helps with house chores (Yes = 1, No = 0) β10 0.018 0.051 0.352 0.725 Works outside of the home (Yes = 1, No = 0) β11 -0.031 0.022 -1.404 0.160 Had breakfast this morning (Yes = 1, No = 0) β12 -0.035 0.041 -0.871 0.384 Read to someone in the community (Yes = 1, No = 0) β13 0.096 0.021 4.570 0.000 *** Went to Reading Camp (Yes = 1, No = 0) β14 0.026 0.047 0.553 0.581 SES β15 0.007 0.003 2.178 0.030 * Percentage of people that help you study β16 0.029 0.042 0.687 0.492 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.2124, R2 -adjusted = 0.2048. Residual Standard Error 0.3643 on 1655 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. 10 Difference in differences model for the variable Fluency (wcpm)- Spanish Coefficient Standard error t￾statistic p-value Intercept 7.979 4.913 1.624 0.105 Treatment (Area 1 =1, Area 2 = 0) β1 -2.929 1.749 -1.675 0.094 . Time (2018 = 1, 2015 =0) β2 17.136 3.829 4.475 0.000 *** Treatment x Time β3 -2.894 4.646 -0.623 0.533 Sex (Male = 0, Female = 1) β4 0.442 0.914 0.483 0.629 Age β5 -0.462 0.384 -1.202 0.230 Attended preschool (Yes = 1, No = 0) β6 1.297 1.233 1.052 0.293 Repeating second grade (Yes = 1, No = 0) β7 -3.317 1.083 -3.064 0.002 ** Speaks Spanish at home (Yes = 1, No = 0) β8 7.867 2.232 3.525 0.000 *** Books at home (scale) β9 0.869 0.624 1.393 0.164 Helps with house chores (Yes = 1, No = 0) β10 2.329 1.803 1.292 0.197 Works outside of the home (Yes = 1, No = 0) β11 -1.874 1.142 -1.641 0.101 Had breakfast this morning (Yes = 1, No = 0) β12 -3.607 2.518 -1.432 0.152 Read to someone in the community (Yes = 1, No = 0) β13 3.731 1.122 3.325 0.001 *** Went to Reading Camp (Yes = 1, No = 0) β14 1.493 1.620 0.922 0.357 SES β15 0.395 0.137 2.885 0.004 ** Percentage of people that help you study β16 0.623 2.130 0.293 0.770 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.260, R2 -adjusted = 0.231. Residual Standard Error 16.94 on 1655 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. Difference in differences model for the variable Accuracy-Spanish Coefficient Standard error t￾statistic p-value Intercept 0.324 0.138 2.346 0.019 * Treatment (Area 1 =1, Area 2 = 0) β1 -0.109 0.053 -2.084 0.037 * Time (2018 = 1, 2015 =0) β2 0.280 0.073 3.837 0.000 *** Treatment x Time β3 -0.029 0.087 -0.332 0.740 Sex (Male = 0, Female = 1) β4 -0.003 0.018 -0.194 0.846 Age β5 -0.015 0.011 -1.309 0.191 Attended preschool (Yes = 1, No = 0) β6 0.045 0.030 1.529 0.127 Repeating second grade (Yes = 1, No = 0) β7 -0.092 0.028 -3.337 0.001 *** Speaks Spanish at home (Yes = 1, No = 0) β8 0.124 0.042 2.938 0.003 ** 11 Books at home (scale) β9 0.015 0.013 1.180 0.238 Helps with house chores (Yes = 1, No = 0) β10 -0.004 0.055 -0.067 0.947 Works outside of the home (Yes = 1, No = 0) β11 -0.036 0.025 -1.482 0.139 Had breakfast this morning (Yes = 1, No = 0) β12 -0.034 0.040 -0.867 0.386 Read to someone in the community (Yes = 1, No = 0) β13 0.101 0.023 4.372 0.000 *** Went to Reading Camp (Yes = 1, No = 0) β14 0.040 0.047 0.857 0.391 SES β15 0.009 0.004 2.475 0.013 * Percentage of people that help you study β16 0.058 0.047 1.241 0.215 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.11928, R2 -adjusted = 0.185. Residual Standard Error 0.4081 on 1655 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. Difference in differences model for the variable Percentage of correct comprehension in Spanish Coefficient Standard error t￾statistic p-value Intercept 0.218 0.096 2.274 0.023 * Treatment (Area 1 =1, Area 2 = 0) β1 -0.032 0.039 -0.827 0.408 Time (2018 = 1, 2015 =0) β2 0.240 0.050 4.786 0.000 *** Treatment x Time β3 0.147 0.062 2.375 0.018 * Sex (Male = 0, Female = 1) β4 -0.035 0.015 -2.357 0.019 * Age β5 -0.016 0.008 -1.908 0.057 . Attended preschool (Yes = 1, No = 0) β6 0.032 0.023 1.410 0.159 Repeating second grade (Yes = 1, No = 0) β7 -0.033 0.019 -1.762 0.078 . Speaks Spanish at home (Yes = 1, No = 0) β8 0.262 0.045 5.801 0.000 *** Books at home (scale) β9 0.010 0.009 1.195 0.232 Helps with house chores (Yes = 1, No = 0) β10 0.007 0.046 0.144 0.885 Works outside of the home (Yes = 1, No = 0) β11 -0.025 0.018 -1.451 0.147 Had breakfast this morning (Yes = 1, No = 0) β12 -0.067 0.033 -2.018 0.044 * Read to someone in the community (Yes = 1, No = 0) β13 0.065 0.018 3.533 0.000 *** Went to Reading Camp (Yes = 1, No = 0) β14 -0.008 0.031 -0.267 0.790 SES β15 0.011 0.003 3.600 0.000 *** Percentage of people that help you study β16 0.020 0.038 0.530 0.597 Note: * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. R2 = 0.3178, R2 - adjusted = 0.3102. Residual Standard Error 0.2827 on 1428 degrees of freedom. The values of the coefficients can be multiplied by 100 to interpret them as percentages. 12 Regression models for second-grade K’iche’ reading skills in area 2, pre-post comparison Kltrspct kmuwpct kwcpm kreadpct kreadcomppct kreadlitcomppct Intercept 0.398** 0.194 -3.641 -0.155 0.016 0.075 Time (2018 = 1, 2015 =0) 0.008 0.311*** 11.303*** 0.237** 0.131*** 0.142*** Sex (Male = 0, Female = 1) -0.011 -0.022 0.477 0.019 -0.007 0.002 Age 0.025 0.004 0.163 0.015 0.002 -0.002 Attended preschool (Yes = 1, No = 0) 0.013 0.034 2.230. 0.062. 0.004 -0.001 Repeating second grade (Yes = 1, No = 0) -0.024 -0.049 -0.356 -0.007 0.002 0.010 Speaks Spanish at home (Yes = 1, No = 0) -0.017 -0.014 0.070 -0.026 -0.041** -0.021 Books at home (scale) -0.009 0.003 0.522 0.018 -0.005 -0.007 Helps with house chores (Yes = 1, No = 0) 0.055 0.028 2.107 0.013 0.021 0.009 Works outside of the home (Yes = 1, No = 0) -0.020 -0.050 -2.255 -0.052 0.005 0.017 Had breakfast this morning (Yes = 1, No = 0) -0.008 -0.044 -0.777 0.027 -0.045 -0.071 Read to someone in the community (Yes = 1, No = 0) 0.072*** 0.055* 2.297. 0.055. 0.026** 0.045*** Went to Reading Camp (Yes = 1, No = 0) 0.060* 0.122* 2.889 0.100. 0.017 -0.028 SES -0.002 0.000 0.214. 0.006. 0.000 0.000 Percentage of people that help you study -0.013 0.005 1.282 0.052 0.025 0.057 N 700 700 700 700 726 573 R 2 0.053 0.338 0.236 0.1818 0.174 0.113 Note: with the exception of the kwcpm variable, coefficients are interpreted as percentages when multiplied by 100. * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. Regression models for second-grade Ixil reading skills in area 1, pre-post comparison iltrspct imuwpct iwcpm ireadpct ireadcomppct ireadlitcomppct Intercept 0.449*** 0.250* 0.603 0.109 -0.041 -0.035 Time (2018 = 1, 2015 =0) 0.238*** 0.293*** 12.143*** 0.281*** 0.437*** 0.410*** Sex (Male = 0, Female = 1) 0.012 -0.007 0.094 -0.011 -0.017 -0.014 Age -0.016 -0.016** -0.409 -0.018 -0.001 -0.001 Attended preschool (Yes = 1, No = 0) 0.013 -0.021 -1.273 -0.013 0.027 0.023 Repeating second grade (Yes = 1, No = 0) -0.043* -0.073** -2.136 -0.066* -0.017 -0.013 Speaks Spanish at home (Yes = 1, No = 0) -0.082 -0.120* -2.750 -0.131 -0.032 -0.071* Books at home (scale) 0.009 -0.008 0.393 0.003 -0.003 -0.002 Helps with house chores (Yes = 1, No = 0) 0.006 -0.035 4.484* 0.057 0.000 0.006 Works outside of the home (Yes = 1, No = 0) 0.007 -0.002 -0.896 -0.040 -0.006 0.000 13 Had breakfast this morning (Yes = 1, No = 0) -0.051 -0.049 -3.112 -0.017 0.022 0.004 Read to someone in the community (Yes = 1, No = 0) 0.028 0.057** 0.797 0.033 0.018 0.013 Went to Reading Camp (Yes = 1, No = 0) -0.005 -0.028 -1.430 -0.038 -0.042* -0.034 SES 0.011*** 0.015*** 0.780*** 0.021*** 0.004 0.004 Percentage of people that help you study -0.002 -0.012 0.352 0.020 0.048 0.051 N 726 726 725 726 600 579 R 2 0.297 0.291 0.244 0.210 0.465 0.464 Regression models for Spanish in area 2, MTE-FE comparison for third grade eltrspct emuwpct dcwpct ewcpm ereadpct ereadcomppct ereadlitcomppct Intercept 0.751*** 0.798*** 0.514* 41.344* 0.679* 0.499* 0.680** Time (2018 = 1, 2015 =0) 0.051** -0.002 -0.003 2.333 0.010 0.068 -0.016 Sex (Male = 0, Female = 1) -0.020 -0.041 -0.063* -2.391 -0.068. -0.100*** -0.071** Age 0.005 0.001 0.009 -0.767 0.004 -0.044** -0.054124* Attended preschool (Yes = 1, No = 0) 0.032* 0.087** 0.082* 9.073** 0.140** 0.136*** 0.137** Repeating first-grade (Yes = 1, No = 0) -0.029** -0.077** -0.074* -8.674*** -0.140*** -0.101** -0.087* Repeating second grade (Yes = 1, No = 0) -0.031 -0.077* -0.091** -5.533. -0.105* -0.038 -0.025 Speaks Spanish at home (Yes = 1, No = 0) -0.015 -0.042 -0.063* -10.949*** -0.063. -0.170*** -0.211*** Books at home (scale) β9 -0.006 -0.011 0.002 1.370 -0.010 0.015 0.020 Helps with house chores (Yes = 1, No = 0) 0.002 -0.057 -0.005 -4.665 -0.074 0.225*** 0.275*** Works outside the home (Yes = 1, No = 0) -0.024 -0.049 -0.050 -8.679** -0.062. -0.088*** -0.115*** Had breakfast this morning (Yes = 1, No = 0) -0.013 -0.095*** -0.079* -8.266 -0.114* -0.161** -0.149* Read to someone in the community (Yes = 1, No = 0) 0.013 0.051** 0.060* 6.277** 0.087** 0.043 0.003 Has a reading buddy (Yes = 1, No = 0) 0.028 0.022 -0.002 0.164 -0.042 -0.027 0.000 Went to Reading Camp (Yes = 1, No = 0) 0.032* 0.023 0.034 4.687 0.092 0.054 -0.001 Prepared reading material in the camp (Yes = 1, No = 0) -0.027* -0.014 -0.015 -1.845 -0.042 -0.008 0.040 SES 0.001 0.002 0.005 0.025 0.002 0.007 0.010* 14 Someone helps you study (Yes = 1, No = 0) 0.059 0.091 0.127 11.969 0.123 0.213** 0.113 Percentage of people that help you study -0.005 0.058 0.070 7.322 0.083 0.079 0.085 N 485 485 485 485 485 454 387 R 2 0.131 0.139 0.122 0.212 0.164 0.329 0.316 With the exception of the variable ewcpm, the coefficients are interpreted as percentages when multiplied by 100. * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. Regression models for K’iche’ in area 2, MTE-FE comparison for third grade Kltrspct kmuwpct kwcpm kreadpct kreadcomppct kreadlitcomppct Intercept 0.523*** 0.043 8.898 0.368 -0.355** -0.513** Time (2018 = 1, 2015 =0) 0.059* 0.057 4.442 0.078 0.095** 0.043 Sex (Male = 0, Female = 1) please this has -0.017 -0.053* -2.262 -0.097** -0.032 -0.043 Age 0.003 0.038* -0.244 -0.008 0.009 0.025 Attended preschool (Yes = 1, No = 0) -0.013 0.048 2.956 0.063 0.006 0.000 Repeating first grade (Yes = 1, No = 0) -0.003 -0.060 -3.650* -0.051 -0.041 -0.058 Repeating second grade (Yes = 1, No = 0) -0.043* -0.100*** -1.217 -0.063 -0.054* -0.042 Speaks Spanish at home (Yes = 1, No = 0) 0.033 0.036 -0.632 0.045 0.032 0.060 Books at home (scale) β9 -0.012 0.001 -0.006 -0.013 -0.032* -0.031 Helps with house chores (Yes = 1, No = 0) 0.009 0.003 -7.033. -0.128. 0.030 0.038 Works outside of the home (Yes = 1, No = 0) -0.027* -0.021 -0.286 0.013 0.010 0.002 Had breakfast this morning (Yes = 1, No = 0) 0.038 0.009 -2.297 -0.019 0.058 0.120* Read to someone in the community (Yes = 1, No = 0) 0.020 0.030 4.383* 0.077* 0.045** 0.009 Have a reading buddy (Yes = 1, No = 0) 0.014 0.019 1.211 0.013 0.031 0.052 Went to Reading Camp (Yes = 1, No = 0) 0.072** 0.018 1.780 0.086 -0.043 -0.138* Prepared reading material in the camp (Yes = 1, No = 0) -0.037 -0.002 -3.050 -0.039 -0.008 0.014 SES 0.001 0.000 0.108 0.000 -0.001 -0.001 Someone helps you study (Yes = 1, No = 0) 0.020 0.070 13.206* 0.115 0.261*** 0.336*** Percentage of people that help you study -0.013 0.018 1.780 0.086 -0.043 -0.138* N 483 483 483 483 422 280 R 2 0.113 0.088 0.088 0.080 0.123 0.109 15 With the exception of the variable kwcpm, the coefficients are interpreted as percentages when multiplied by 100. * significance at 5%, ** significance at 0.01%, *** significance less than 0.0001. Annex 6. Literacy abilities by sex and area Second grade Year Area Sex % correct letters in Spanish % correct MUW in Spanis h % correct decoding in Spanish % fluency in Spanish % accuracy in Spanish % compreh ension in Spanish 2015 2 Male 0.73 0.51 0.43 9.78 0.37 0.21 Female 0.68 0.43 0.35 9.63 0.33 0.17 1 Male 0.55 0.27 0.23 4.67 0.19 0.11 Female 0.57 0.29 0.23 5.41 0.21 0.1 2016 2 Male 0.86 0.8 0.69 32.93 0.71 0.38 Female 0.86 0.78 0.65 33.58 0.66 0.3 1 Male 0.82 0.67 0.57 33.48 0.54 0.35 Female 0.81 0.64 0.55 29.71 0.54 0.37 2018 2 Male 0.89 0.76 0.68 27.39 0.18 0.49 Female 0.89 0.79 0.7 30.44 0.2 0.45 1 Male 0.76 0.52 0.46 21.5 0.14 0.55 Female 0.8 0.55 0.52 21.9 0.14 0.46 Year Area Sex % correct letters in K'iche' % correct MUW in K'iche' % correct decoding in K'iche' % fluency in K'iche' % accuracy in K'iche' % compreh ension in K'iche' 2015 2 Male 0.69 0.26 --- 4.08 0.16 0.04 Female 0.65 0.22 --- 4.11 0.18 0.03 2016 Male 0.63 0.54 --- 17.05 0.44 0.12 Female 0.63 0.5 --- 16.02 0.37 0.09 16 2018 Male 0.69 0.61 --- 16.8 0.14 0.19 Female 0.71 0.62 --- 19.24 0.15 0.17 Year Area Sex % correct letters in Ixil % correct MUW in Ixil % correct decoding in Ixil % fluency in Ixil % accuracy in Ixil % compreh ension in Ixil 2015 1 Male 0.34 0.09 --- 1.65 0.09 0.03 Female 0.35 0.09 --- 2.04 0.08 0.03 2016 Male 0.54 0.43 --- 24.52 0.39 0.29 Female 0.54 0.41 --- 21.37 0.39 0.31 2018 Male 0.58 0.38 --- 16.04 0.1 0.51 Female 0.61 0.4 --- 17.39 0.11 0.41 Third grade Year Area % correct letters in Spanish % correct MUW in Spanish % decoding in Spanish % fluency in Spanish % accuracy in Spanish % compreh ension in Spanish 2016 2 Male 0.860 0.800 0.690 32.930 0.710 0.380 Female 0.860 0.780 0.650 33.580 0.660 0.300 2018 Male 0.930 0.830 0.720 40.330 0.260 0.500 Female 0.900 0.820 0.700 40.630 0.260 0.460 % correct letters in K'iche' % correct MUW in K'iche' % correct decoding in K'iche' % fluency in K'iche' % accuracy in K'iche' % compreh ension in K'iche' 2016 2 Male 0.630 0.540 --- 17.050 0.440 0.120 Female 0.630 0.500 --- 16.020 0.370 0.090 2018 Male 0.720 0.610 --- 25.380 0.200 0.200 17 Female 0.690 0.600 --- 23.160 0.190 0.200