Mid Term Evaluation IDEA Project, SC/USDA ASOCIACIÓN DE DESARROLLO ORGANIZACIONAL COMUNITARIA ADOC ADOCGUATEMALA@GMAIL.COM, FRINEPAZ@YAHOO.COM, FACEBOOK: ASOCIACION ADOC TELEFONO: 54148700 -42900838 2016 Photo: Children Amacchel School FY 2014 USDA McGovern-Dole IDEA Guatemala Project Midterm Evaluation Program: McGovern Dole Agreement Number: FFE-520-2014/043-00 Funding Year: Fiscal Year 2014 Project Duration: 2019-2022 Implemented by: Save the Children Evaluation Authored by: ASOCIACIÓN DE DESARROLLO ORGANIZACIONAL COMUNITARIA (ADOC) DISCLAIMER: This evaluation was conducted by an independent third party firm. The views expressed in this report do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this report can be made available upon request by emailing FAS.monitoring.evaluation@usda.gov. Mid-Term Evaluation IDEA Project SCI P a g e 1 | 81 TABLE OF CONTENTS 1. ACKNOWLEDGEMENTS ................................................................................................................ 4 2. EXECUTIVE SUMMARY.................................................................................................................... 7 3. SCOPE OF THE CONSULTANCY .............................................................................................. 10 4. CONTEXT ......................................................................................................................................... 11 4.1 Project Background ................................................................................................................. 11 4.2 Key project partners and stakeholders............................................................................... 12 5. RESEARCH PLAN ............................................................................................................................. 13 5.1 Key research questions .......................................................................................................... 13 5.2 Methodology for the Mid-term Evaluation ......................................................................... 13 5.3 Secondary Data ........................................................................................................................ 14 5.4 Quantitative Data .................................................................................................................... 14 5.5 Qualitative Data ....................................................................................................................... 15 5.6 Analysis of Secondary Data ................................................................................................... 16 5.7 Quantitative data analysis ....................................................................................................... 17 5.8 Qualitative Data Analysis ....................................................................................................... 17 6. SECTION I: RELEVANCE, EFFECTIVENESS AND EFFICIENCY OF THE PROJECT ..... 19 6.1 Project Relevance ........................................................................................................................ 19 6.2 Effectiveness and Efficiency .................................................................................................... 21 7. SECTION II: IMPACT AND PERFORMANCE OF SO1. IMPROVE LITERACY OF THIRD GRADE CHILDREN ................................................................................................................... 24 7.1 Profile of Literacy Abilities in Spanish ................................................................................. 24 7.2 Reading comprehension in Spanish ...................................................................................... 25 7.3 Regressions: Gains in Spanish Literacy Abilities ................................................................ 26 7.4 Profile of Ixil Literacy Abilities .............................................................................................. 27 7.5 Profile of Literacy Abilities in K´iche´ ................................................................................. 29 8. SECTION III: IMPACT AND PERFORMANCE OF SO1. IMPROVEMENT OF LITERACY IN SECOND GRADE CHILDREN ........................................................................................................ 32 8.1 Spanish Literacy Abilities of Second Grade Children ...................................................... 32 9. SECTION IV: PERFORMANCE IN INTERMEDIATE RESULTS OF S01 ............................. 36 9.1 Quality of Literacy Instruction (MGD 1.1) ......................................................................... 36 Mid-Term Evaluation IDEA Project SCI P a g e 2 | 81 9.2 Educational Staff ....................................................................................................................... 37 9.3 Student attention (MGD 1.2) ................................................................................................ 37 9.4 School attendance (MGD 1.3) .............................................................................................. 40 9.5 Other results related to SO1 ................................................................................................ 42 9.6 Equity in Learning ..................................................................................................................... 45 10. SECTION V: PERFORMANCE SO2. INCREASE IN HEALTH AND DIETARY PRACTICES ................................................................................................................................................ 48 10.1 Food Preparation and Storage (MGD 2.1) ......................................................................... 48 10.2 Health and nutrition practices and knowledge (MGD 2.2) ............................................ 49 10.3 Access to water and sanitation services (MGD 2.3)........................................................ 49 10.4 Access to preventive health and interventions (MGD 2.4) ............................................ 50 10.5 Access to food preparation equipment and instruments (MGD 2.5) .......................... 50 10.6 Fundamental Result for SO2: Alliance with local organizations and community groups ..................................................................................................................................................... 51 11. SECTION VI: INTERNAL AND EXTERNAL FACTORS THAT AFFECT RESULTS ....... 53 12. SECTION VII: SUSTAINABILITY .................................................................................................. 57 13. CONCLUSIONS ............................................................................................................................... 59 14. RECOMMENDATIONS .................................................................................................................. 61 15. LESSONS LEARNED ........................................................................................................................ 63 16. SUCCESS STORIES ........................................................................................................................... 64 17. ANNEXES ........................................................................................................................................... 66 17.1 Description of the children’s evaluation tool .................................................................... 66 17.2 Inter-interviewer reliability .................................................................................................... 67 17.3 Characteristics of the students in the sample ................................................................... 68 17.4 Spanish Reading Abilities by Sample Group ....................................................................... 69 17.5 Regression of the dependent variable for gain in the percentage of letters read correctly in Spanish between the control and intervention groups ......................................... 69 17.6 Regression of dependent variable gain in percentage of correct Spanish MUW between the control and intervention groups .............................................................................. 70 17.7 Regression of the variable gain in percentage of correct Spanish decoding ............... 72 17.8 Regression of the dependent variable gain in the percentage of fluidity (wcpm) in Spanish .................................................................................................................................................... 74 17.9 Regression of dependent variable gain in Spanish accuracy ........................................... 76 Mid-Term Evaluation IDEA Project SCI P a g e 3 | 81 17.10 Regression of the dependent variable gain in the percentage of correct comprehension in Spanish .................................................................................................................. 78 17.11 Comparative Data from results of the student interviews BL and MTE, Literacy Environment .......................................................................................................................................... 80 17.12 Health and Attention by Sample Group ............................................................................. 81 Mid-Term Evaluation IDEA Project SCI P a g e 4 | 81 1. ACKNOWLEDGEMENTS The mid-term evaluation is an opportunity to recognize and express appreciation for the hard work carried out by the entire team composed of MOE authorities, teachers and school directors, community leaders, parents and the professionals at Save The Children, who share a common goal of generating immediate and lasting change in the lives of the children of Guatemala and whose joint efforts and talents have begun to demonstrate positive results. We are profoundly grateful to the children who participate in the IDEA Project, for being willing to share their feelings and their experiences. With their efforts at such an early age they are ushering in hope for a better future. This study would not have been possible without the work of the ADOC team. Thus we recognize the technicians who carried out the field work as well as the professionals who designed the study, analyzed the results and reported on them in this document. And finally we are grateful to Save The Children: their Program Director, Mauro Tartaglia, for the worthy labor carried out for the wellbeing of Guatemalan children. Special thanks goes to Dr. Lucrecia Méndez, IDEA Project Manager, for promoting children’s rights and for providing us with the opportunity to observe and demonstrate the great results that have been achieved. And to the Research and Learning Director, Clara Pava, for her support in achieving the study’s objectives. Appreciation also goes to the monitoring and evaluation team of the IDEA Project in the central and field offices for their support in coordinating and carrying out the study. ADOC Team Mid-Term Evaluation IDEA Project SCI P a g e 5 | 81 ACRONYMS AND ABBREVIATIONS ADOC Asociación de Desarrollo Organizacional and Comunitaria ATLAS.TI Software Tool for analysis of qualitative data BL Baseline COCODES Community Development Council CODEDE Departmental Development Council CSB Corn Soy Blend CTA Technical Administrative Coordinator ECD Early Childhood Education EMT Mid-term evaluation FE Final Evaluation GF Focus Group IDEA Investment for Educational Development in the Highlands INCAP Institute of Nutrition of Central America and Panama INE National Statistics Institute L1 Primary Language L2 Secondary Language BL Baseline ML Midline MAD Minimum Acceptable Diet MGD Mc Govern Dole Food MESAN Better Healthy Schools MOE Ministry of Education MOH Ministry of Health MTE Mid-term Evaluation MUW Most Used Words M&E Monitoring and Evaluation NGO Non-Governmental Organization PTA Parent Teacher Association PP Percentage point Q Symbol for Guatemalan money (Quetzales) SAN Food and Nutritional Security Mid-Term Evaluation IDEA Project SCI P a g e 6 | 81 SC Save the Children SESAN Secretariat for Food and Nutrition Security SEGEPLAN General Secretariat for Planning of the Presidency SO Strategic Objective TDR Terms of Reference TM Metric Tons UNDAF UN Development Assistance Framework UNICEF United Nationals Children’s Fund USAID United States Agency for International Development USDA United States Department of Agriculture wcpm Words correct per minute Mid-Term Evaluation IDEA Project SCI P a g e 7 | 81 2. EXECUTIVE SUMMARY Scope of the Mid-Term Evaluation The mid-term evaluation of the Investment for Educational Development of the Highlands – IDEA – Project was carried out from May to August 2016 and covered the project implementation period from October 2014 to May 2016. The purpose of the evaluation was to determine the quality of implementation, achievements and/or progress to date, to identify areas which need improvement and to generate recommendations for the remaining implementation period. The MTE also determined the relevance, efficiency, effectiveness, performance, impact and sustainability of the interventions in terms of the project’s two strategic objectives. Context The IDEA project was designed to improve literacy levels and increase the use of better food and health practices in primary school age children in the department of El Quiché using products and funds provided by FAS. The project was expected to benefit at least 40,000 children in 261 schools of six municipalities of El Quiché. During the first year it focused on interventions in the Ixil area (made up of three municipalities: Nebaj, Chajul and Cotzal). In the second year the project expanded to the northern area of El Quiché covering the municipalities of Uspantán, Sacapulas and Cunén. Both areas are among the most marginalized regions of the country where access to basic education and health and nutrition services provided by the government or NGOs is limited. The main activities of the project are: Distribution of school materials and supplies, strengthening of teachers’ capacity and knowledge, preparation of books and reading material, activities to promote reading, delivery of school feeding, capacity building at the local, regional and national levels, training in food preparation and storage, construction/rehabilitation of school infrastructure, distribution of deworming medications and vitamins. The IDEA project has strong and successful involvement and collaboration with the Ministry of Education at the central, departmental, municipal and local levels and with the Ministry of Health at the departmental, municipal and local levels as well as with local NGOs, parent-teacher associations, municipal governments and private enterprise. Methodology Following the sample design from the M&E Plan for the MTE, a sample was obtained for quantitative evaluation of 815 third grade students for the longitudinal analysis and 573 children from second grade for cross-sectional analysis. Additionally 137 teachers and 59 school directors were interviewed in 62 schools in the six target municipalities. The methodology for data measurement and collection selected by the project for the BL, MTE and FE included an oral evaluation (one on one) composed of five components distributed over five sub-evaluations: letter recognition, word recognition, reading fluency and accuracy, and a combination of questions for reading comprehension, related to paragraph fluency and accuracy, environment and literary resources. The MTE included a literature review of secondary data, the majority of which were internal project documents that provided information about design and background, the initial situation (BL Report), Mid-Term Evaluation IDEA Project SCI P a g e 8 | 81 progress reports and arguments which backstopped the findings of the MTE. External sources were used as means of comparison of the results obtained with indicators measured by other studies including official national studies. The quantitative data were analyzed statistically using regression analyses for different variables. A comparison was made of the changes between cohorts of the control and intervention groups. For the comparison of reading abilities between boys and girls from both areas, a comparison of medians by t￾tests was used, assuming an unequal variance between samples. The summary statistics accompanied by t-tests were utilized to analyze students’ performance in each of the reading sub-evaluations. A large volume of qualitative data was obtained from interviews and focus groups and was transcribed and subsequently, categories and codes were created to analyze the information through semantic maps using the Atlas.ti software. Principal results The Mid-Term Evaluation revealed that in the implementation period from the baseline survey to the present, the IDEA Project has generated significant changes in the school age children, in the teaching, and in the community, municipal and national contexts. The longitudinal study carried out with third grade children demonstrated that they had significant gains in Spanish language reading ability. The children of the intervention area showed greater progress. Reading comprehension was the area with greatest improvement. In the intervention area, of the children who had not been able to read, 23.7 percent were now able to read with understanding and 11.4 percent were classified as beginning readers. In the Control area less progress was observed. Of the children who were not able to read, 4.7 percent were now able to read with understanding, and 22.7 percent had become beginning readers. The ability to read in native languages also showed progress. Greater progress was seen in the children who read in the Ixil language. The results of the cross sectional study with second grade children in the MTE, compared with the results of second grade children studied in the BL, showed that the reading abilities profile is much greater now. The achievements in terms of children’s reading abilities can be associated with the IDEA project intervention. It is greater in the intervention area given that the project has been present there for more time. The actions promoted by the project which support parents and leaders, the promotion of reading activities, access to reading material and school supplies, and professionalization of the teachers are giving the expected results. In the MTE it was confirmed that 100% of the schools in the Intervention and Control Areas have access to IDEA project foods. In March, 40,484 school age children received daily school feeding in the schools. In July, the number increased to 43,000 children. The foods provided by IDEA are complemented with the cash transfers that the MOE provides to the parent organizations. Only 5 percent of the children are absent in the schools participating in the IDEA project. This indicator has increased substantially given that in the BL, 20% of the children had absences. School enrollment is an indicator that is slipping at the national level and this pattern was also seen in the areas and schools attended by the project. It was noted that 9 percent of the children drop out of school each year. The results demonstrated that the IDEA project is on track towards sustainability supported by the “Manual for preparing a sustainability strategy”, “Guide for the use and maintenance of school infrastructure” and the “Guide for parents” that promotes and facilitates parent participation in their children’s education process. This facilitates implementation of key actions that are achieving high and inclusive community participation, active involvement of educational and municipal authorities, schools with appropriate methodology, materials and equipment, and especially, teachers who are trained to teach reading. Mid-Term Evaluation IDEA Project SCI P a g e 9 | 81 Short term reduction of hunger is one of the most relevant results seen in the MTE. This has contributed to improving children’s school attendance, has significantly improved their attention level and motivation in class, and involves parents, community leaders and municipal authorities. The main recommendations to improve or increase the project’s impact and sustainability are:  It is recommended that monitoring and follow up of teachers during class be increased to ensure appropriate application of what they have learned in the diploma course they have taken. And all teachers, especially the new ones, should be included in the training given that on a yearly basis, teachers rotate grades.  Strengthen the cash transfer system of the Ministry of Education for school feeding. One way to do this would be to support the legal initiative for the “School Feeding Law” presented by the MOE to Congress. This law would promote the provision of school foods by small producers and thus the consumption of local foods which would support the local economy.  It is recommended that a more in-depth study be carried out to determine whether the foods provided by the project are culturally acceptable, if they are adapted to the traditional diet of the rural population and if they will be permanently available after the project ends. The study should provide recommendations on which foods would be most appropriate depending on the culture of the area.  It is recommended that practical tools be created so that the various stakeholders increase and practice their abilities to seek and manage means to make the project activities sustainable.  Continue the processes initiated to scale up the application of the Parents’ Guide, the methodology for Enhancing Reading and the diploma program for professionalizing the educators to teach reading in Spanish as well as the local languages. Mid-Term Evaluation IDEA Project SCI P a g e 10 | 81 3. SCOPE OF THE CONSULTANCY The mid-term evaluation of the Investment for Educational Development in the Highlands Project – IDEA – covers the period from October 2014 to May 2016. The purpose of the evaluation was to determine the relevance, efficiency, efficacy, performance, impact and sustainability of the interventions implemented under the two strategic objectives. ADOC, in response to the objectives presented in the Terms of Reference, focused the evaluation on determining the quality of program implementation, achievements and/or progress, in order to generate recommendations that would facilitate improvement. Specifically the evaluation focused on:  Determining the achievements to date based on the stated objectives and goals, identifying areas that need improvement based on the evaluation questions.  Identifying the external and internal factors that have contributed to, or affected the results, which facilitated determination of lessons learned, best practices and challenges.  Identifying changes or reorientation of strategies for achievement of objectives and goals.  Addressing critical factors for IDEA Project performance and sustainability of the interventions. . Mid-Term Evaluation IDEA Project SCI P a g e 11 | 81 4. CONTEXT 4.1 Project Background Save The Children (SC) in Guatemala is implementing the Investment for Educational Development in the Highlands– IDEA project, which was designed to improve the literacy levels of school age children and to increase the use of best practices for feeding and health in primary school age children in the department of El Quiché. SC uses donated commodities and funds provided by FAS, in the implementation of this school education project. The project was designed to benefit at least 40,000 children in 243 schools of six municipalities of El Quiché. During the first year, SC focused IDEA Project interventions in the Ixil area (composed of three municipalities: Nebaj, Chajul and Cotzal). Starting in the second year, the project expanded to the northern area of El Quiché, covering the municipalities of Uspantán, Sacapulas and Cunén. These areas are two of the most vulnerable territories of the country, where access to basic education, health and nutrition services provided by the Government or NGOs, is extremely limited. The project’s strategic objectives are: SO1: Improve literacy levels in school age children:  Increase children’s reading and comprehension ability in Spanish, Ixil and K’iche’ by improving support by parents and the community for reading activities, increasing adequate access to reading materials and school supplies and supporting the efforts of Ministry of Education (MOE) staff and teachers for facilitating instruction for reading activities in the classroom;  Improve children’s school attendance by addressing the issue of hunger in the short term by means of delivery of school feeding (breakfasts); and  Improve children’s school attendance by implementing community sensitization activities to inform about the benefits of education and to increase children’s enrollment in school. SO 2: Increase the use of food and health practices:  Increase knowledge of health and hygiene, nutrition practices and good food preparation and storage practices, and provide tools and equipment for food preparation and storage; and  Improve the physical infrastructure of schools to support positive behavior changes in health for children when building and rehabilitating school infrastructure, including kitchens, storage areas, latrines and hand washing stations. The project carries out the following activities:  Distribution of/support in: materials and school supplies, strengthening teachers’ capacity and knowledge, preparation of books and supplementary reading materials, reading camps, reading buddies, reading marathons, delivery of school feeding, enrollment campaigns, training of trainers for community activities, reading awareness workshops for members of the community, reading fairs  Capacity Building: at the local, regional and national levels  Training: food preparation and storage  Construction/Rehabilitation: latrines and hand washing stations Mid-Term Evaluation IDEA Project SCI P a g e 12 | 81  Distribution: deworming medications, vitamins and minerals  Construction/Rehabilitation: kitchens and infrastructure, improved stoves and trainings for the parent teacher associations. The IDEA project has strong and successful collaborative relationships with the Ministry of Education, Ministry of Health, local NGOs, and parent teacher associations at the community level. Coordination with the Ministries of Education and Health is as follows:  Departmental Education District: Ensure that the interventions proposed by the IDEA Project are included within the annual operating plan and that they are implemented according to what was established. This includes the participation of quality educational services providers in administrative processes and in training activities, and guaranteeing that the required supplies and materials are provided without interruption in the intervention areas (Nebaj, Chajul, Cotzal, Uspantán, Cunén and Sacapulas) during project implementation.  Health Area Bureaus in Quiché and the Ixil area: Will ensure that the interventions proposed in the IDEA Project are included in annual operating plans (delivery of deworming medicines, vitamins and minerals)  Coordination with the Municipal Districts: Support the implementation of project interventions by ensuring adequate delivery of high quality and culturally appropriate educational and health and nutrition services  Community Organization: Ensure that the proposed interventions take place at the community level to increase nutrition, education and health services by means of involvement of leaders and parents. IDEA seeks to improve student attendance and attention by means of better nutrition and health habits and by increasing access to school feeding, respectively. The “Promote Literacy” program supports the quality of instruction inside and out of school by means of training for teachers, community reading activities, creation of materials, reading evaluations, and literacy sensitization activities at the community level. The IDEA Project seeks to improve students’ reading comprehension so that by the end of the project, at least 50% of the children in second grade can read and understand the textbooks for their academic grade in Spanish. When compared to a prior evaluation of their abilities, it is expected that this will mean that at least twice as many children will be able to read and understand what they read. 4.2 Key project partners and stakeholders Save The Children implements the IDEA Project in conjunction with the Ministry of Education, which is its primary partner. This partnership is based on the Cooperation Agreement signed in April 2015, in which each organization’s individual as well as shared responsibilities are outlined. SC coordinates the planning process and implementation of the interventions with six Bureaus of the MOE: DICONIME – Bureau for National and International Agreements for the Ministry of Education; DIGEBI - Bureau of Intercultural Bilingual Education; DIGECADE- Bureau for Educational Quality Management; DIPLAN – Bureau for Educational Planning; DIGEFOCE – Bureau for Strengthening of the Educational Community; DIDEDUC –Departmental Education Bureau of El Quiché. Additionally it coordinates the nutrition and health interventions with the Ministry of Health– MOH– through the two Health Area Mid-Term Evaluation IDEA Project SCI P a g e 13 | 81 Bureaus for Ixil and Santa Cruz del Quiché, and it is part of the Departmental Commission for Healthy Schools lead by the DIDEDUC and MOH. In order to ensure the success of the project and to promote institutionalization of the interventions, SC works closely with representatives of the educational community of the 261 schools where the IDEA Project is implemented. This includes community leaders, COCODES and religious leaders. With regard to key, fundamental aspects for building sustainability into the activities, SC has established cooperation and partnership mechanisms with the public-private sector through: Asociación Chajulense, Bank of Central America, Colgate Palmolive, two Municipalities (Cunén and Uspantán), University of San Carlos – Guatemala, and the Equine Association. 5. RESEARCH PLAN 5.1 Key research questions The Mid-term evaluation will address critical factors regarding the Project’s performance and thus will respond to key questions in the following categories: i. Relevance ii. Effectiveness and efficiency iii. Impact and performance iv. External and internal factors that affect the results v. Sustainability and scalability 5.2 Methodology for the Mid-term Evaluation The project has a broad Performance Monitoring and Evaluation plan that complies with FAD regulations (7DFR part 1599.13). This plan includes the system for monitoring the process and the design of the Project’s Baseline, Mid-term and Final Evaluations. SC planned that the three evaluations would use the same research design and instruments. According to the M&E plan, the mid-term evaluation should be carried out by an external consultant in 2016 and its scope would be to measure as many third grade students as possible in 2016 who were also sampled in the baseline survey. The M&E plan includes the expected representative sample for each evaluation: BL - 1260 students; MTE - 1,108 students and FE - 945 students, assuming an attrition rate of 25% from the baseline to the final evaluation and a similar number of students who will be lost during each stage. For the MTE a sample of children in the same grade was used as a common and cost-effective method for estimating the impact of the project in the first three grades of primary school without having to take a sample from each grade. The baseline cohort of second graders is the “sample” for the first grades of primary school. The impact in the second grade students is a good approximation of the impact of the project in all three grades since it provides follow up of the students, up to the beginning of fourth grade. Following the sample model from the M&E plan in the MTE the estimated sample size was of 1,108 children in third grade who were evaluated in the BL and who were in the 35 schools of the Intervention Area (Ixíl) and 35 schools of the Control Area (K’iche’). The BL did not achieve the sample of 1,260 but only 953 children were measured so that the expected sample for the MTE was 953 children. Nevertheless, the attrition rate was greater than expected and Mid-Term Evaluation IDEA Project SCI P a g e 14 | 81 the sample was reduced to 815 children from 62 schools. The reasons for this attrition were: seven schools from the control group and one from the intervention group that were measured in the BL decided not to continue in the project. Other causes were: 6.7% of the children evaluated in the BL dropped out of school, 3.9% did not attend class on the day of the evaluation and 0.4% refused to be evaluated in the MTE. Within the study a sample of 573 second grade students was included to evaluate their Spanish reading level. These children were selected at random in the schools where there were more than 10 children. The sample included 50% girls. The methodology for measurement and data collection selected by the project for the BL, MTE and FE was as follows: an oral evaluation (one on one) consisting of five components distributed in five sub￾evaluations: recognition of letters (vowels and consonants), recognition of words (reading the most used words), fluidity and accuracy when reading (words per minute read correctly and total percentage of the paragraph read correctly) and a combination of reading comprehension questions, related to the fluidity and accuracy of the paragraph. This last part was only carried out with those students who could read independently (reading comprehension). All the instructions were given in the child’s native language. The text for measuring reading in the Ixil language was standardized and adjusted to the three variations of the language that are present in the communities which were evaluated. The student evaluation was accompanied by an interview about their socioeconomic situation, educational experience and the literacy environment in their home. Questions were included about knowledge and behaviors related to health, nutrition and hygiene. The student’s evaluation as well as their interview, were carried out in K’iche’ and Ixil, the native languages in these areas. Focus group discussions with teachers, parents, parent associations, educational authorities and community leaders were also held regarding their perception of project activities, the main results and to hear recommendations for improving project effectiveness. To ensure reliability of the information, double data entry was carried out for 12% (n=109) of the student evaluations (one interviewer interviewed while the other observed and both entered their data into the system). See annex 17.2. 5.3 Secondary Data The ADOC team carried out a literature review of secondary data, the majority of which were internal project documents. These included: Evaluation Plan, records and data base from the project’s M&E system, baseline survey report, community plans, agreements between SC and partners, notebooks and reports from field visits, and school files, among others. The following key documents were also reviewed: a) Third progress report on achievement of Millennium Development Goals for 2010 prepared by the General Secretariat for Planning of the Presidency of Guatemala – Segeplan￾b) UNESCO Documents: SERCE Study 2008 c) National Development Plan K’atun: Our Guatemala 2032 d) Document on Educational Challenges in Guatemala, Guatemala Social Programs 2015 e) USAID country strategy for development cooperation f) Fourth National Census on public sector height of first graders of the Republic of Guatemala. 5.4 Quantitative Data For the quantitative sample in the MTE a total of 815 third grade students and 573 second grade students were evaluated and 137 teachers and 59 directors were interviewed in a total of 62 schools in the six target municipalities (Intervention Area and Control Area). See Tables 1, 2 and 3. Mid-Term Evaluation IDEA Project SCI P a g e 15 | 81 For the quantitative analysis, students were evaluated on five reading abilities and the literacy environment. The same methodology, tools and level of difficulty were used as in the BL. The evaluation instrument of Enhancing Reading for Students can be found in Annex 17.1 Additionally an interview was carried out with teachers asking for their perceptions about students’ attention. School directors were asked about school conditions, schedule for provision of food for the children, number of new and multi-grade teachers, and organizations that support the schools. Table 1. Number of participants in MTE interviews Interview MTE Goal MTE Achievement Interviews with third grade students Total 953 815 Interviews with teachers Total 186 137 Interviews with school Total 62 59 Evaluation with children in second grade Total 620 573 Source: MTE Data Base, IDEA, June 2016 Table 2. Number of schools and students that participated in the study by municipality Source: MTE Data Base, IDEA, June 2016 Table 3. Average number of students per school and schools with more than 20 students in third grade Intervention Control Average number of students per school 13 14 Number of schools with more than 20 students in third grade 15 12 Source: MTE Data Base, IDEA, June 2016 5.5 Qualitative Data The MTE designed and applied qualitative techniques by means of open interviews and focus groups to learn from key project stakeholders about their perceptions, opinions, recommendations for the project and relevant aspects about the context that influence the teaching-learning process. Observation techniques were also used for the school installations and classrooms. The qualitative study, was carried out with a homogenous sample of cases. The following informants were included: Municipal mayors, CTAs, teachers, community leaders, parents of beneficiary children. Qualitative information was collected from six mayors through open interviews, 12 CTAs (two from each municipality) through group interviews, 19 second and third grade teachers including school directors through group interviews, 53 fathers and 61 mothers. Both groups of parents participated in focus groups. Mid-Term Evaluation IDEA Project SCI P a g e 16 | 81 i. Selection criteria for the key informants Schools: Priority was given to the schools which were part of the quantitative sample. One school where the work dynamic and parent-teacher participation was excellent, one where the work dynamic and parent￾teacher participation was inadequate and one where the work dynamic and parent-teacher participation was average. The three schools selected in the Intervention Area were: Sumal Grande in Nebaj, Ilom in Chajul, and Pulay in Cotzal. In the Control Area three schools were randomly selected: San Siguán in Cunen, Trapichitos in Sacapulas and Laguna Danta in Uspantán. This selection was made in conjunction with the Save The Children team. The selection of participants for the focus groups for fathers and mothers was made based on the following criteria: Fathers and mothers with children in the school and who are members of the parents’ and community leader committees; mothers who participate in food preparation, two community leaders who participate in the project. Six focus groups were held in: Chajul, Cotzal and Nebaj (Intervention Area) and six in Sacapulas, Cunen and Uspantán (Control Area). In order to ensure that women’s opinions were heard, separate focus groups for men and women were convened. The selection of teachers to be interviewed was made based on the following criteria: teachers from second and third grade, those who have participated in the project from the beginning, those who have been trained by the project. Four teachers and the director of each school were selected. In the case of the CTA who were interviewed, those who participate in the project were selected. ii. Qualitative Data Collection For qualitative data collection, four previously validated tools were used: a) interview guide for mayors, teachers, CTAs, b) focus group guide to learn the opinions of mothers, fathers and community leaders, c) observation guide to evaluate the situation of school infrastructure, the use of educational methodologies in the classroom, verification of food storage conditions in the school (storage areas), the condition of the kitchens and basic sanitation in the schools; d) observation guide for evaluating the practices learned in the trainings provided by the project: food preparation, hand washing, tooth brushing clean surroundings. All the tools were validated in the school of Pamaxán in Cotzal. The methodology and tools used were approved by SC. Prior to qualitative data collection, participants’ permission was obtained for collecting data and recording meetings. The information was collected by facilitators who were trained on: the role of facilitator and observer; correct use of the guides; the objectives of the MTE; the institutional policies for the current evaluation; and more information about the IDEA Project. The focus groups were carried out at times that were convenient for the fathers and mothers. The teachers were interviewed during the school day. The CTA were interviewed in the municipal seats of Nebaj and Cunen. Each mayor was interviewed in his or her respective municipality. 5.6 Analysis of Secondary Data Mid-Term Evaluation IDEA Project SCI P a g e 17 | 81 The secondary data sources were analyzed to understand the project’s design and background, to understand its starting point (BL report), to understand the project development process, the evidence of progress, to find arguments to back up the findings of the MTE as well as means for comparison of the results obtained with indicators measured by other studies including official national reports and to identify the coherence and response of the IDEA Project interventions with the priorities defined in the National Education Plan of the current government. 5.7 Quantitative data analysis The data were analyzed with statistical methods, regression analyses with different variables: gender, socio-economic status and the literacy environment in the home and the community. Comparisons were carried out for the changes between cohorts between the control and intervention groups. Comparison of the reading abilities of the children in both areas will be carried out by means of t tests, assuming an unequal variance between the samples. The summary statistics, accompanied by the t scores were used to analyze the performance of the students in each of the reading sub-evaluations. 5.8 Qualitative Data Analysis A large volume of qualitative data was obtained from the interviews and focus groups. This was subsequently transcribed, and then categories and codes were created to analyze the information by means of semantic maps using the Atlas.ti software. Atlas.ti is a useful tool for qualitative analysis of large volumes of textual data. This tool facilitated the activities required for qualitative analysis and interpretation given that it organizes, regroups and manages the material to be analyzed in a creative and systematic way with techniques such as segmentation of the text in passages or quotes, coding, organizing and structuring of data by topics and subtopics, or the writing of comments and annotations. The following criteria for accuracy and quality were used: dependence, credibility, transference and confirmation. This required that the coding be:  Open or first level comparing units to create categories and codes  Axial or second level to compare categories that allow interpretation of the data, development of themes or patterns  Selective to codify the generation of theories, narratives and explanations Mid-Term Evaluation IDEA Project SCI P a g e 18 | 81 Photo: Girl, Amacchel School RESULTS OF THE MID TERM EVALUATION SECTION I: RELEVANCE, EFFECTIVENESS AND EFFICIENCY OF THE PROJEC Mid-Term Evaluation IDEA Project SCI P a g e 19 | 81 6. SECTION I: RELEVANCE, EFFECTIVENESS AND EFFICIENCY OF THE PROJECT 6.1 Project Relevance IDEA Project actions are aligned with the MOH 2016-2020 Strategic Plan, and its fundamental principles: Transparency, Inclusion, Dialogue and Social Participation, Linguistic and Cultural Ownership, Multiculturalism and Interculturalism, Education as a Right, and Equity and Equality. The project’s alignment with the strategic guidelines of the current MOE Strategic Plan can be seen in actions such as: promotion of inclusion of the groups that are least covered by the system, which aligns with the strategy of: Equal opportunities for all. Modernization of the educational system is another MOE strategic direction. The schools present present a delay in modernization. In the rural area they are far from having access to technology for education. The limited professionalization of the teachers means that the methodology, educational material and pedagogical approaches being used are obsolete. The innovations promoted by the project are strongly supporting modernization. The educational tools for strengthening reading ability, the formal training of teachers, the methods and educational materials, the distribution of school supplies, learning materials and backpacks support the educational quality that is an important part of the MOE Strategic Plan. Achieve adequate and well equipped schools is another component of the MOE Strategic Plan and the project contributes to it through the rehabilitation/construction of school infrastructure, and equipping kitchens and storage areas. This is a contribution that adds value to the beneficiary schools which are in the areas least attended by the state. The educational material for children and for the teachers also support this strategy. Education as an inalienable right is another component where the project and Save The Children as an institution that promotes children’s rights, advocates with national and local authorities so that children can enjoy this right to education and they promote this right with local leaders and especially parents. Commitment and participation of everyone to support improved education is another strategic component of the MOE and is strengthened by the participatory actions promoted by the project. In addition to the support that the project provides for the MOE, the project strengthens the achievement of the Peace Accords which, although they were signed in 1996, still represent a national agenda towards reducing the structural causes that lead to the exclusion and inequality of large population groups who live in poverty and extreme poverty. The IDEA Project is also aligned with the “National Development Plan, el K’atun: Our Guatemala 2032” in terms of educational priorities. This Plan defines the country’s development over the next 20 years. The IDEA Project is also aligned with the programmatic priorities of the United Nations Development Assistance Framework -UNDAF- especially with regard to Social Development. It shares the commitment to development of the same target population: boys, girls, women and indigenous populations and strives for inclusive and sustainable development, opening spaces for a multicultural and diverse society. The IDEA Project coincides with the USAID NATIONAL STRATEGY FOR DEVELOPMENT COOPERATION - GUATEMALA 2012-2016, specifically in Development Objective Two that seeks greater economic growth and greater social development in the western highlands. USAID seeks results Mid-Term Evaluation IDEA Project SCI P a g e 20 | 81 in education focused on improved quality and access to education (RI 2.3). Its goals are: Better reading ability for children in primary school. USAID’s support for reading and writing is focused on improving the education models and on strengthening the quality of basic education with the long term goal of increasing the number of students in the early grades who read at the level established in the standards1 . The different parties to the project are: The MOE authorities in the Central offices, in the project area (CTA) and in schools: Directors and Teachers. Additionally the municipal governments represented by the mayors and finally the Community leaders who represent the communities and the households from which the beneficiary students come. The central office of the MOE authenticates the diploma training program that the teachers receive about the Reading Enhancement methodology. The MOE authorities trust that with the project’s intervention there will be important improvements in the key educational indicators that will orient education policies and programs in the country. The Technical Administrative Coordinators –CTA-, directors and teachers are aware of the agreements and contracts that have been signed, they are organized, participate actively in the planned actions and are willing to help accomplish the objectives. The teachers and directors recognize that the project’s methodology directly contributes to the achievement of the objectives aimed at integrating the country’s cultural and linguistic diversity and to improve the reading and nutrition indicators of school children. The parents as well as the leaders understand that the project’s contribution goes beyond mere distribution of food, school supplies and improvements to the school infrastructure. They recognize the value of the project with regard to improving their children’s reading abilities and their nutrition as they relate these two benefits to having an impact on their children’s lives. The Municipal Mayors see the project as an opportunity for improvement in their municipality and thus have signed cooperation agreements (political, logistical and resources) in order to also contribute to achievement of the objectives. The project is based on the Norms and Guidelines of the Ministry of Education and complements the policy of free school feeding and the policy of improving the infrastructure and sanitation of MOE schools. The project complements the MOE’s educational quality actions. Their principal actions complement the National Reading Program - “Let’s Read Together”. Although the organized community leaders do not always have formal plans, they know which improvements the communities need and among them are child protection, educational quality, improved school infrastructure and decreased hunger. For this reason they see the project as a complement to their labors. At the beginning of 2016 there was a change in Municipal Governments and the project continued implementation during this transition. During the political campaign the project stayed on the margins of proselytizing activities and once the new authorities took possession of their posts, close coordination was begun in order to seek complementarity for the benefit of the communities. The project has not lost relevance. Its objectives, plans, strategies and activities continue being relevant and appropriate in the current context which has actually changed very little. Some indicators have improved such as community organization, community participation in the educational process. Other indicators continue the same, such as poverty, low educational quality, low investment in development, the lack of plans and actions that promote sustainable development. Other indicators have gotten worse, 1 USAID. ESTRATEGIA DE COOPERATION PARA EL DESARROLLO DEL PAÍS - GUATEMALA 2012-2016. https://www.usaid.gov/sites/default/files/documents/1862/Guatemala_CDCS_Spanish.pdf Mid-Term Evaluation IDEA Project SCI P a g e 21 | 81 such as primary school coverage, violence, decreased public investment and the political crisis. In this context the project is to beacon of hope for the children, their parents, the leaders and local governments. 6.2 Effectiveness and Efficiency The team that implements the project is technically very efficient and is characterized by many years of experience in development. They have well-recognized experience in management of programs and projects focused on health, nutrition and education in the Guatemalan context. They recognize and are deeply sensitive to the conditions of children, especially in the rural areas and they have significant experience in managing US funded multi-year programs. The management team has negotiation skills which when added to Save The Children’s positive reputation allows them to achieve high levels of coordination and the establishment of networks and associations with government, the private sector, and cooperation agencies and enables them to involve key local stakeholders. The entire project team is notably committed to the achievement of project objectives. The field team is fluent in the local language, and thus maintains good communication with the population. Another strong point is the ability the team has for developing methods, tools, and contextualized educational materials adapted to various Mayan languages. The project’s coordination with the Guatemalan government through the Ministry of Education is successful. Coordination is maintained with the General Bureau of Bilingual Intercultural Education – DIGEBI- with the Bureau of Educational Community Strengthening –DIGEFOCE-, with the Bureau of Educational Quality Management – DIGECADE -, with the Educational Planning Bureau – DIPLAN- and with the Departmental Bureau –DIDEDUC in El Quiché. The project coordinates with the Ministry of Health by means of the Ixil and Santa Cruz del Quiché Area Health Offices, Health Centers in the six municipalities and with the Departmental Commission of Health Schools. Close coordination is maintained with the Municipal Governments enabling the mayors to provide valuable contributions. For example the municipality of Cunen provides office space for the project, the Municipalities of Cotzal, Cunen, Uspantán and Sacapulas provide transportation for the foods, material for improving the roads that lead to the schools and materials for construction of storage areas. Based on advocacy actions, the private sector has joined efforts with the project establishing alliances to achieve the objectives. The main relationships with the private sector are: the Fundación Equinos has provided a donation of Story Books for reading banks in the beneficiary schools, BAC bank has provided a donation of more than 100 computers for the M&E system, COLGATE has provided donations of oral hygiene kits for the children in the target schools, coordination with the Eye Unit of Guatemala facilitated eye exams, treatment and glasses for those children that need them, the Technological Institute of Advanced Studies of Uspantán supported the construction of a hand washing station in schools, the Asociación Chajulense coordinated the use of a warehouse for food storage, and coordination with Radio la Chaparra facilitated the transmission of radio messages. The teachers have been trained in the “Enhancing Reading” methodology, by means of a diploma program validated by the MOE and the University of San Carlos. The effectiveness of the training can be seen in the children’s growing reading abilities and the ability of teachers to apply modern, creative, entertaining, innovative and contextualized techniques thus breaking traditional teaching paradigms. (See Section II on reading ability results) Mid-Term Evaluation IDEA Project SCI P a g e 22 | 81 Another relevant factor is the ability of the teachers to teach children to read in their native languages. This is important because reading methodology and material in the local languages is limited and in the case of the Ixil language, its variations make the situation even more complex. With the training received, teachers’ enthusiasm has grown, they have created learning communities, they are more active in extracurricular activities that promote reading (camps, festivals, marathons, etc.) and have improved relationships with parents motivating them to participate in achieving the objectives and they are continuously using adult education methods to communicate with them. The project has a Monitoring and Evaluation system based on the results framework which records, measures and analyzes the process and results indicators. It records all the activities that are carried out and the total number of beneficiaries per activity. Using this data various reports are obtained in a timely manner and these are disaggregated by gender and by component, by school and by geographic area. Based on the results, the Directors, specialists and coordinators make decisions focused on achieving the goals and objectives. The monitoring is done by means of electronic controls backed up by physical evidence. The work team is very competent at data management and thus the information that is processed is of high quality, reliable, confidential, punctual and timely. At this time the project is ready to use to a virtual platform for monitoring and report generation. The use of this technology will be a plus for the monitoring system given that the data base will provide reports in real time, more rapidly and with better connection of variables. Mid-Term Evaluation IDEA Project SCI P a g e 23 | 81 RESULTS OF THE LONGITUDINAL STUDY SECTION II: IMPACT AND PERFORMANCE OF SO1:IMPROVEMENT OF LITERACY IN THIRD GRADE CHILDREN Photo: Children in Santa Avelina School Mid-Term Evaluation IDEA Project SCI P a g e 24 | 81 7. SECTION II: IMPACT AND PERFORMANCE OF SO1. IMPROVE LITERACY OF THIRD GRADE CHILDREN 7.1 Profile of Literacy Abilities in Spanish Literacy abilities are conceived of as the process of capacity and skill development reached by the student, which result from the combination of their literacy knowledge, knowledge, expression and comprehension skills, cognitive, linguistic and emotional habits and attitudes by means of direct contact with, and enjoyment of literary works, to be able to establish assessments and associations with regard to literature. Graph 01 shows the results of the students in the baseline and in the mid-term evaluation with regard to their literacy abilities in both the area where the intervention began in 2015 (Intervention Area) and in the area where the intervention began in 2016 (Control Area). In the Graph it can be seen that abilities have increased and although the Control Area continues to present better conditions in the majority of abilities, progress in the Intervention Area is significant. The literacy abilities where progress is greatest in closing the gap between the two areas are: letter recognition (4 percentage points) and fluidity (1.9 correct words). Additionally, in reading comprehension in Spanish the gap not only was closed but the average result of the students in the Intervention Area passed that of those in the Control Area in 1.5 percentage points (pp). Graph 01. Comparison of the Spanish Reading Abilities Profile in the BL and in the MTE Source: Evaluation of third grade children Value -P for the differences in the Intervention and Control areas in the mid-term evaluation~10%, *5%, **1%, ***0.1% An additional analysis shows the great gains in literacy abilities in the Intervention Area (See Graph 02). In spite of the fact that the Control Area also improved in reading abilities, the gain found in the Intervention Area is greater, which indicates the success of the IDEA Project. The difference between % letters correct in Spanish*** % MUW correct in Spanish*** % correct decoding in Spanish*** Fluidity (wcpm) - Spanish*** Accuracy - Spanish*** % Spanish correct comprehensi on*** Intervention area BL 55.2 26.7 21.7 4.3 18.4 8.6 Intervention area MTE 81.6 65.1 55.2 31.3 53.6 35.8 Control area BL 74.2 52.8 44.3 10.8 39.9 21.5 Control area MTE 86.2 79.4 67.3 33.2 68.7 34.3 Mid-Term Evaluation IDEA Project SCI P a g e 25 | 81 the increase in literacy abilities between the Intervention and Control Areas from the baseline to the Mid￾term evaluation is significant to 0.1% for all the abilities evaluated. The gains across all the abilities are notable. The literacy ability with greatest gains for the children of the Intervention Area was most used words (MUW) where there was an increase of 38.4pp and the one with least gains was in letter recognition with an increase of 26.4pp. Graph 02. Comparison of the gains found between the BL and MTE in the Intervention and Control Areas Source: Evaluation of third grade children. Value -P for the difference between the Intervention and control areas in the mid-term evaluation~10%, *5%, **1%, ***0.1% 7.2 Reading comprehension in Spanish Reading comprehension reflects children’s ability to understand what they read both in terms of the meaning of the words in a text as well as overall understanding of what they are reading. Graph 03 shows a notable increase in the third grade children who have become readers with comprehension in the project’s Intervention Area. In spite of the fact that in the Control Area there is also an increase in this category of readers, the difference is greater in the Intervention Area, which can be associated with the success of the project. There was also a change in the number of children who do not read, with an increase in beginning readers and readers with comprehension with greater increases in the Intervention area than in the control area. While in the BL 78.8% of the children in the Intervention Area did not know how to read, in the MTE this percentage was reduced to 43.6% and the percentage of children who read with understanding went from 3.7% to 27.4%. In the Control Area, 57% of the children were not able to read in the BL and this percentage was reduced in the MTE to 24.5%. Additionally, the percentage of children who read with comprehension in this area increased from 14.8% to 19.5%. % letters correct in Spanish *** % MUW correct in Spanish *** % correct decoding in Spanish*** Fluidity (wcpm)- Spanish ** Accuracy Spanish * % Spanish correct comprehensi on *** Control Area 12.0 26.6 23.0 22.4 28.9 12.8 Intervention Area 26.4 38.4 33.5 27 35.2 27.2 Mid-Term Evaluation IDEA Project SCI P a g e 26 | 81 Graph 03. Distribution of third grade students in Spanish reading comprehension levels Source: Evaluation of third grade children2 7.3 Regressions: Gains in Spanish Literacy Abilities In order to carry out a more rigorous analysis of the association between the project and the gains in the literacy abilities of the children, a linear regression analysis was carried out. The analysis used the socioeconomic characteristics of the children evaluated, the result in the baseline and a variable that indicates whether the child is part of the Intervention or Control areas as independent variables. The analysis used the gain obtained in each reading ability as a dependent variable (for more details on the regressions carried out and the variables which were used, see Annexes 17.5 to 17.10). This type of analysis allows us to analyze whether there is a significant association between the greater gains obtained in the children of the Intervention Area compared to the Control Area even controlling for different variables that have an influence on this result. The results show that being part of the Intervention Area is positively associated with a greater gain in the results of three of the six abilities that were evaluated: letters, fluidity and reading comprehension, controlling for socioeconomic characteristics and the results of the baseline of the student. For letters and fluidity the result is significant to 5% and for reading comprehension to 0.1%. Although causality cannot be affirmed between the project and the results due to the evident differences between the comparison and intervention groups in the baseline and the fact that at the moment of data collection for the mid-term evaluation both areas had the intervention, this result is indicative of the positive effect that the IDEA Project has on increased literacy abilities of the children who participate in the interventions. 2 The Non-readers are defined as students with a fluency of less than five words per minute. The beginning readers are defined as those students who do not stop but answer less than 80% of the questions correctly. Readers with comprehension are those who read the paragraph and correctly answer 80% or more of the questions related to the reading. 43.6 78.8 57.0 24.5 28.9 17.5 28.2 55.9 27.4 3.7 14.8 19.5 Intervention Area MTE Intervention Area BL Control Area BL Control Area MTE Those who don't read Beginners Readers Mid-Term Evaluation IDEA Project SCI P a g e 27 | 81 7.4 Profile of Ixil Literacy Abilities Besides analyzing the change in abilities of the children who were evaluated in the BL for Spanish language, the Mid-term evaluation also looked at the gains in their native language in each of the areas. The results of the survey in Ixil, the native language of the children in the Intervention Area, are presented below. Graph 04. Results of the BL and gains in native language (Ixil) reading abilities of third grade students Source: Evaluation of third grade children Value -P for the difference between the BL and MTE ~10%, *5%, *1%, ***0.1% Graph 04 presents the gains in the different reading abilities that were evaluated. The increase in children’s reading abilities in the Intervention Area are obvious. The abilities that showed greatest gains were: most used words (33.5 pp) and accuracy in the reading passage (31 pp). Additionally, the percentage of children who can read a passage that is appropriate for their age in Ixil increased by 32.5 pp. It should be noted that the change in the results from the BL to the MTE in all the reading abilities is significant to 99.9%. % letters correct in Ixil*** % MUW correct in Ixil *** Reading in Ixil*** Fluidity (wcpm)- Ixil*** Accuracy-Ixil *** % Ixil correct comprehensio n *** Gain in MTE Ixil 19.3 33.5 32.5 21.4 31.0 28.2 BL Ixil 34.7 8.4 7.8 1.4 7.9 2.0 Mid-Term Evaluation IDEA Project SCI P a g e 28 | 81 Graph 05. Distribution of third grade students by reading comprehension levels (Ixil) Source: Evaluation of third grade children in Ixil3 Graph 05 shows reading comprehension in the Intervention Area. As can be seen in the Graph the number of children who cannot read was reduced by 32.4 pp. Additionally, the fact that in the BL there were no children with reading comprehension and that in the MTE, 28% of the children can read with comprehension, demonstrates the contribution of the IDEA Project in the intervention schools. 3 The Non-readers are defined as students with a fluency of less than five words per minute. The beginning readers are defined as those students who do not stop but answer less than 80% of the questions correctly. The readers with comprehension are those who read the paragraph and correctly answer 80% or more of the questions related to the reading. 59.8 92.2 12.3 7.8 28 0 Intervention Area MTE Intervention Area BL Those who don´t read Beginners Readers Mid-Term Evaluation IDEA Project SCI P a g e 29 | 81 7.5 Profile of Literacy Abilities in K´iche´ Graph 06. Results of the BL and gains in third grade students’ reading abilities in their native language (K´iche´) Source: Evaluation of third grade children in K´iche´ Value -P for the difference between the BL and MTE ~10%, *5%, **1%, ***0.1% Graph 06 shows the results of the BL and the increase in reading abilities in the Control Area where the children’s native language is K´iche´. Except in the case of letter recognition in K´iche´ which decreased in comparison to the BL, the results show a positive change in literacy abilities. The abilities with greatest gains were reading in K´iche´ (26.6 pp), most used words MUW (23.9 pp) and accuracy in the reading passage (20.3 pp). % letters correct in K´iche´ *** % MUW correct in K´iche´ *** Reading in K´iche´ *** Fluidity (wcpm)- K´iche´ *** Accuracy￾K´iche´ *** % K´iche´ correct comprehensio n *** BL K´iche´ 72.1 28.3 25.3 5.1 20.8 4.4 Gain MTE K´iche´ -9.1 23.9 26.6 11.4 20.3 6.3 Mid-Term Evaluation IDEA Project SCI P a g e 30 | 81 Graph 07. Distribution of third grade students by reading comprehension (K´iche´) levels Source: Evaluation of third grade children in K’iche´ 4 The data in Graph 07 show modest progress in reading comprehension for the children in the Control Area in their native language. In spite of the fact that there was a reduction of 26.6 pp for children who do not read, and an increase in the percentage of beginning readers of 25.9 pp, the percentage of readers with comprehension only increased by 0.7 pp. The project activities have only been going on for a short time in this area and thus the small change in readers with comprehension is not surprising. In the final evaluation the project’s influence on students’ K’iche’ reading comprehension (the predominant language in the Control Area) can be more clearly measured. 4 The Non-readers are defined as students with a fluency of less than five words per minute. The beginning readers are defined as those students who do not stop but answer less than 80% of the questions correctly. The readers with comprehension are those who read the paragraph and correctly answer 80% or more of the questions related to the reading. 48 74.6 49.3 23.4 2.7 2 Control Area MTE Control Area BL Those who don't read Beginners Readers Mid-Term Evaluation IDEA Project SCI P a g e 31 | 81 RESULTS OF THE CROSS-SECTIONAL STUDY SECTION III: IMPACT AND PERFORMANCE OF SO1:IMPROVEMENT OF LITERACY IN SECOND GRADE CHILDREN Photo: Children in Santa Avelina School Mid-Term Evaluation IDEA Project SCI P a g e 32 | 81 8. SECTION III: IMPACT AND PERFORMANCE OF SO1. IMPROVEMENT OF LITERACY IN SECOND GRADE CHILDREN During the Mid-term evaluation an additional random sample of second grade students was selected to look at the profile of abilities that they present and compare it to the second grade students surveyed in the baseline before the intervention began. These children have had the opportunity to be part of the project for more than a year and are a good additional indicator of the influence that the project is having in the early grades. 8.1 Spanish Literacy Abilities of Second Grade Children Graph 08. Comparison of the profile of reading abilities of second grade children in the MTE by sex. Source: Evaluation of second grade children in Spanish Value -P for the difference between the sexes ~10%, *5%, **1%, ***0.1% Graph 08 shows that the reading ability with greatest presence in the second grade children was the percentage of letters read correctly. Fluidity (wcpm) is the lowest ability with a result of 22.6 and 24 words correct per minute for boys and girls, respectively. The different between the results for boys and girls is not statistically significant. % letters correct in Spanish % MUW correct in Spanish % correct decoding in Spanish % Children who read the passage in Spanish Fluidity- (wcpm)- Spanish Accuracy￾Spanish % Spanish correct comprehensio n Male 76.8 68.2 62.3 57.1 22.6 54.6 52.7 Female 74.4 64.9 58.2 55.9 24.1 49.8 52.3 Mid-Term Evaluation IDEA Project SCI P a g e 33 | 81 Graph 09. Comparison of the profile of reading abilities for second grade children in the intervention area in the BL and MTE Source: evaluation of second grade children in Spanish Value -P for the difference between the sexes ~10%, *5%, **1%, ***0.1% The second grade children evaluated in the MTE showed greater literacy abilities than the second grade children evaluated in the BL. This is indicative of the influence and effectiveness of the IDEA Project with regard to the children’s abilities. The results that are statistically greater between the BL and the MTE in the second grade group are: recognition of letters in Spanish (difference of 20.8 pp), correct MUW (difference of 40.3 pp), correct decoding (difference of 38.3 pp), children read the passage (difference of 35.3 pp), fluidity (difference of 19 correct words), accuracy (difference of 33.6 pp), reading comprehension (difference of 44.4 pp). % letters correct in Spanish *** % MUW correct in Spanish*** % correct decoding in Spanish*** % Children who read the passage in Spanish *** Fluidity (wcpm)- Spanish *** Accuracy￾Spanish *** % Spanish correct comprehensi on *** MTE 2nd grade 45.6 66.6 60.3 23.3 52.0 52.5 57.0 BL Intervention 55.2 26.7 21.7 4.3 18.4 8.6 21.2 Mid-Term Evaluation IDEA Project SCI P a g e 34 | 81 Graph 10. Distribution of second grade students according to reading comprehension levels (Spanish-separated boys and girls) Source: evaluation of second grade children in Spanish5 One of the most important indicators for the project is the MGD SO1, which in the case of the IDEA Project shows the percentage of children who, at the end of second grade, can read and understand an appropriate text for their age. Graph 10 presents the results for this indicator. The progress that the children have had over the life of the project is evident. In the baseline only 8.5% of second grade girls and 9.9% of boys could read with comprehension. In the mid-term evaluation this percentage is much greater reaching approximately 20% for boys as well as girls. It should be noted that the indicator shows the result for both areas (intervention and control) and in the Control Area the project had not had much time to work at the time of data collection. The results in the Final Evaluation will show more clearly the effect in the two areas after the project has had more time in the Control Area. 5 The Non-readers are defined as students with a fluency of less than five words per minute. The beginning readers are defined as those students who do not stop but answer less than 80% of the questions correctly. The readers with comprehension are those who read the paragraph and correctly answer 80% or more of the questions related to the reading. 70.1 69.1 44.1 42.9 21.4 21 36.3 36.6 8.5 9.9 19.6 20.6 2nd grade BL (girls) 2nd grade BL (boys) 2nd grade MTE (girls) 2nd grade MTE (boys) Those who don't read Beginners Readers Mid-Term Evaluation IDEA Project SCI P a g e 35 | 81 SECTION IV: PERFORMANCE IN INTERMEDIATE RESULTS OF SO1: IMPROVE LITERACY OF SCHOOL AGE CHILDREN Photo: Girl, Visiquichum School Mid-Term Evaluation IDEA Project SCI P a g e 36 | 81 9. SECTION IV: PERFORMANCE IN INTERMEDIATE RESULTS OF S01 9.1 Quality of Literacy Instruction (MGD 1.1) The quality of literacy instruction is demonstrated in the different activities and resources that promote reading and which can be seen in Graph 11 It can be noted that a greater number of the activities that promote reading are being implemented in the Intervention Area due to the time that the IDEA Project has been working in this area. Additionally, children’s participation is high: 100% of the children between first and third grade participated in the friends of reading and reading camp activities in all the beneficiary schools. Graph 11. Practices and resources for improving reading in the children evaluated in the MTE Source: Interview with Directors In terms of the materials and resources received, the schools have received 68,620 books and supplementary reading materials from SC. The CTA from both areas indicated that with the material that has been received the classrooms look ready for reading. They commented, “There are more posters, materials and resources for conducting the classes, the teachers use materials in L1 and L2. The teachers have updated the posters in their rooms – there are no more posters that say, ‘Welcome to the 1970 class year.’ Now the teachers are more systematic, energetic and flexible.” These comments were confirmed in the observations where it was noted that the classrooms have teaching material in Spanish as well as Ixil or K’iche´. In terms of the use of materials, the observations and interviews demonstrated good use of the materials. Of the teachers, 81.35% use and have visible these tools for developing reading abilities, 93.22% have a well prepared classroom, with photographs, children’s homework and posters, and 13.55% of the teachers did not have them out and visible and indicated that they only bring them out when they are going to use them. The teachers interviewed commented that SC has facilitated the teaching process by means of the teaching material received which is contextualized and in native languages. Specifically in the Control Area, the teachers said that with the teaching materials they received they have improved the way they % Book bank in school? % reading buddies implemented in school? % Reading camps implemented ? % This school has a library? % The library allows students to check out books? Control Area 30.4 82.6 34.8 84.6 90.9 Intervention Area 63.6 100 100 72.7 70.8 Mid-Term Evaluation IDEA Project SCI P a g e 37 | 81 teach. Now they are more creative and innovative. Nevertheless, the CTA of this area has observed that 25% of the teachers do not replicate what they have learned in their classrooms (they don’t prepare educational materials or put into practice what they have learned). 9.2 Educational Staff According to SC’s monitoring data from the first semester of fiscal year 2016, a total of 1,092 teachers and directors who teach first, second, and third grade have been trained in the reading abilities of recognizing letters and phonemes. At the beginning of 2016, 19% in the Control Area and 15% in the Intervention Area were new teachers who had not received the benefits of the project. In the Control Area there are a greater number of teachers who have received SC trainings than in the Intervention Area (100% and 94%) with 6% of the teachers in the Intervention Area not having been trained. The teachers recognize the value of the trainings, the materials and the resources received from SC. According to the teachers, the trainings given by SC strengthen their knowledge and they adapt the tools and materials to the grade that they are teaching and to the local language. The teachers appreciate the teaching material they have received and state that it is adequate and contextualized to the area and the level of the children. The teachers in the Control Area suggest that the trainings be given more frequently, that they have more follow up in the classroom and that all elementary teachers participate in the training rather than just those teaching first to third grades. This is because every year teachers might be assigned to teach a different grade. The teachers in the Control Area reported that not all the reading camps that had been programmed have been carried out. The teachers from the Intervention Area, especially those from the Ilom school, claim that the trainings have been very dynamic with clear objectives and are taught by capable facilitators. They said that the trainings have always been given in Spanish and that it would be more important to teach them in the local language, although they recognize the complexities of this given the variations of the Ixil language. They also noted that the lack of punctuality on the part of the participants is a problem in the trainings. 9.3 Student attention (MGD 1.2) i. Attention in class Graph 12 shows the increase in children’s attention in class from the perspective of the teachers consulted in the MTE. According to the teachers, currently 47% of the students pay attention during class while in the BL only 5% were paying attention. In the MTE the teachers state that 40% of the children try to pay attention while in the BL only 15% of the children tried to pay attention. With regard to the percentage of children who do not pay attention, the MTE shows that only 13% of the children do not pay attention. This indicator experienced a notable decrease compared to the results of the baseline that reported that 80% of the children did not pay attention. The teachers were asked what factors they thought influenced this improvement in children’s attention in class and all of them agreed that the school feeding provided to the children is the most important factor as it helps the children to feel less hunger, to have energy and thus they are attentive and motivated. “They are no longer hungry. This was a problem that prevented them from paying attention …,” said several teachers. The teachers also said that the techniques and tools that they are applying to improve students’ reading and writing, help to keep the children interested, motivated and active during class. The materials that the teachers prepare are creative, illustrative and motivate participation. Mid-Term Evaluation IDEA Project SCI P a g e 38 | 81 Graph 12. Percentage of children who pay attention during class Source: Interview with children ii. Reduction of hunger In the BL survey it was noted that 49% of the students indicated that they weren’t hungry during the school day and 51% stated that they were very hungry or always hungry during the school day. Graph 13. Children’s perception about their hunger in the BL compared with the MTE Source: Interview with children Value -P for the difference between the areas~10%, *5%, **1%, ***0.1% % of children who pay attention % of children who try to pay attention % of children who do not pay attention BL 5 15 80 MTE 47 40 13 Control Area MTE Control Area BL Intervention Area BL Intervention Area MTE*** % of children who feel hungry occasionally or continually during the school day 7.4 9 22.8 4.5 Mid-Term Evaluation IDEA Project SCI P a g e 39 | 81 The children in the MTE indicated that they continuously felt less hunger during the school day in both areas. The percentage was higher in the Intervention Area where it was reduced by 18.3 pp, compared to the Control Area where there was a 1.6 pp reduction. The data show that the children in the Control Area felt less hunger at the time of the BL, possibly associated with the fact that this area has better socioeconomic conditions than the Intervention Area. Nevertheless, in the MTE it can be seen that the children of the Intervention Area feel less hunger compared to the BL and compared to the children of the Control Area. According to the mothers, the children go to school happily because besides learning and playing, they are going to eat. They said that they have reduced the food rations they give to the children at home because they receive food at school and come home less hungry. The mothers reported that they serve fewer meals at home. The children are tired when they come home and many times don’t eat dinner; or they get up early to go to school and don’t eat breakfast because they know they will eat at school. In some cases when they come home they have eaten at school and don’t have lunch but rather start immediately on their household chores. The project’s contribution to the household economy is significant but leads to concern about sustainability and the risk that it implies for the children when they no longer receive food at school. iii. Access to school feeding Some 40,484 school age children receive meals daily in the schools (breakfast, snack, lunch) as part of the IDEA Project. This provides them with the minimum acceptable diet which takes into account the inclusion of the different food groups. The meals are complemented with the MOE contribution and parents’ contributions. Access to food doubled in the two areas and the quantity and quality of the food also improved. The change in food access from the baseline to the MTE is notable as observed in Graph 14. In the baseline, 50% of the schools in the six municipalities where the IDEA Project is implemented stated that they had food. In the MTE it was noted that 100% of the schools in the Intervention and Control Areas had access to the IDEA Project foods. Graph 14. Percentage of schools where the IDEA Project is implemented that stated they had foods Source: Interview with directors LB EMT % of schools where IDEA Project is implemented that have food 50 100 Mid-Term Evaluation IDEA Project SCI P a g e 40 | 81 Access to foods for students is an extremely important contribution give that the nutritional status of school age children presents stunting or chronic malnutrition, according to the Fourth National Census of Height in public sector first grade students6 carried out in 2015. According to this census, “the prevalence of chronic malnutrition or growth faltering in height for age, in the Republic of Guatemala has decreased in 1.3 percentage points from 1986 to 2001, in 4.2, from 2001 to 2008 and in 8.0 percentage points from 2008 to 2015, but it is still higher than expected given that none of the regions, departments or municipalities are considered free from nutritional damage. All present growth faltering rates are well above the 2.5% expected for a normal population according to WHO recommendations.” The census reveals that the national average prevalence of chronic malnutrition is 37.6%. Among the departments with prevalences higher than the national average we find Totonicapán with 59.5%, and Quiché with 55.3%, which are classified as having very high vulnerability to food-nutrition problems. The stunting rates in the municipalities where the IDEA Project works are: Chajul, 72.0% (this is among the 10 municipalities with the highest prevalence of chronic malnutrition), San Juan Cotzal, 69.1%, Cunén, 66.3%, Nebaj, 61.9%, Uspantán, 55.9% and Sacapulas, 55.9%. All are classified as having Very High Stunting Prevalence. The census shows an analysis by ages and reveals that the differences in prevalence of chronic malnutrition among the different ages, shows that these are greater for eight and nine year old children with prevalences of 45.7% and 54.2%, respectively, while for six and seven year olds the prevalences were 30.9% and 32.7%, which reveals that the older the first grade boys and girls are, the greater their degree of nutritional damage. Additionally, the census reports that the children whose native language is Spanish reach a prevalence of 26.4%, while those children whose native language is not Spanish, reach 51.0% stunting prevalence or chronic malnutrition. 9.4 School attendance (MGD 1.3) i. Absences related to illness By means of the monitoring records, consultations with teachers and parents, the MTE established that only 2% of the children were absent from school. According to the CTA of the Intervention Area, the actual number is a little higher and in spite of the fact that in the last year attendance increased, there is a 5% absentee rate in IDEA Project participating schools. This is a notable improvement from the BL with an absentee rate of 20%. While in the BL, absences were only attributed to student illness (diarrhea, respiratory infections and conjunctivitis), in the MTE those who were consulted added other causes such as farm chores in planting and harvest seasons, handicaps, and especially in the Intervention Area mothers associated drop outs and absenteeism to school violence or bullying. The information provided in the interviews and focus groups about decreased absenteeism gives reason to associate the project intervention to the improvement of this indicator. The fathers and mothers who were consulted said that, “the children only miss one or two days of school; they don’t like to miss 6 http://www.sesan.gob.gt/index.php/noticias/censo-talla-2015/75-informe-ejecutivo-censonacional-2015/file Mid-Term Evaluation IDEA Project SCI P a g e 41 | 81 school.” The parents in both areas agreed that attendance had improved due to the provision of school feeding, school supplies, backpacks, and tooth brushing kits from the IDEA Project and also due to parents’ encouragement for their children to attend school. The teachers consider that the children don’t like to miss school and are happier; they are less tired and less hungry. The directors and teachers indicated that through meetings they have made parents aware of the importance of education for their children and this has also been accomplished by means of radio programs in Spanish, Ixil and K´iche´. ii. School Enrollment School enrollment is one of the indicators that showed decreases at the national level. In recent years school enrollment has gone down and drop out rates have increased. Of the children who begin school in first grade, only 60% finish sixth grade. An average of 7.71% students abandon primary school each year and 14.86% fail their grade. Generally the children who fail a grade do not continue in school. Primary education coverage continues to decline. In 2009 primary enrollment was almost universal at 98.3%7 , but it began to decrease so much that the net rate of school enrollment in 2013 was 85.4%, in 2014, 82.0%, and in 2015, 81.0%. This pattern is confirmed in the areas and schools attended by the project as can be observed in Graph 15. In the municipalities where the project is carried out, the tendency for enrollment in the last five years is in decline. Graph 15. Initial Enrollment in the Last Five Years in the Municipalities of the IDEA Project Source: ADOC presentation with data from MOE Annual Report 7 Segeplan. Third report on progress toward Millenium Development Goal 2010 Objectives. 0 5000 10000 15000 20000 25000 30000 35000 40000 10 11 12 13 14 Initial Enrollment Last 5 years Intervention Control Mid-Term Evaluation IDEA Project SCI P a g e 42 | 81 Using project monitoring data, it was noted in the MTE that 9% of the children leave school each year. The reasons for dropping out are different in the two areas. While the parents in the Intervention Area attribute it to bullying, the parents in the Control Area say that the cause is agricultural labor and migration. The teachers in the two areas agree that the cause for drop outs are socioeconomic situations and child labor. They do not, however, list bullying as a cause. Specifically in the Intervention Area, teachers mentioned that the school generates expenses for parents who must buy school implements and this is one of the reasons why children abandon school. iii. Community Understanding of Benefits of Education The project aims to increase community understanding of the benefits of education as a key strategy for children’s success in school. This indicator was very low during the BL given that none of the parents consulted could identify a minimum of three benefits of education. During the MTE a significant increase was seen given that 100% of the fathers and mothers consulted identified more than three benefits of education in their children’s lives. They relate education to people’s general progress, with intellectual growth and development, with improvements to children’s health and nutrition and they mentioned that they understood that girls have a right to education. According to the project’s monitoring data, a total of 164 sensitization workshops have been held on the importance of reading with the participation of 8,624 parents and community leaders. In coordination with the DIDEDUC they also carried out three radio campaigns in Ixil and K´iche´ to make parents aware of the importance of sending their children to school. In the focus groups the parents appreciated the project’s contribution to making them aware and they recognize that the support that their children receive motivates them to send them to school. Additionally they valued the strategies for improving reading. For example, they commented on the fact that “the children now go to reading camps instead of going to work in the fields.” 9.5 Other results related to SO1 i. Results in the literacy environment in children’s homes The literacy environment of the home is closely related to the opportunities that children have to practice their reading abilities. To determine the household literacy environment, the third grade children were asked about access to printed material in their homes and their interactions with different members of the household. The first section describes the gains obtained between the BL and the MTE for the children in the Intervention Area compared to the Control Area with regard to the different types of printed materials including: text books, religious books, magazines, newspapers, story books and coloring books. Mid-Term Evaluation IDEA Project SCI P a g e 43 | 81 Graph 16. Printed materials children have in the home Source: interview with third grade children Value -P for the difference from the BL to the MTE ~10%, *5%, **1%, ***0.1% As observed in Graph 17 there are significant differences between the Intervention and Control Area in the increase of coloring books, newspapers and magazines. It should be noted that access to materials was measured by means of a question to the children who were evaluated and thus it is not an exact figure, nevertheless it seems that access to some types of printed material in their homes has improved. Graph 17. People identified by the children as having read or told them stories in the Intervention Area Source: interview with third grade children Value -P for the difference from the BL to the MTE ~10%, *5%, **1%, ***0.1% % Difference for text books % Difference for religious books % Difference for magazines** % Difference for newspapers* ** % Difference for story books % Difference for coloring books*** Gain in Intervention Area -0.2 -0.2 2.4 -13.8 6 -9.5 Gain in Control Area -6.6 -2.1 8.4 12.1 7.9 9.8 % of family members who child saw reading % of family members who read to the child*** % of family members who told stories to the child* % of family members who helped the child to study** Gain in Intervention Area -2.1 10.7 11.3 -3.8 Gain in Control Area -4.6 1.5 5.8 -10.3 Mid-Term Evaluation IDEA Project SCI P a g e 44 | 81 According to Graph 17 the Intervention Area has an increase in the percentage of people who read to children of 10.7% and in people who tell stories, 11.3%. In the Control Area improvement was also seen but was less than in the Intervention area – the gain in percentage of people in the home who read to children was 1.5%, and the increase in people in the home who told the child a story was 5.8%. The differences in the increases between the Control Area and the Intervention Area in these two types of interactions were statistically significant. Additionally, the total number of people who help the child with their studies went down, possibly because the children’s educational level increased (from second to third grade) and the parents are less able to help them. Another factor which could have had an effect is the time of year when the MTE was carried out (beginning of the agricultural season), a time when parents are busier and less able to attend to their children. With regard to the results of the focus groups, the parents who were consulted in both areas indicated that they do not do any reading activities with their children in the home and they were also unable to name any other person in the home who provides this support. Nevertheless, the fathers and mothers of the Intervention Area agreed that the reading camps are the place where their children participate in reading activities. They also mentioned that the camps are conducted by volunteers from the community who are trained by SC in how to do so. Graph 18. Increase in children who report reading activities in their community Source: Interview with third grade children Value -P for the difference from the BL to the MTE ~10%, *5%, **1%, ***0.1% % Someone in the community read to the child % The child read to someone in the community % The child borrowed a story book from somewhere outside home Gain in Intervention Area 2.2 0.1 -0.1 Gain in Control Area 2.8 2.9 2.9 Mid-Term Evaluation IDEA Project SCI P a g e 45 | 81 Graph 18 shows that there is a greater increase in community reading activities in the Control Area than in the Intervention Area. When asked this question, perhaps the children identified the community volunteers as the people who read to them. The data also show that the children in the Control Area are the ones who have most participated in reading in their community. 9.6 Equity in Learning The equity analysis below shows that after having attended the project for more than one year, some of the children’s characteristics that were associated in the baseline with better performance, continue to play an equally important role in children’s reading abilities level. Early Education (ECD) After the IDEA Project began the methodology of Enhancing Reading the gaps in basic and advanced reading/writing skills in Spanish between children who did or did not attend an early childhood education program (ECD) closed. The children who attended an ECD program presented better qualifications with a difference of 1pp in letter recognition and MUW. The skills that showed greatest differences were fluidity and correct decoding (3 pp). All the differences are significant to 10%. Graph 19. Spanish Reading Abilities by ECD in the MTE P-value: ~10% *5% **1% ***0.1% Repetition Graph 20 shows that the children who did not repeat a grade have higher results in Spanish reading abilities in the MTE. The development of reading abilities influences the way in which children progress in the rest of their learning. It can be assumed that the children who improve their reading abilities have a greater chance of passing their grade. The difference in reading abilities is significant to 10%. % letters correct in Spanish ~ % MUW correct in Spanish ~ % correct decoding in Spanish ~ Fluidity (wcpm)- Spanish ~ Accuracy-Spanish ~ % Spanish correct comprehension ~ No ECD 81 64 53 29 55 36 ECD 82 65 56 32 53 36 Mid-Term Evaluation IDEA Project SCI P a g e 46 | 81 Graph 20. Spanish Reading Abilities Related to Repetition in the MTE P-value: ~10% *5% **1% ***0.1% Socioeconomic Level Graph 21 shows that socioeconomic level continues to be an influential factor in achieving Spanish reading abilities in the MTE. The children with better socioeconomic conditions have better performance in basic and advanced abilities. Graph 21. Spanish Reading Abilities by Socio-economic Status in the MTE P-value P-: ~10% *5% **1% ***0.1% % letters correct in Spanis * % MUW correct in Spanish * % correct decoding in Spanish Accuracy Spanish * % Spanish correct comprehension** * Quintiles SES 2.00 74.4 49.9 42.1 38.7 26.5 Quintiles SES 3.00 83.0 66.7 56.5 55.4 41.0 % letters correct in Spanish * % MUW correct in Spanish * % correct decoding in Spanish~ Accuracy-Spanish * % Spanish correct comprehension *** Quintiles SES 2.00 74.4 49.9 42.1 38.7 26.5 Quintiles SES 3.00 83.0 66.7 56.5 55.4 41.0 ~ Mid-Term Evaluation IDEA Project SCI P a g e 47 | 81 SECTION V: PERFORMANCE SO2. INCREASE IN HEALTH AND DIETARY PRACTICES Photo: Poster in Santa Xecaxlut School Mid-Term Evaluation IDEA Project SCI P a g e 48 | 81 10. SECTION V: PERFORMANCE SO2. INCREASE IN HEALTH AND DIETARY PRACTICES 10.1 Food Preparation and Storage (MGD 2.1) SC has focused efforts on training the community staff (parents) in food preparation and storage. According to monitoring system records for 2015, 8,625 women and 1,063 men were trained in food preparation, exceeding the number of people to be trained according to the established target. The trainings were practical and focused on recipes to use the foods donated by USDA, to take advantage of locally produced foods and/or those which are available in the community, complementing the meals with foods purchased using the MOE cash transfers for school feeding. In both areas mother guides were identified to be coordinators of the food preparation process in the schools. The main functions of the mother guides are: to coordinate the mother groups and replicate with them the trainings that they received from SC to ensure that the foods for the children are nutritious and healthy. In both areas the parents are responsible for providing firewood. Each student contributes three pieces of wood per week. In 52% of the schools in the Control Area and 46% in the Intervention Area, weekly menus were observed. In the focus groups the mothers agreed that even though they recognize that the foods given by SC are nutritious, that they are an important contribution, that they support children’s nutrition and stimulate their growth, the fact is that the children get bored with them and complain about always eating beans, rice and CSB gruel. Some mothers shared the difficulty they have in preparing large quantities of food and that occasionally the beans and rice burn or get scorched and are wasted. Additionally they have to arrive at the school very early which affects their household activities. Some mothers dedicate eight hours to food preparation. These factors have caused some mothers to stop helping with food preparation. It was observed in the schools that the mothers implement hygiene practices in food preparation such as: keeping the environment clean and free of animals, they use hair nets and aprons, they use cans or bags for garbage, and they wash hands. With regard to food storage, the parents in the Intervention Area have been trained in food storage practices: installing platforms, warehouse cleanliness and security. They stated that the training had been given in their native language and thus they understood well and are applying what they learned. It was observed that food storage conditions are adequate: clean, orderly storage areas without moisture and no evidence of pests. The conditions of the storage areas are better in the Control Area compared to those in the Intervention Area. All of the schools in the Control Area have their inventory cards up to date while in the Intervention Area only 54% of the schools are up to date; 46% are not up to date. All of the schools that distribute foods have equipment for measuring and have delivery control records for the project’s foods in the schools. The project implemented a guide for estimating foods to orient parents about adequate rations to be distributed on a daily basis. This has reduced food loss. The fathers and mothers expressed satisfaction with the trainings they have received. Some suggestions for improvement were related to variation in weekly menus, control of pests in the kitchens and storage areas, as well as improving the mothers’ skills for cooking large quantities of food. Mid-Term Evaluation IDEA Project SCI P a g e 49 | 81 10.2 Health and nutrition practices and knowledge (MGD 2.2) The project’s school health and nutrition activities include promotion of healthy behaviors and nutritional education for parents, teachers and directors, deworming campaigns, provision of vitamins and minerals. The deworming campaigns are coordinated with the Ministry of Health which assigns staff and delivers the deworming medicine and minerals (especially iron). Through the monitoring data, the MTE confirmed that the project has trained 660 people in childhood health and nutrition and 1810 teachers in good health and nutrition practices. The training that has been provided has influenced children’s hygiene practices. In 53% of the schools evaluated it was observed that the children wash their hands before eating; in 49% of the schools it was observed that the children brush their teeth after eating; in 60% of the schools it was observed that the women wash their hands before preparing the food, and 93% of the kitchens were observed to be clean. The project contributes to school children’s nutrition with the distribution of food in 100% of the schools. They distribute: rice, black beans, wheat flour, CBS and oil. From these foods the school age children obtain the minimal acceptable diet. The food that is given to the school children offers 57.8% of the individual daily kcal needs as established by the Institution of Nutrition of Central America and Panama - INCAP -. Thanks to the project’s coordination with the Ministry of Education, the ministry also provides contributions so that the children have a better diet. The mothers in the two areas recognize that SC has contributed to the health and nutrition of the boys and girls by means of promotion of health practices such as deworming, provision of vitamins and micronutrients. They also recognize that with this support their children have gotten sick less, are happy and contribute to the family’s savings since they don’t have to transport them to seek health care services. The mothers in the Ilom community in the Intervention Area stated that some of the mothers sell the vitamin packets (chispitas) they are given for their children. The IDEA Project staff is part of the Departmental Healthy Schools Commission – CODES –which promotes and guarantees healthy environments, water, sanitation and hygiene actions, and practices to improve and diversify the school aged population’s diet and nutrition. The IDEA Project has promoted and contributed to the formation of the Municipal and Community Healthy Schools Commissions– COMES – and CODES, respectively. 10.3 Access to water and sanitation services (MGD 2.3) In the MTE it was confirmed that 90.9% of the schools in the Intervention Area and 73.1% of the schools in the Control Area have access to water. All of the schools in the Intervention Area and 96.2% in the Control Area have latrines or toilets. With information from the monitoring system it was confirmed that 299 schools have been rehabilitated, 118 schools use better water sources, and 41 schools have improved their sanitation services, latrines and hand washing stations, benefiting 12,785 students. The project delivered eco-filtros to 100% of the schools. Access to water and sanitation services has improved since the project began given that in the baseline it was reported that in the Intervention Area only 34% of the schools used an improved water source and 75% of the schools had bathrooms or latrines, the majority of which were in poor conditions. Mid-Term Evaluation IDEA Project SCI P a g e 50 | 81 10.4 Access to preventive health and interventions (MGD 2.4) In the MTE it was confirmed that 21,147 children have received deworming medicines and micronutrients thanks to the negotiations that SC carried out with the health area. Other actions promoted by the project that support preventive health are the foods and training on nutrition that parents receive. In the focus groups the mothers in the two areas reported that with the preventive health actions, their children don’t get sick as often. 10.5 Access to food preparation equipment and instruments (MGD 2.5) The MTE confirmed by means of observation and the data from the monitoring system that 93% of the schools in the Control Area and 54% of the schools in the Intervention Area have a kitchen. The monitoring data showed that the project has rehabilitated or built 106 kitchens. According to the MTE, 100% of the schools received utensils and equipment for food preparation from the project: table scales, plastic buckets, pitchers, funnels, rope, pots, kitchen utensils, aprons, hair nets. Graph 22. Sample of the type of stoves used in the schools during the MTE Source: Observation data from the schools evaluated in the MTE In the Intervention Area, 70% of the schools used improved stoves (59%) or firewood saving stoves (11%). The remaining 30% cooked over an open fire which is not recommended. In the case of the school kitchens in the Control Area, 83% used improved stoves (41%), or firewood saving stoves (42%) and 17% of the schools in this area cook over open fires which is the least desirable because it uses more firewood, affects hygiene and produces more smoke contamination. % Improved Stove % Firewood saving stove % Cook over open fire Intervention Area 59 11 30 Control Area 41 42 17 Mid-Term Evaluation IDEA Project SCI P a g e 51 | 81 According to the data from the monitoring system, the project provided improved stoves, which benefited 9,446 students. The school stoves have improved given that in the BL only 25% of the schools had stoves in good condition. 10.6 Fundamental Result for SO2: Alliance with local organizations and community groups The IDEA Project has involved various key stakeholders from the beginning of its interventions. These include: community leaders, representatives from the Community Development Council – COCODE-, parents, directors and teachers of the schools. SC has established public-private alliances with seven organizations which include: the Asociación Chajulense (which provide a locale for the main warehouse located in the village of Batzul, Chajul); agreements with two municipalities (Cunén and Uspantán), University of San Carlos of Guatemala, Bank of Central America, Colgate Palmolive and Asociación Equinos. During the MTE, evidence was examined related to the coordination and association with key stakeholders such as 261 commitment letters signed by educational community representatives and leaders to establish the roles and responsibilities for project implementation; Municipal Agreement No. 04-2015 with the mayor’s office of Cunén for provision of office space for the IDEA Project, warehouse for educational materials, transportation of materials and foods, and investment in access to piped water. Commitment letters were available from the municipalities of Uspantán and Sacapulas for the transportation of school supplies and inputs for students and teachers. Ratification of MOE agreements with the new authorities in 2016 were also seen. Mid-Term Evaluation IDEA Project SCI P a g e 52 | 81 SECTION VI: INTERNAL AND EXTERNAL FACTORS THAT AFFECT RESULTS Photo: Chajul Mid-Term Evaluation IDEA Project SCI P a g e 53 | 81 11. SECTION VI: INTERNAL AND EXTERNAL FACTORS THAT AFFECT RESULTS Save The Children has several strengths that have facilitate the success of the IDEA Project. In the first place, it is recognized for its experience and credibility by the Government of Guatemala, especially the Ministry of Education at the central, departmental, municipal and local levels, Municipal Governments, international cooperation entities and it is also recognized by peer organizations for its successful implementation of programs and projects focused on promoting and protecting the rights of children and adolescents. SC’s specific structure, process, policies and procedures facilitate effective implementation of their various programs and projects. Their internal control and monitoring / evaluation systems reflect constant and transparent accountability. Their human resources facilitate project progress because they possess the ideal characteristics: they have a profound understanding of, and sensitivity to, the realities of Guatemalan families, especially those who live in poverty and extreme poverty and who encounter marginalization and limited access to their rights. The work team has experience in the creation of educational models and models that seek food security with the added value of being contextualized, modern and innovative. The work team also is well-versed in interinstitutional coordination with peer organizations and especially with interested parties from the various projects and programs being implemented. They value democratic, consensual and inclusive participation while functioning as facilitators of the process. Another advantage of the field team is that they are able to function in the local languages. External factors that facilitate project success include the willingness and strong participation of the interested parties in the implementation of projects focused on educational quality. The Ministry of Education’s (MOE) acceptance of SC’s innovative contributions and the political will to validate them, put them in practice and scale them up. The community leaders’ interest in improving educational quality and thus their involvement and support. The parents are a key factor for social auditing which in the country’s current socio-political environment is a factor that motivates teachers to increase teaching quality. The big challenges that face the educational system are circumstances that limit achievement of expected results. The educational indices in Guatemala are the lowest in the Central American region. The main indicators for this situation are: a) Primary education coverage continues to decline: The net schooling rate in 2013 was 85.4%, in 2014, 82.0% and in 2015, 81.0%. In 2009, primary school attendance was nearly universal at 98.3%8 . b) Increased drop out rates. Of the children who begin school in first grade, only 60% finish sixth grade and only 39% are the right age when finishing. An average of 7.71% of students abandon primary school every year and 14.86% fail their grade. a. The extremely limited budget for the educational system in general. Of the budget assigned to Education, 70% is used for teachers’ salaries and 29% for administrative staff salaries. Only 1% is invested in infrastructure and projects to improve educational quality. The school snack program is the area that receives fewest resources. The MOE invests Q1.11 (USD 8 Segeplan. Third report on progress in accomplishment of the 2010 Millenium Development Goals. Mid-Term Evaluation IDEA Project SCI P a g e 54 | 81 $0.15) for snack per day per student in urban schools, and schools in the rural and remote areas are assigned Q2.08 per student (US $0.27). b. Minimal and weak school infrastructure. In 2015, no new schools were built and no damaged infrastructure was repaired. c. The lack of teachers and their poor distribution. There is a prevalence of schools where only one teacher covers the entire school population. d. Reading and Mathematics evaluations in 2015 continue to show that students are behind in their management of the expected knowledge and abilities. Reading achievement for 2015 was 25.97 %, while in 2014 it was 26.02 %.9 The SERCE study published by UNESCO in 2008, placed Guatemala close to last place in Latin America and the Caribbean in Math and Reading tests.10 e. Inadequate and limited teacher training. f. The lack of linguistic and cultural relevance in the educational system.11 g. In Guatemala, illiteracy is a reflection of the exclusionary economic, social and cultural environment that has relegated the educational system to adults over 15 years of age, especially in relation to women and youth living in poverty and extreme poverty in the rural areas. Between 2002 and 2011, illiteracy in people over 15 was tending to decrease as it went down from 29.32 to 17.46%. That is, it dropped 11.86 percentage points. Nevertheless, literacy for women is still a challenge. In 2011, three of every 10 women over 14, and one of every 10 between 15 and 24, do not know how to read and write. One of every two indigenous women over 14 do not know how to read and write12 . In addition to the structural problems, the effects of poverty and extreme poverty in which children are living is reflected in high malnutrition and child labor rates. Studies have shown that malnutrition negatively affects enrollment, permanence in school and academic performance, especially in primary school students13 . Another recent factor that puts the project’s results at risk are the high crime rates that force children to be absent from school. The situation of violence and insecurity that characterizes the country is a serious obstacle for progress and democracy and other development processes. At the local level some of the limiting factors are: Conflicts between leaders and teachers in some communities. A reduced number of class days. Strikes and other unofficial activities reduce the number of class days by approximately 32%. Poor school infrastructure: in some cases there are not enough classrooms, they are in poor condition and the land does not belong to the Ministry so they cannot make investments there. Some directors, teachers and Educational Supervisors are also resistant to change. Parents often have to seek work in a variety of ways and thus their accessibility for participating and learning is limited. 2015 was an election year which made it difficult to coordinate with municipal governments as the project avoided getting caught up in the political campaigns. 9 Results published in the page of the General Bureau for Educational Evaluation and Investigation (Digeduca) of the MOE. 10 Study SERCE - UNESCO 2008 11 National Development Plan K’atun: Our Guatemala 2032 12 National Development Plan K’atun: Our Guatemala 2032 pp 168 13 Educational challenges for Guatemala http://programassocialessandratorres.com.gt/2015/06/desafios-educativos-de-guatemala/ Mid-Term Evaluation IDEA Project SCI P a g e 55 | 81 Some internal SC factors that limit project success include: the delivery of materials and foods takes up time for the technical team which limits the time they can spend following up and supporting teachers in the classroom to monitor and advise them on the implementation of the educational methodology, support for reading camps, collection of information to compile lessons learned and good practices, and limited participation in opportunities for coordination, organization and negotiations at the municipal level (COMUDE). Mid-Term Evaluation IDEA Project SCI P a g e 56 | 81 SECTION VII: SUSTAINABILITY Photo: SC Sustainability Manual Mid-Term Evaluation IDEA Project SCI P a g e 57 | 81 12. SECTION VII: SUSTAINABILITY The IDEA Project has a sustainability strategy that is systematized in the “Manual for preparing a sustainability strategy”. This was created with participation of all the main stakeholders who were consulted in January and February 2016. The manual includes the concept of sustainability as well as the steps for the community to take to plan the sustainability of their projects. This manual has good teaching language and can be used by communities without technical support. The project promotes sustainability as a cross cutting them and to date has carried out various actions in the communities, schools, with Municipal governments and authorities from the Ministries of Education and Health in the department seat of El Quiché. Assemblies have been held in the communities to provide information about the project’s purpose. Commitment letters were signed by the communities. Leaders are trained for coordination with other non-governmental and private entities. They were given the “Manual for preparing a sustainability strategy.” Community self- management is promoted. Families are already involved in the construction and rehabilitation of school infrastructure. Volunteerism is promoted in the communities to support reading and writing. Parents’, leaders’ and teachers’ knowledge and abilities with regard to maintenance and preventive and corrective use of school sanitation infrastructure was strengthened. A guide was created to support this. Greater awareness of the benefits of education in parents will make the actions promoted by the project sustainable for families and communities. The project involves various officials form the Municipal governments in the implementation of education, health and nutrition actions as well as in the construction of the school sanitation infrastructure. The project prepared a “Guide for use and maintenance of infrastructure,” which is illustrated to teach water and sanitation infrastructure maintenance, preventive maintenance of school infrastructure, correct use and maintenance of the infrastructure for food preparation and storage. This useful guide has been given to the Ministry of Education, to Directors and teachers of the target schools and to leaders and parents. This guide will ensure the sustainability of the improvements to school infrastructure. The Municipal governments hire teachers and personal for food preparation in some schools and also contribute with transportation of food and materials to the schools. A key action in seeking sustainability is the increase in teachers’ abilities for teaching reading and writing using entertaining and innovative methodologies. The teachers who have completed the training have been certified in the methodology for enhancing reading, with the endorsement of the University of San Carlos of Guatemala through the University Center in El Quiché. This certification increases their credits for the teaching pay scale. The project has influenced and advised the Departmental Bureau of Education for the legalization of land in the name of the State to thus be able to build or improve schools, especially: sanitation infrastructure (toilets and hand washing stations), kitchens, firewood saving stoves and storage areas. The project coordinates with the Healthy Schools Commission for the implementation of actions that ensure healthy and fitting environments for students. Achieving sustainability is not easy. Some factors that limit this are: the limited vision of sustainability held by the key stakeholders in the community: authorities, teachers, leaders and parents. The communities lack plans for local development. They cannot envision their future and thus are always Mid-Term Evaluation IDEA Project SCI P a g e 58 | 81 waiting for outside help. Their organizational structure is weak with limited management capacity and with a hard-to-overcome paternalistic focus. The parents and community leaders have not identified other stakeholders with whom they could coordinate so that the actions of the IDEA Project can continue and they also lack management and negotiation abilities. For the mothers group, sustainability is less visible given that the women have limited local decision making participation and their schooling level is very low. The Municipal governments lack development plans and have limited economic and human resources so investment in education is limited. Additionally, the municipal authorities have limited abilities for seeking resources. MOE investment is limited and is dependent on the policies of whichever government is in office. Potential for scaling up To date the most successful actions which could be taken to the national scale are: a) The Guide for parents used for sensitizing and including them in the educational community to seek the wellbeing of their children. This guide is a compilation of the methods that have been used and was developed in coordination with the MOE. This guide is certified by the Ministry of Education in Resolution DACPI/DIGEACE (Bureau for Accreditation and Certification) No. 0068-2016 26 July 2016. c) The Enhancing Reading methodology can be implemented at the municipal, departmental and national levels by first to third grade teachers. It is a methodology that addresses the strengthening of the five reading abilities that children should acquire. d) The Enhancing Reading diploma program to professionalize teachers in the area of reading, both in Spanish and in local languages. Mid-Term Evaluation IDEA Project SCI P a g e 59 | 81 13. CONCLUSIONS The Mid-term Evaluation revealed that between the baseline survey and the MTE, the IDEA Project has generated significant changes in the school age population, the methodologies used for teaching reading and writing, and the participation and commitment of community leaders, parents, and government stakeholders at the Municipal and National levels. The main conclusions of the evaluation based on the quantitative and qualitative results obtained are shared below. 1. The IDEA project continues strong. Its objectives, strategies and activities are relevant and appropriate in the current context at the national and local level. It is coherent and aligns itself with the norms and directions established in the Ministry of Education’s 2016-2020 Strategic Plan. It complements the efforts to improve school feeding, to provide fitting and healthy environments for children’s learning. In this context, the project brings hope for children, parents, leaders and local governments. 2. A relevant aspect is the involvement and commitment of two municipalities (Uspantán and Cunén) for supporting actions that improve school health and nutrition. This is demonstrated through hiring of teachers, transporting foods from the SC warehouse to school storage areas, contributions for rehabilitating and/or building school infrastructure. 3. SC has established effective mechanisms for coordination with the MOE at the central, departmental and municipal levels for implementation of the main activities, and has achieved ownership of the interventions. 4. The involvement of representatives from the educational community from the beginning of the project was critical for promoting and ensuring their commitment, for defining roles and responsibilities, recognizing the role of all stakeholders, especially parents and teachers, in the process of educating children. The community leaders value participation that is requested rather than imposed with participatory decision making. 5. The IDEA project is aligned with the program priorities of the United Nations Development Assistance Framework -UNDAF- especially in the topic of Social Develop and coincides with the USAID COOPERATION STRATEGY FOR NATIONAL DEVELOPMENT - GUATEMALA 2012-2016. 6. The involvement of stakeholders at all levels - local, municipal, departmental and national -has been successful. At the local level it is seen as an opportunity to participate in the development of their community. They value the fact that participation is requested, not imposed with participatory decision making. 7. The IDEA project is on track with regard to achieving sustainability for the interventions from the community base, strengthening linkages among parents, educators, local and municipal authorities. It has designed and is in the process of validating the “Manual for preparing a sustainability strategy” and the “Guide for use and maintenance of school infrastructure with emphasis on sanitary facilities, storage areas, kitchens and stoves”. For this they have ensured the participation, involvement and contributions of parents, leaders, educators and children in putting together each of these documents. 8. The IDEA project, in conjunction with the MOE through the Bureau for Strengthening of the Educational Community – DIGEFOCE –, contributed technically to the preparation of nine modules for the Guide that promotes parent participation in the process of educating their children. The guide was certified by the Bureau for Accreditation and Certification of the MOE. Mid-Term Evaluation IDEA Project SCI P a g e 60 | 81 9. The results of the longitudinal study of the MTE demonstrated that the children in third grade showed significant and better gains in their Spanish language reading abilities in the intervention area compared to the control area. In the cross sectional study of second graders, it was especially interesting to note that in both areas, those who read with understanding went from 8.5% and 9.9%, to 19.6% and 20.6% for boys and girls, respectively. 10. Reading abilities in the native language showed greater improvements in the intervention area (Ixil); this improvement can be associated with the contributions of the IDEA project given that the government school system has limitations in methodology, material and educators trained to teach reading in the native languages. 11. The educators recognize and value the training process to improve and strengthen reading abilities by means of the Diploma program accredited by the University of San Carlos. From the perspective of the educators this has been an effective strategy because it has facilitated improvement in the instruction process for reading and writing, applying creative, innovative and fun techniques that have been contextualized and are breaking traditional teaching patterns. 12. Hunger reduction in the short term is another of the most relevant results shown by the MTE. This has contributed to improved school attendance, has significantly improved children’s attention and motivation levels in the classroom, and involves parents, community leaders and municipal authorities. This demonstrates that the project responds to the needs of the population given that the statistical data show high percentages of boys and girls who drop out of school, who are frequently absent, and who fail their grade. 13. The rehabilitation/construction of school infrastructure, specifically, toilets/latrines, hand washing stations, kitchens, storage areas and improved stoves, have facilitated the creation of fitting and healthy spaces for children’s learning. 14. The project is effective and efficient. SC’s own structure, process, policies and procedures allow for effective implementation. Their internal control and monitoring / evaluation systems allow for constant and transparent accountability. A key factor for the project’s effectiveness is that it is carried out by a team of professionals who have extensive experience in the field of development with recognized experience in the management of large multi-year programs with funds from the US government. The capacity of the human resources coupled with the Save The Children’s good reputation also allows them to achieve good results in order to establish networks and partnerships with government, private sector and cooperation agencies as well as involving key local stakeholders. 15. The Guatemalan educational system is still facing great challenges which are external to the project but limit the achievement of the expected results. The main challenges in the educational system are: Decreasing primary school coverage, increasing drop out rates, extremely limited budget for the educational system in general, scarce and weak school infrastructure, the lack of teachers and their poor distribution, low student scores at the national level in Math and Reading, inadequate and limited teacher education, a lack of linguistic and cultural ownership of the educational system, general illiteracy in the population over 15 years of age, especially in rural women, plus the poverty and extreme poverty in which children are living which results in high rates of malnutrition and child labor. Mid-Term Evaluation IDEA Project SCI P a g e 61 | 81 14. RECOMMENDATIONS Although the MTE reveals that the IDEA project is being implemented successfully, the ADOC team presents several recommendations focused on improving the process, its sustainability and its impact. The main recommendations are presented below. 1. In order to have better monitoring of the project process and results, it is recommended that they use more virtual technology with all its advantages to incorporate into the monitoring system other important variables such as children’s school attendance, academic failure, overage students, absenteeism, drop out rates and their causes, and although there is no significant difference between boys’ and girls’ reading rates it is recommended that they continue monitoring the right to education and quality education for girls and to highlight the results. It is recommended that they relate the children’s reading progress with general school success (higher grades and greater knowledge). 2. Systematize the successful collaborative working experiences that the project has achieved with the Municipalities of Uspantán and Cunén, to increase the interest of other municipalities that have not yet signed agreements to support the project. 3. Increase support for the children’s home and/or school reading environment with books, magazines, newspapers and other materials given that although more than 60,000 books were given out, the MTE reported that the children lack reading material at home. Activities such as camps that promote reading are very successful but it is recommended that they incorporate other types of competitive activities to promote reading such as poetry, public speaking and story writing. 4. Increase monitoring and follow up with teachers during classes to ensure that they use well what they have learned in the diploma program. Include all the teachers and especially the new teachers in the training given that on a yearly basis, teachers rotate grades. 5. Take advantage of the partnership with the MOE to strengthen the cash transfer system that the Ministry of Education provides to the parents’ organizations for school feeding. One way to do so would be to support the legal initiative for the “School Feeding Law” presented by the MOE to Congress. This law would promote the provision of school foods by small producers and facilitate consumption of local foods which would support the local economy. 6. Carry out an in-depth study to determine whether the foods provided by the project are culturally acceptable, if they are adapted to the traditional diet of the rural population and if they will be permanently available after the project ends. The study should provide recommendations on which foods would be most appropriate depending on the culture of the area. 7. Carry out a categorization of the schools based on the size of the school population in order to determine aspects such as: type and quantity of school infrastructure needed, quantity and type of kitchen utensils and space needed for storage and cooking, number of fathers and mothers to be trained, number of teachers to be trained. 8. In the food estimate tables, take children’s ages into account. The rations should be different depending on the age group so that large portions are not given to small children. 9. Coordinate with SESAN or other agencies that focus on nutrition, especially in Chajul, given that this municipality is among the 10 with the highest rates of chronic malnutrition in children under 7 years of age. Mid-Term Evaluation IDEA Project SCI P a g e 62 | 81 10. Monitor for “Unintended consequences” of the project. For example, in some groups that were consulted, comments such as this were heard: “families now save money on food for their children”, “ now they don’t give them food at home because they eat at school”, “when they come home they aren’t hungry, they go straight to work and go to bed early without eating dinner, they don’t eat breakfast at home because they know they will get food at school”. 11. Create practical tools so that the various stakeholders increase and practice their abilities to seek and manage means to make the project activities sustainable. Continue the processes initiated to scale up the application of the Parents’ Guide, the methodology for Enhancing Reading and the diploma program for professionalizing educators to teach reading in Spanish as well as the local languages. Mid-Term Evaluation IDEA Project SCI P a g e 63 | 81 15. LESSONS LEARNED a. A good analysis and understanding of the context situation, in conjunction with the design of integrated, complementary strategies and effective management of the different coordination levels which are evident in the IDEA Project, make possible the expected results in the reading abilities of children who are in vulnerable groups facing deterioration in the educational process. b. In such complex contexts as those where the project is being implemented, it is not possible to totally associate the changes in school performance (promotion, enrollment and drop outs) to the project. It should be noted that the impact on these indicators depends on external variables beyond the scope of the project and they also require longer term actions. Nevertheless, according to the information collected it can be seen that the integrated actions have achieved very positive changes in students’ reading abilities and school attendance. c. Food is a strategy that promotes school attendance. It is necessary to transform the conditions in the environment by means of complementary actions, avoiding dispersion, to obtain the significant results that are seen in the project. Nevertheless, there are inherent conditions in the context that make children’s participation and school enrollment difficult and which foster drop out rates, such as: violence in the school, child labor, migration, socioeconomic and political situations. d. The educational proposal of integrating health, nutrition and hygiene are relevant for complementing, in an integrated manner, the nutrition status of children. By means of these actions they have also improved school interactions with the community by means of strengthening the capacities of fathers and mothers, and increasing their involvement in their children’s education. e. For better utilization of the food, it is important to develop parents’ capacities for its management and administration. The table for estimation of amounts of foods by weight to determine the number of rations to prepare that was developed by the IDEA Project, is a very useful tool to avoid waste. Additionally, mothers’ knowledge of how to cook large quantities should be strengthened in order to avoid food loss because of the way it is prepared. It should not be assumed that just because they are mothers they know how to cook. For this reason the trainings should take into account the preparation of large quantities of food. f. Within the definition of categories or characteristics of the schools to include, besides their situation and context it would be important to categorize them by the size of the student population in order to define the contributions needed based on the number of children, especially in areas such as: size of the kitchen, type, size and number of kitchen utensils, type and size of stove, number of bathrooms and wash areas, warehouse size and conditions, number of mothers to be trained. Additionally it is important to define the number of children per age group given that the rations are the same for small and large children. These should be differentiated based on the requirements of each age group. Mid-Term Evaluation IDEA Project SCI P a g e 64 | 81 16. SUCCESS STORIES BUILDING TOGETHER A PARTICIPATORY AND INCLUSIVE MODEL OF TEACHING/LEARNING FOR THE BOYS AND GIRLS OF SAN SIGUAN The Official Rural Mixed School of the Aldea San Siguán is located in the municipality of Cunén, Quiché. Ortilia de León is the school director and her desire is, in conjunction with her colleagues, to implement a model. Miss Ortilia, as she is affectionately called, states that, “We are committed to change. I have the privilege of working with a special group of staff who share this dream and we have support from Save The Children. This is the best moment to do this as we have the resources and while they may not be enough, it gives us something to start with.” In the community of San Siguán there are children with handicaps (deafness and blindness), but for Miss Ortilia this is no problem. She shares, “This is part of the innovation that we are working towards, the model that we are promoting and the challenge we have taken on. We want to be an inclusive school. We believe that all the boys and girls have the right to education and the fact that we don’t have a center for special education doesn’t mean that we will not attend them. Of course it requires a double effort on the part of the teachers but we are willing to do this. We also believe that this will encourage other institutions and people to join in this vision. In the sector we want to be the pacesetters and hopefully this will be replicated in the future in the rest of the communities”. For Miss Ortilia de León, the support that they receive from Save The Children in food, educational materials for teachers, girls and boys, personal and dental hygiene accessories, and especially the trainings in methodologies as well as the commitment of her colleagues, Mr. Obed Us and Miss Yaneth de León, have made the conditions for change possible. Miss Ortilia continues, “The children also are in the best moment for this. Their physical and mental health conditions have improved, they are motivated and we need to take advantage of that. Their parents have agreed to participate in this change because they are recognizing the results that we have achieved in a short time as we work in conjunction with Save The Children and the educational community. We also have backing from the CTA of the sector.” Miss Ortilia continues by saying, “Together with Save The Children, teachers, parents, and children, we are building a legacy that will transcend history. It will change lives and will transform our nation.” Mid-Term Evaluation IDEA Project SCI P a g e 65 | 81 DEVELOPING CAPACITIES IN PARENTS AND LEADERS TO MANAGE IMPROVEMENTS FOR THEIR CHILDREN’S WELLBEING Rogelio lives in Aldea Sumal Grande and is the father of several children who are beneficiaries of the IDEA Project. He is also a member of the COCODE and President of the School Committee. He shared about his experiences since Save The Children arrived at their school, “When the IDEA Project came, I immediately joined because I saw the benefits for my children and the children of the community. I encouraged the other parents to also accept the project and together with the teachers we agreed that the project should come to our school.” Rogelio has participated actively in the IDEA Project and has been trained in topics of food storage and community organization for preparation of local management plans. Rogelio worked for a long time in Guatemala City and learned about solar panels. Then he migrated to Santa Cruz del Quiché and had the opportunity to work installing panels. Since he knew how to install and use solar panels he saw an opportunity to join forces with his friends to negotiate the acquisition of some solar panels for the community, and they were able to negotiate with a private enterprise and the municipality for the subsidies needed to have electricity in their community. According to Rogelio, the trainings received from Save The Children have opened their eyes to improve the way that they can present their proposals. He also says that he is more aware of the importance of education and it is his hope that his children will study and get ahead to have a better life than he has. He tells his children that it is important to be humble and be able to serve in whatever ways are possible. He reminds them that if he had had the opportunity to study with all the resources that the IDEA Project provides, they would be living a different life. He hopes with all his heart to see them get ahead as graduated, successful, good men. Mid-Term Evaluation IDEA Project SCI P a g e 66 | 81 17. ANNEXES 17.1 Description of the children’s evaluation tool Student information Examples General Gender, age, language spoken at home, work/chores School Repetition, history, ECD attendance Health Breakfast Reading/writing environment in the home Access to printed publications Materials presents in the home, type of materials Reading activities in the home Presence and percentage of family members who the children see reading and who can do reading/writing activities with the children Reading result Description Spanish Ixil K’iche’ Letters correct Number of known letters/sounds 27 27 37 MUW correct Number of individual words read correctly 20 20 20 Decoding Number of invented words read correctly 20 -- -- Fluidity Number of words in a short story that are read correctly in one minute Accuracy Percentage of words in a short story that are read correctly Comprehension Number of questions (of 10 for each language evaluated) related to the short story that are answered correctly. The questions are divided in: 1 summary question, 5 literal questions and 4 advanced comprehension questions (2 deductive questions and 2 argumentative questions). Mid-Term Evaluation IDEA Project SCI P a g e 67 | 81 17.2 Inter-interviewer reliability To evaluate reliability across interviewers, 12% of the students (109 of 927) were evaluated by two interviewers simultaneously. ANOVA extensive unidirectional techniques were used to calculate the interclass correlation within the pairs of evaluators to measure reliability. Using the Fleiss points of reference for Excellent (ICC>0.75), Good or Regular (0.75>=ICCA>0.4) and Bad (0.4>=ICC), all the literacy results variables showed excellent inter-interviewer reliability. Given that the inter-evaluator reliability was high, we can be sure that the team of evaluators measured the children’s reading abilities in a consistent manner. Subprueba de habilidades de lectoescritura Confiabilidad interevaluador Calificación Conocimiento de escritura en español 0.75 Bueno Conocimiento de escritura en k’iche’ 0.91 Excelente Conocimiento de escritura en ixil 0.85 Excelente Palabras más utilizadas en español 0.76 Excelente Palabras más utilizadas en k’iche’ 0.75 Bueno Palabras más utilizadas en ixil 0.84 Excelente Decodificación de palabras en español 0.80 Excelente Fluidez en español 0.93 Excelente Fluidez en k’iche’ 0.85 Excelente Fluidez en ixil 0.86 Excelente Precisión en español 0.93 Excelente Precisión en k’iche’ 0.90 Excelente Precisión en ixil 0.90 Excelente Comprensión de lectura en español 0.80 Excelente Comprensión de lectura en k’iche’ 0.88 Excelente Comprensión de lectura en ixil 0.82 Excelente Mid-Term Evaluation IDEA Project SCI P a g e 68 | 81 17.3 Characteristics of the students in the sample Background Characteristics Control Communities Intervention Communities Statistical Value General Characteristics Age 9.68 9.57 ~ Attended early childhood education [ECD] (%) 75.73 78.19 Female Gender (%) 43.27 49.68 ~ Repeated 1st grade (%) 35.19 32.90 Repeated 2nd grade (%) 31.22 33.77 Language spoken in the home - Ixil (%) 0.00 69.98 *** Language spoken in the home - K’iche’ (%) 58.56 10.58 *** Language spoken in the home – Spanish (%) 27.54 3.67 *** Language spoken in the home – Spanish and Ixil (%) 0.00 11.45 *** Language spoken in the home – Spanish and K’iche’ (%) 13.37 4.32 *** Language spoken in the home - Other (%) 0.00 0.00 Household Members 5.38 5.31 Does household chores (%) 99.21 97.41 * Works outside the home (%) 47.35 30.59 *** Total types of tasks 3.07 2.13 *** Spends a lot of time carrying out tasks (%) 52.77 0.00 Observations 463.00 374.00 Value p: ~10%, *5%, **1%, ***0.1% Mid-Term Evaluation IDEA Project SCI P a g e 69 | 81 17.4 Spanish Reading Abilities by Sample Group 17.5 Regression of the dependent variable for gain in the percentage of letters read correctly in Spanish between the control and intervention groups Summary of the model ANOVAa Sum squared gl Quadratic Medium F Sig. Regression 20.416 13 1.570 63.265 .000b Residual 17.550 707 .025 Total 37.967 720 Dependent variable: Gain in percentage of letters in Spanish read correctly. Habilidades Lectoras en Español LB MT Valor Estadístico % de letras correctas en español 55.24 81.64 *** % Palabras más utilizadas correctamente en español 26.72 65.14 *** % decodificación correcta en español 21.72 55.22 *** % de niños que leen un fragmento en español 21.17 56.37 *** Fluidez (palabras correctas por minuto) - español 4.30 31.27 *** Exactitud (% de palabras leídas correctamente) – español 18.35 53.57 *** % comprensión de lectura correcta en español 8.60 35.79 *** Observaciones 463 463 valor p: ~10%, *5%, **1%, ***0.1% R R cuadrado R cuadrado ajustado Error estándar de la estimación .733a .538 .529 .15756 Mid-Term Evaluation IDEA Project SCI P a g e 70 | 81 Coefficientsa Non￾standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta (Constant) .542 .049 11.002 .000 Intervention (Intervened_Control) .028 .013 .060 2.098 .036 Attended Preschool -.008 .014 -.015 -.566 .572 Repeated 1 st or 2 nd grade -.026 .012 -.056 -2.137 .033 Number of actives .006 .004 .045 1.650 .099 Number of animals .007 .005 .035 1.327 .185 Ate breakfast this morning? -.008 .019 -.011 -.427 .670 Members of the household -.002 .004 -.017 -.647 .518 Someone in the community read to me last week -.023 .013 -.051 -1.761 .079 Read to someone in the community .049 .014 .103 3.572 .000 Borrowed a book /story from someone in the community -.018 .013 -.038 -1.404 .161 Did you borrow a book from the book bank last week? -1.780E-05 .000 -.011 -.438 .662 Feels hungry occasionally or continuously during the school day -.030 .026 -.030 -1.166 .244 Value from BL -.612 .025 -.705 -24.532 .000 Dependent variable: Gain in the percentage of letters in Spanish read correctly. 17.6 Regression of dependent variable gain in percentage of correct Spanish MUW between the control and intervention groups Summary of the model Mid-Term Evaluation IDEA Project SCI P a g e 71 | 81 R R squared R squared adjusted Standard Error of the estimate .542a .294 .281 .27825 ANOVAa Sum squared gl Quadratic medium F Sig. Regression 22.772 13 1.752 22.624 .000b Residual 54.740 707 .077 Total 77.512 720 Dependent variable: Gain in the percentage of correct Spanish MUW Coefficientsa Non￾Standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta 1 (Constant) .558 .080 6.967 .000 Intervention (Intervened_Control) -.007 .024 -.011 -.314 .754 Went to preschool -.027 .025 -.036 -1.099 .272 Repeated 1 st or 2nd grade -.066 .022 -.101 -3.039 .002 Number of actives .013 .007 .065 1.941 .053 Number of animals .005 .010 .016 .492 .623 Ate breakfast this morning? -.015 .033 -.015 -.456 .649 Members of the household -.006 .006 -.033 -1.038 .300 Someone in the community read to me last week -.049 .023 -.074 -2.077 .038 I read to someone in the community .087 .024 .129 3.603 .000 I borrowed a book /story from someone in the community -.017 .022 -.026 -.754 .451 Did you borrow books from the book bank last week? 1.058E-05 .000 .005 .148 .883 Mid-Term Evaluation IDEA Project SCI P a g e 72 | 81 Feels hungry occasionally or continuously during the school day .017 .045 .012 .364 .716 Value from BL -.487 .032 -.548 -15.090 .000 Dependent variable: Gain in the percentage of correct Spanish MUW 17.7 Regression of the variable gain in percentage of correct Spanish decoding Summary of the model R R squared R squared adjusted Standard Error of the estimate .519a .270 .256 .30661 ANOVAa Sum squared gl Quadratic medium F Sig. Regression 24.567 13 1.890 20.102 .000b Residual 66.464 707 .094 Total 91.031 720 Dependent variable: gain in the percentage of correct Spanish decoding Mid-Term Evaluation IDEA Project SCI P a g e 73 | 81 Coefficientsa Non￾Standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta 1 (Constant) .497 .087 5.735 .000 Intervention (Intervened_Control) .008 .026 .011 .304 .761 Went to preschool -.034 .027 -.041 -1.262 .207 Repeated 1st or 2nd grade -.085 .024 -.119 -3.548 .000 Number of actives .013 .007 .059 1.739 .083 Number of animals .007 .011 .022 .673 .501 Ate breakfast this morning? -.015 .036 -.013 -.406 .685 Members of the household -.014 .007 -.064 -1.969 .049 Someone in the community read to me last week -.067 .026 -.094 -2.593 .010 I read to someone in the community .113 .026 .154 4.257 .000 I borrowed a book /story from someone in the community -.016 .024 -.023 -.660 .510 Did you borrow books from the book bank last week? 3.887E-05 .000 .016 .493 .622 Feels hungry occasionally or continuously during the school day .044 .050 .028 .873 .383 Value from BL -.483 .034 -.508 -14.111 .000 Dependent variable: gain in the percentage of correct Spanish decoding Mid-Term Evaluation IDEA Project SCI P a g e 74 | 81 17.8 Regression of the dependent variable gain in the percentage of fluidity (wcpm) in Spanish Summary of the model R R squared R squared adjusted Standard Error of the estimate .261a .068 .051 26.08843 ANOVAa Sum squared gl Quadratic medium F Sig. Regression 35203.625 13 2707.971 3.979 .000b Residual 481188.569 707 680.606 Total 516392.194 720 Dependent variable: gain in the percentage of fluidity (wcpm) in Spanish Mid-Term Evaluation IDEA Project SCI P a g e 75 | 81 Coefficientsa Non￾Standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta 1 (Constant) 19.704 7.258 2.715 .007 Intervention (Intervened_Control) 6.785 2.135 .126 3.178 .002 Went to preschool 1.864 2.317 .030 .805 .421 Repeated 1st or 2nd grade -7.590 2.017 -.142 -3.763 .000 Number of actives .922 .613 .058 1.502 .133 Number of animals -1.002 .913 -.041 -1.098 .272 Ate breakfast this morning? -2.607 3.096 -.031 -.842 .400 Members of the household -.867 .585 -.055 -1.480 .139 Someone in the community read to me last week -3.831 2.193 -.072 -1.747 .081 I read to someone in the community 4.471 2.254 .081 1.984 .048 I borrowed a book /story from someone in the community 3.278 2.080 .061 1.576 .115 Did you borrow books from the book bank last week? .000 .007 .002 .047 .963 Feels hungry occasionally or continuously during the school day .094 4.253 .001 .022 .982 Value from BL .158 .082 .077 1.921 .055 Dependent variable: gain in the percentage of fluidity (wcpm) in Spanish Mid-Term Evaluation IDEA Project SCI P a g e 76 | 81 17.9 Regression of dependent variable gain in Spanish accuracy Summary of the model R R squared R squared adjusted Standard Error of the estimate .528a .279 .266 .36890 ANOVAa Sum squared gl Quadratic medium F Sig. Regression 37.245 13 2.865 21.053 .000b Residual 96.214 707 .136 Total 133.459 720 Dependent variable: Gain in Spanish accuracy Mid-Term Evaluation IDEA Project SCI P a g e 77 | 81 Coefficientsa Non￾Standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta (Constant) .721 .102 7.053 .000 Intervention (Intervened_Control) -.044 .030 -.051 -1.458 .145 Went to preschool .010 .033 .010 .299 .765 Repeated 1st or 2nd grade -.133 .029 -.155 -4.674 .000 Number of actives .010 .009 .038 1.129 .259 Number of animals -.013 .013 -.033 -.997 .319 Ate breakfast this morning? -.024 .044 -.018 -.543 .587 Members of the household -.025 .008 -.098 -3.002 .003 Someone in the community read to me last week -.048 .031 -.056 -1.550 .121 I read to someone in the community .108 .032 .122 3.397 .001 I borrowed a book /story from someone in the community -.004 .029 -.004 -.122 .903 Did you borrow books from the book bank last week? 8.481E-05 .000 .029 .894 .372 Feels hungry occasionally or continuously during the school day .056 .060 .030 .927 .354 Value from BL -.579 .037 -.547 -15.566 .000 Dependent variable: Gain in Spanish accuracy Mid-Term Evaluation IDEA Project SCI P a g e 78 | 81 17.10 Regression of the dependent variable gain in the percentage of correct comprehension in Spanish Summary of the model R R squared R squared adjusted Standard Error of the estimate R .488a .238 .224 .30240 ANOVAa Sum squared gl Quadratic medium F Sig. Regression 20.187 13 1.553 16.980 .000b Residual 64.653 707 .091 Total 84.840 720 Dependent variable: gain in correct comprehension in Spanish Mid-Term Evaluation IDEA Project SCI P a g e 79 | 81 Coefficientsa Non￾Standardized Coefficients Standardized Coefficients B t Sig. Standard Error Beta (Constant) .221 .083 2.667 .008 Intervention (Intervened_Control) .091 .024 .132 3.732 .000 Went to preschool .029 .027 .036 1.087 .278 Repeated 1st or 2nd grade -.092 .023 -.134 -3.962 .000 Number of actives .027 .007 .131 3.732 .000 Number of animals -.015 .011 -.047 -1.385 .166 Ate breakfast this morning? .013 .036 .013 .375 .707 Members of the household -.025 .007 -.126 -3.749 .000 Someone in the community read to me last week -.012 .025 -.017 -.464 .643 I read to someone in the community .045 .026 .064 1.732 .084 I borrowed a book /story from someone in the community -.021 .024 -.031 -.883 .378 Did you borrow books from the book bank last week? .000 .000 .071 2.112 .035 Feels hungry occasionally or continuously during the school day .032 .049 .022 .658 .511 Value from BL -.505 .041 -.443 -12.284 .000 Dependent variable: gain in correct comprehension in Spanish Mid-Term Evaluation IDEA Project SCI P a g e 80 | 81 17.11 Comparative Data from results of the student interviews BL and MTE, Literacy Environment Value p: ~10%, *5%, **1%, ***0.1% Ambiente propiciopara Lectoescritura en el Hogar y Comunidad (%) Tiene libros de texto 71.06 73.65 Tiene libros religiosos 79.27 73.87 ~ Tiene revistas 0.86 3.46 ** Tiene periódicos 20.30 14.47 * Tiene libros de cuentos 31.97 27.86 Tiene libros para colorear 25.70 26.13 % de miembros de la familia que fueron vistos leyendo 47.14 45.02 % de miembros de la familia que ayudan al niño a estudiar 68.97 65.19 * % de miembros de la familia que leen al niño 28.60 39.35 *** % de miembros de la familia que cuentan historias al niño 30.81 42.14 *** Alguien de la comunidad te leyó algo? 28.29 48.70 *** Leíste algo para alguien en la comunidad? 38.23 62.42 *** Pediste prestado un libro de cuentos fuera de tu hogar? 34.85 40.39 ~ Observaciones 463 463 Muestra LB MT Valor Estadístico Mid-Term Evaluation IDEA Project SCI P a g e 81 | 81 17.12 Health and Attention by Sample Group Salud y Atención (%) LB MT Valor Estadístico Desayunó esta mañana 90.50 84.23 ** La escuela brinda meriendas 49.24 91.14 *** Utiliza letrina en el hogar 79.69 50.54 *** Siente hambre en diversas ocasiones o constantemente durante la jornada escolar 22.78 4.55 *** Pone atención todo el tiempo o la mayoría del tiempo mientras está en clase 91.34 96.11 ** Observaciones 463 463 valor p: ~10%, *5%, **1%, ***0.1%